easy_sbatch - Batch submitting Slurm jobs with script templates

Overview

easy_sbatch

easy_sbatch - Batch submitting Slurm jobs via script templates

Install

git clone https://github.com/shenwei356/easy_sbatch.git
mkdir -p ~/bin
cp easy_sbatch ~/bin/

Quick start

  1. A simple command (no file or data given):

     $ easy_sbatch 'cat /etc/hostname'
    
     $ ls
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 21:50 easy_sbatch.15565347.err
     -rw-r--r-- 1 shenwei cqmu   9 Dec 27 21:50 easy_sbatch.15565347.out
     -rw------- 1 shenwei cqmu 314 Dec 27 21:50 easy_sbatch.15565347.slurm
    
  2. Handling multiple files and specifying a job name:

     $ easy_sbatch 'ls {}' *.fq.gz -J list
    
     $ ls
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 21:54 list.15565371-read_1.fq.gz.err
     -rw-r--r-- 1 shenwei cqmu  13 Dec 27 21:54 list.15565371-read_1.fq.gz.out
     -rw------- 1 shenwei cqmu 317 Dec 27 21:54 list.15565371-read_1.fq.gz.slurm
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 21:55 list.15565373-read_2.fq.gz.err
     -rw-r--r-- 1 shenwei cqmu  13 Dec 27 21:55 list.15565373-read_2.fq.gz.out
     -rw------- 1 shenwei cqmu 317 Dec 27 21:54 list.15565373-read_2.fq.gz.slurm
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 20:04 read_1.fq.gz
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 20:04 read_2.fq.gz
    
  3. From stdin:

     $ ls *.fq.gz | easy_sbatch 'echo {/} {%} {%^.fq.gz}'
    
  4. Processing paired-ends FASTQ files:

     $ ls read_1.fq.gz \
         | easy_sbatch 'echo seqtk mergepe {} {^_1.fq.gz}_2.fq.gz' -J pe
    
     $ ls
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 21:56 pe.15565379-read_1.fq.gz.err
     -rw-r--r-- 1 shenwei cqmu  40 Dec 27 21:56 pe.15565379-read_1.fq.gz.out
     -rw------- 1 shenwei cqmu 340 Dec 27 21:56 pe.15565379-read_1.fq.gz.slurm
    
     $ cat pe.15565379-read_1.fq.gz.out
     seqtk mergepe read_1.fq.gz read_2.fq.gz
    

Default template

You can choose another one via the option -t/--template.

You can edit the value of an option, e.g., replace $partition with the default partition of your cluster. But note that the corresponding option from the command line will not take effect since no variable name is ready to replace.

$ cat ~/.easy_sbatch/default.slurm 
#!/bin/bash
#
#SBATCH --partition $partition
#SBATCH --job-name $name
#SBATCH --nodes=1
#SBATCH --cpus-per-task=$ncpus
#SBATCH --mem=$mem
#SBATCH --time=$walltime
#SBATCH --output=$output
#SBATCH --error=$error


cd $$SLURM_SUBMIT_DIR
# echo run on node: $$HOSTNAME >&2

$cmd

Usage

. .out]) -e ERROR, --error ERROR file for batch script's standard error (default: . .err]) -v, --verbose verbosely print information. -vv for just printing command not creating scripts and submitting jobs https://github.com/shenwei356/easy_sbatch ">
usage: easy_sbatch [-h] [-lp | -ls] [-J NAME] [-c NCPUS] [-m MEM] [-p PARTITION] [-w WALLTIME] [-t TEMPLATE]
                   [-O SCRIPT] [-o OUTPUT] [-e ERROR] [-v]
                   command [files ...]

easy_sbatch - Batch submitting Slurm jobs with script template

https://github.com/shenwei356/easy_sbatch 

Slurm script template:

  - Default template is ~/.easy_qsub/default.slurm .
  - You can also choose another one via the option -t/--template.
  - You can edit the value of an option, e.g., replace "$partition"
    with the default partition of your cluster. But note that the
    corresponding option from the command line will not take effect
    since no variable name is ready to replace.

Supported replacement strings in command:

  - "{}" for the full path of a input file.
  - "{/}" for dirname.
  - "{%}" for basename.
  - "{^suffix}" for clipping $suffix.
  - "{%^suffix}" for clipping $suffix from basename.

The generated Slurm scripts are saved in /tmp/easy_qsub-user/ by default,
unless the flag -O/--script is set. If jobs are submitted successfully,
the scripts will be moved to the current directory. If not, they will
be removed.

Examples:

  1. No file or data given:
      $ easy_sbatch 'cat /etc/hostname'

  2. From position arguments:
      $ easy_sbatch 'ls {}' *.fq.gz

  3. From stdin:
      $ ls *.fq.gz | easy_sbatch 'echo {/} {%} {%^.fq.gz}'

  4. Slurm script and its output files:
      $ easy_sbatch 'ls' -J hello
        -rw-r--r-- 1 shenwei cqmu   0 Dec 27 20:57 hello.15565039.err
        -rw-r--r-- 1 shenwei cqmu  85 Dec 27 20:57 hello.15565039.out
        -rw------- 1 shenwei cqmu 281 Dec 27 20:57 hello.15565039.slurm

  5. Processing paired-ends FASTQ files:
      $ ls read_1.fq.gz | easy_sbatch 'echo seqtk mergepe {} {^_1.fq.gz}_2.fq.gz' -J pe

      $ cat pe.15565114.out 
        seqtk mergepe read_1.fq.gz read_2.fq.gz

positional arguments:
  command               command to submit
  files                 input files

optional arguments:
  -h, --help            show this help message and exit
  -lp, --local_p        run commands locally, parallelly
  -ls, --local_s        run commands locally, serially
  -J NAME, --name NAME  job name (default: easy_sbatch)
  -c NCPUS, --ncpus NCPUS
                        number of cpus required per task (default: 24)
  -m MEM, --mem MEM     memory (default: 24gb)
  -p PARTITION, --partition PARTITION
                        partition requested (default: kshdnormal)
  -w WALLTIME, --walltime WALLTIME
                        walltime (default: 30-00:00:00)
  -t TEMPLATE, --template TEMPLATE
                        script template (default: /home/shenwei/.easy_sbatch/default.slurm
  -O SCRIPT, --script SCRIPT
                        output script file
  -o OUTPUT, --output OUTPUT
                        file for batch script's standard output (default: 
      
       .
       
        .out])
  -e ERROR, --error ERROR
                        file for batch script's standard error (default: 
        
         .
         
          .err])
  -v, --verbose         verbosely print information. -vv for just printing command not creating scripts and submitting jobs

https://github.com/shenwei356/easy_sbatch


         
        
       
      

License

MIT License

History

This utility is modified from easy_qsub, which was written for PBS job scheduler.

Owner
Wei Shen
Bioinformatician, postdoc
Wei Shen
log4shell pwner for vulnerable minecraft servers

Log4-hell name supposed to be Log4$hell but oh well log4shell pwner for vulnerable minecraft servers install all reqs python + a minecraft client for

1 Jan 05, 2022
Wunderland desktop wallpaper and Microsoft Teams background.

Wunderland Professional Impress your colleagues, friends and family with this edition of the "Wunderland" wallpaper. With the nostalgic feel of the or

3 Dec 14, 2022
This python application let you check for new announcements from MMLS, take attendance while your lecturer is sharing QR Code on the screen.

This python application let you check for new announcements from MMLS, take attendance while your lecturer is sharing QR Code on the screen.

wyhong3103 5 Jul 17, 2022
Build your own Etherscan with web3.py

Build your own Etherscan with web3.py Video Tutorial: Run it pip3 install -r requirements.txt export FLASK_APP=app export FLASK_ENV=development flask

35 Jan 02, 2023
This is a Poetry plugin that will make it possible to build projects using custom TOML files

Poetry Multiproject Plugin This is a Poetry plugin that will make it possible to build projects using custom TOML files. This is especially useful whe

David Vujic 69 Dec 25, 2022
My tools box script for sigma

sigma_python_toolbox My tools box script for sigma purpose My goal is not to replace sigma but to put at disposal the scripts that I think to help me

4 Jun 20, 2022
- Auto join teams teams ( from calendar invite )

Auto Join Teams Meetings Requirements: Python 3.7 or higher Latest Google Chrome This script automatically logins to your account and joins the meetin

Prajin Khadka 10 Aug 20, 2022
Painel simples com consulta de cep,CNPJ,placa e ip

Painel mpm Um painel simples com consultas de IP, CNPJ, CEP, PLACA, TELEFONE, CPF e NOME Início 🌐 apt update && apt upgrade -y pkg i python git pip i

8 Feb 27, 2022
Video Stream is an Advanced Telegram Bot that's allow you to play Video & Music on Telegram Group Video Chat

Video Stream is an Advanced Telegram Bot that's allow you to play Video & Music on Telegram Group Video Chat 📊 Stats 🧪 Get SESSION_NAME from below:

dark phoenix 12 May 08, 2022
WhyNotWin11 - Detection Script to help identify why your PC isn't Windows 11 Release Ready

WhyNotWin11 - Detection Script to help identify why your PC isn't Windows 11 Release Ready

Robert C. Maehl 5.9k Dec 31, 2022
Distribute PySPI jobs across a PBS cluster

Distribute PySPI jobs across a PBS cluster This repository contains scripts for distributing PySPI jobs across a PBS-type cluster. Each job will conta

Oliver Cliff 1 Feb 10, 2022
Functional interface for concurrent futures, including asynchronous I/O.

Futured provides a consistent interface for concurrent functional programming in Python. It wraps any callable to return a concurrent.futures.Future,

A. Coady 11 Nov 27, 2022
A webapp that timestamps key moments in a football clip

A look into what we're building Demo.mp4 Prerequisites Python 3 Node v16+ Steps to run Create a virtual environment. Activate the virtual environment.

Pranav 1 Dec 10, 2021
A hackers attempt at an MVP anki plugin

my anki plugin if you have found this by accident, you should probably run away this is nothing more than a hackers attempt at an MVP anki plugin I re

Chris Hall 1 Nov 02, 2021
HOWTO: Downgrade from nYNAB to YNAB4

HOWTO: Downgrade from nYNAB to YNAB4 This page explains how to move from nYNAB to YNAB4 while retaining as much information as possible. See Appendix

Tobias Kunze 10 Dec 29, 2022
Hacktoberfest2021 🥳- Contribute Any Pattern In Any Language😎 Every PR will be accepted Pls contribute

✨ Hacktober Fest 2021 ✨ 🙂 All Contributors are requested to star this repo and follow me for a successful merge of pull request. 🙂 👉 Add any patter

Md. Almas Ali 103 Jan 07, 2023
Script para generar automatización de registro de formularios IEEH

Formularios_IEEH Script para generar automatización de registro de formularios IEEH Corresponde a un conjunto de script en python que permiten la auto

vhevia11 1 Jan 06, 2022
A fluid medium for storing, relating, and surfacing thoughts.

Conceptarium A fluid medium for storing, relating, and surfacing thoughts. Read more... Instructions The conceptarium takes up about 1GB RAM when runn

115 Dec 19, 2022
Python module for creating the circuit simulation definitions for Elmer FEM

elmer_circuitbuilder Python module for creating the circuit simulation definitions for Elmer FEM. The circuit definitions enable easy setup of coils (

5 Oct 03, 2022
A community-driven python bot that aims to be as simple as possible to serve humans with their everyday tasks

JARVIS on Messenger Just A Rather Very Intelligent System, now on Messenger! Messenger is now used by 1.2 billion people every month. With the launch

Swapnil Agarwal 1.3k Jan 07, 2023