easy_sbatch - Batch submitting Slurm jobs with script templates

Overview

easy_sbatch

easy_sbatch - Batch submitting Slurm jobs via script templates

Install

git clone https://github.com/shenwei356/easy_sbatch.git
mkdir -p ~/bin
cp easy_sbatch ~/bin/

Quick start

  1. A simple command (no file or data given):

     $ easy_sbatch 'cat /etc/hostname'
    
     $ ls
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 21:50 easy_sbatch.15565347.err
     -rw-r--r-- 1 shenwei cqmu   9 Dec 27 21:50 easy_sbatch.15565347.out
     -rw------- 1 shenwei cqmu 314 Dec 27 21:50 easy_sbatch.15565347.slurm
    
  2. Handling multiple files and specifying a job name:

     $ easy_sbatch 'ls {}' *.fq.gz -J list
    
     $ ls
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 21:54 list.15565371-read_1.fq.gz.err
     -rw-r--r-- 1 shenwei cqmu  13 Dec 27 21:54 list.15565371-read_1.fq.gz.out
     -rw------- 1 shenwei cqmu 317 Dec 27 21:54 list.15565371-read_1.fq.gz.slurm
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 21:55 list.15565373-read_2.fq.gz.err
     -rw-r--r-- 1 shenwei cqmu  13 Dec 27 21:55 list.15565373-read_2.fq.gz.out
     -rw------- 1 shenwei cqmu 317 Dec 27 21:54 list.15565373-read_2.fq.gz.slurm
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 20:04 read_1.fq.gz
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 20:04 read_2.fq.gz
    
  3. From stdin:

     $ ls *.fq.gz | easy_sbatch 'echo {/} {%} {%^.fq.gz}'
    
  4. Processing paired-ends FASTQ files:

     $ ls read_1.fq.gz \
         | easy_sbatch 'echo seqtk mergepe {} {^_1.fq.gz}_2.fq.gz' -J pe
    
     $ ls
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 21:56 pe.15565379-read_1.fq.gz.err
     -rw-r--r-- 1 shenwei cqmu  40 Dec 27 21:56 pe.15565379-read_1.fq.gz.out
     -rw------- 1 shenwei cqmu 340 Dec 27 21:56 pe.15565379-read_1.fq.gz.slurm
    
     $ cat pe.15565379-read_1.fq.gz.out
     seqtk mergepe read_1.fq.gz read_2.fq.gz
    

Default template

You can choose another one via the option -t/--template.

You can edit the value of an option, e.g., replace $partition with the default partition of your cluster. But note that the corresponding option from the command line will not take effect since no variable name is ready to replace.

$ cat ~/.easy_sbatch/default.slurm 
#!/bin/bash
#
#SBATCH --partition $partition
#SBATCH --job-name $name
#SBATCH --nodes=1
#SBATCH --cpus-per-task=$ncpus
#SBATCH --mem=$mem
#SBATCH --time=$walltime
#SBATCH --output=$output
#SBATCH --error=$error


cd $$SLURM_SUBMIT_DIR
# echo run on node: $$HOSTNAME >&2

$cmd

Usage

. .out]) -e ERROR, --error ERROR file for batch script's standard error (default: . .err]) -v, --verbose verbosely print information. -vv for just printing command not creating scripts and submitting jobs https://github.com/shenwei356/easy_sbatch ">
usage: easy_sbatch [-h] [-lp | -ls] [-J NAME] [-c NCPUS] [-m MEM] [-p PARTITION] [-w WALLTIME] [-t TEMPLATE]
                   [-O SCRIPT] [-o OUTPUT] [-e ERROR] [-v]
                   command [files ...]

easy_sbatch - Batch submitting Slurm jobs with script template

https://github.com/shenwei356/easy_sbatch 

Slurm script template:

  - Default template is ~/.easy_qsub/default.slurm .
  - You can also choose another one via the option -t/--template.
  - You can edit the value of an option, e.g., replace "$partition"
    with the default partition of your cluster. But note that the
    corresponding option from the command line will not take effect
    since no variable name is ready to replace.

Supported replacement strings in command:

  - "{}" for the full path of a input file.
  - "{/}" for dirname.
  - "{%}" for basename.
  - "{^suffix}" for clipping $suffix.
  - "{%^suffix}" for clipping $suffix from basename.

The generated Slurm scripts are saved in /tmp/easy_qsub-user/ by default,
unless the flag -O/--script is set. If jobs are submitted successfully,
the scripts will be moved to the current directory. If not, they will
be removed.

Examples:

  1. No file or data given:
      $ easy_sbatch 'cat /etc/hostname'

  2. From position arguments:
      $ easy_sbatch 'ls {}' *.fq.gz

  3. From stdin:
      $ ls *.fq.gz | easy_sbatch 'echo {/} {%} {%^.fq.gz}'

  4. Slurm script and its output files:
      $ easy_sbatch 'ls' -J hello
        -rw-r--r-- 1 shenwei cqmu   0 Dec 27 20:57 hello.15565039.err
        -rw-r--r-- 1 shenwei cqmu  85 Dec 27 20:57 hello.15565039.out
        -rw------- 1 shenwei cqmu 281 Dec 27 20:57 hello.15565039.slurm

  5. Processing paired-ends FASTQ files:
      $ ls read_1.fq.gz | easy_sbatch 'echo seqtk mergepe {} {^_1.fq.gz}_2.fq.gz' -J pe

      $ cat pe.15565114.out 
        seqtk mergepe read_1.fq.gz read_2.fq.gz

positional arguments:
  command               command to submit
  files                 input files

optional arguments:
  -h, --help            show this help message and exit
  -lp, --local_p        run commands locally, parallelly
  -ls, --local_s        run commands locally, serially
  -J NAME, --name NAME  job name (default: easy_sbatch)
  -c NCPUS, --ncpus NCPUS
                        number of cpus required per task (default: 24)
  -m MEM, --mem MEM     memory (default: 24gb)
  -p PARTITION, --partition PARTITION
                        partition requested (default: kshdnormal)
  -w WALLTIME, --walltime WALLTIME
                        walltime (default: 30-00:00:00)
  -t TEMPLATE, --template TEMPLATE
                        script template (default: /home/shenwei/.easy_sbatch/default.slurm
  -O SCRIPT, --script SCRIPT
                        output script file
  -o OUTPUT, --output OUTPUT
                        file for batch script's standard output (default: 
      
       .
       
        .out])
  -e ERROR, --error ERROR
                        file for batch script's standard error (default: 
        
         .
         
          .err])
  -v, --verbose         verbosely print information. -vv for just printing command not creating scripts and submitting jobs

https://github.com/shenwei356/easy_sbatch


         
        
       
      

License

MIT License

History

This utility is modified from easy_qsub, which was written for PBS job scheduler.

Owner
Wei Shen
Bioinformatician, postdoc
Wei Shen
SciPy library main repository

SciPy SciPy (pronounced "Sigh Pie") is an open-source software for mathematics, science, and engineering. It includes modules for statistics, optimiza

SciPy 10.7k Jan 09, 2023
Minimal, super readable string pattern matching for python.

simplematch Minimal, super readable string pattern matching for python. import simplematch simplematch.match("He* {planet}!", "Hello World!") {"p

Thomas Feldmann 147 Dec 01, 2022
carrier.py is a Python package/module that's used to save time when programming

carrier.py is a Python package/module that's used to save time when programming, it helps with functions such as 24 and 12 hour time, Discord webhooks, etc

Zacky2613 2 Mar 20, 2022
The official Repository wherein newbies into Open Source can Contribute during the Hacktoberfest 2021

Hacktoberfest 2021 Get Started With your first Contrinution/Pull Request : Fork/Copy the repo by clicking the right most button on top of the page. Go

HacOkars 25 Aug 20, 2022
Ningyu Jia(nj2459)/Mengyin Ma(mm5937) Call Analysis group project(Group 36)

Group and Section Group 36 Section 001 name and UNI Name UNI Ningyu Jia nj2459 Mengyin Ma mm5937 code explanation Parking.py (1) Calculate the rate of

1 Dec 04, 2021
An example module hooking system, will be used in PySAMP.

An example module hooking system, will be used in PySAMP.

2 May 01, 2022
One destination for all the developer's learning resources.

DevResources One destination for all the developer's learning resources. Find all of your learning resources under one roof and add your own. Live ✨ Y

Gaurav Sharma 33 Oct 21, 2022
Process GPX files (adding sensor metrics, uploading to InfluxDB, etc.) exported from imxingzhe.com

Xingzhe GPX Processor 行者轨迹处理工具 Xingzhe sells cheap GPS bike meters with sensor support including cadence, heart rate and power. But the GPX files expo

Shengqi Chen 8 Sep 23, 2022
MDAnalysis tool to calculate membrane curvature.

The MDAkit for membrane curvature analysis is part of the Google Summer of Code program and it is linked to a Code of Conduct.

MDAnalysis 19 Oct 20, 2022
B-Pkg is a simple tool in python for installing all basic package in termux

Basic-Pkg 👉🏻 Basic-Pkg 👈🏻 B-Pkg is a simple tool in python for installing all basic package in termux This is my first tool, I hope you will like

Macgaiver 3 Oct 21, 2021
Simple project to assist in tracking/logging my working hours

Fill working hours Basic script to assist in the logging/tracking of my working hours How it works Create a file called projects.json in this director

Robin Kennedy-Reid 2 Oct 31, 2022
VHDL to Discrete Logic on PCB Flow

PCBFlow Highly experimental set of scripts to transform a digital circuit described in a hardware description language (VHDL or Verilog) into a discre

Tim 77 Nov 04, 2022
Simple project to learn more about Bézier curves

Python Quadratic Bézier Simple project to learn more about Bézier curves. On this project i used some api's to graphics and gui pygame thorpy in theor

Kenned Ferreira 2 Mar 06, 2022
A collection of python exercises to help your learning path!

How to use Step 1: run this command git clone https://github.com/TechPenguineer/Python-Exercises.git Step 2: Run this command cd Python-Exercises You

Tech Penguin 5 Aug 05, 2021
An app to automatically take attendance by scanning students' bar coded ID card as they enter the classroom.

Auto Classroom Attendance This application may be run on a PC to automatically scan students' ID card using a generic bar code scanner and output the

1 Nov 10, 2021
Sacred is a tool to help you configure, organize, log and reproduce experiments developed at IDSIA.

Sacred Every experiment is sacred Every experiment is great If an experiment is wasted God gets quite irate Sacred is a tool to help you configure, or

IDSIA 4k Jan 02, 2023
A python library for writing parser-based interactive fiction.

About IntFicPy A python library for writing parser-based interactive fiction. Currently in early development. IntFicPy Docs Parser-based interactive f

Rita Lester 31 Nov 23, 2022
Blender addon for executing the operator in response to the received OSC message.

I/F Joiner 受信したOSCメッセージに応じてオペレータ(bpy.ops)を実行するアドオンです. OSC通信に対応したコントローラやアプリをインストールしたスマートフォンを使用してBlenderを操作することが可能になります. 同時開発しているAndroidコントローラ化アプリMocopa

simasimataiyo 6 Oct 02, 2022
If Google News had a Python library

pygooglenews If Google News had a Python library Created by Artem from newscatcherapi.com but you do not need anything from us or from anyone else to

Artem Bugara 1.1k Jan 08, 2023
Customisable coding font with alternates, ligatures and contextual positioning

Guide Ligature Support Links Log License Guide Live Preview + Download larsenwork.com/monoid Install Quit your editor/program. Unzip and open the fold

Andreas Larsen 7.6k Dec 30, 2022