lsf_stats

Summarize LSF job properties by parsing log files of workflows executed by Snakemake.

Installation

$ pip install lsf_stats

Usage

$ lsf_stats --help
Usage: lsf_stats [OPTIONS] COMMAND [ARGS]...

  Summarize LSF job properties by parsing log files.

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  gather     Aggregate information from log files in single dataframe.
  summarize  Summarize and visualize aggregated information.

Example

Assume that you executed your Snakemake workflow using the lsf-profile and all generated log files are stored in the directory ./logs/:

$ snakemake --profile lsf
[..]

You can then quickly aggregate resource, rule and other types of information about the workflow execution into a single dataframe:

$ lsf_stats gather -o workflow_stats.csv.gz ./logs/
[..]

This dataframe can then be summarized in various ways:

$ lsf_stats summarize \
    --query 'status == "Successfully completed."' \
    --split-wildcards \
    --grouping-variable category \
    workflow_stats.csv.gz
[..]

For example, the following plots will be generated:

Job execution	Job resources

Summarize LSF job properties by parsing log files.

Related tags

Overview

lsf_stats

Installation

Usage

Example

Owner

Kim

Pretty-print tabular data in Python, a library and a command-line utility. Repository migrated from bitbucket.org/astanin/python-tabulate.

Simple and versatile logging library for python 3.6 above

A demo of Prometheus+Grafana for monitoring an ML model served with FastAPI.

Keylogger with Python which logs words into server terminal.

Python logging made (stupidly) simple

Python logging package for easy reproducible experimenting in research

Command-line tool that instantly fetches Stack Overflow results when an exception is thrown

Small toolkit for python multiprocessing logging to file

APT-Hunter is Threat Hunting tool for windows event logs

ClusterMonitor - a very simple python script which monitors and records the CPU and RAM consumption of submitted cluster jobs

Multi-processing capable print-like logger for Python

Lazy Profiler is a simple utility to collect CPU, GPU, RAM and GPU Memory stats while the program is running.

Espion is a mini-keylogger tool that keeps track of all keys a user presses on his/her keyboard

Beautifully colored, quick and simple Python logging

A basic logging library for Python.

The easy way to send notifications

A simple, transparent, open-source key logger, written in Python, for tracking your own key-usage statistics.

Debugging-friendly exceptions for Python

Monitoring plugin to check disk io with Icinga, Nagios and other compatible monitoring solutions

A python logging library