lsf_stats

Summarize LSF job properties by parsing log files of workflows executed by Snakemake.

Installation

$ pip install lsf_stats

Usage

$ lsf_stats --help
Usage: lsf_stats [OPTIONS] COMMAND [ARGS]...

  Summarize LSF job properties by parsing log files.

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  gather     Aggregate information from log files in single dataframe.
  summarize  Summarize and visualize aggregated information.

Example

Assume that you executed your Snakemake workflow using the lsf-profile and all generated log files are stored in the directory ./logs/:

$ snakemake --profile lsf
[..]

You can then quickly aggregate resource, rule and other types of information about the workflow execution into a single dataframe:

$ lsf_stats gather -o workflow_stats.csv.gz ./logs/
[..]

This dataframe can then be summarized in various ways:

$ lsf_stats summarize \
    --query 'status == "Successfully completed."' \
    --split-wildcards \
    --grouping-variable category \
    workflow_stats.csv.gz
[..]

For example, the following plots will be generated:

Job execution	Job resources

Summarize LSF job properties by parsing log files.

Related tags

Overview

lsf_stats

Installation

Usage

Example

Owner

Kim

Ransomware leak site monitoring

Splunk Add-On to collect audit log events from Github Enterprise Cloud

Espion is a mini-keylogger tool that keeps track of all keys a user presses on his/her keyboard

This is a DemoCode for parsing through large log files and triggering an email whenever there's an error.

Json Formatter for the standard python logger

Simple and versatile logging library for python 3.6 above

Python logging package for easy reproducible experimenting in research

A Fast, Extensible Progress Bar for Python and CLI

Monitor and log Network and Disks statistics in MegaBytes per second.

Summarize LSF job properties by parsing log files.

ClusterMonitor - a very simple python script which monitors and records the CPU and RAM consumption of submitted cluster jobs

Multi-processing capable print-like logger for Python

A basic logging library for Python.

Pretty-print tabular data in Python, a library and a command-line utility. Repository migrated from bitbucket.org/astanin/python-tabulate.

A demo of Prometheus+Grafana for monitoring an ML model served with FastAPI.

Soda SQL Data testing, monitoring and profiling for SQL accessible data.

A Python package which supports global logfmt formatted logging.

A simple package that allows you to save inputs & outputs as .log files

Key Logger - Key Logger using Python

Python script to scan log files/system for unauthorized access around system