ClusterMonitor - a very simple python script which monitors and records the CPU and RAM consumption of submitted cluster jobs

Overview

ClusterMonitor

A very simple python script which monitors and records the CPU and RAM consumption of submitted cluster jobs.

Usage

To start recording use the cpu_ram_log.py script. This script requires 3 arguments:

  • -u which corresponds to your username.
  • -o which specifies the output file (in tsv format)
  • --interval which specifies the time (in seconds) over which the CPU and RAM usage gets averaged

Example:

python cpu_ram_log.py -u nickhir -o cpu_ram.log --interval 5

The script can simply be included in your cluster job submittion like this:

#!/bin/bash
#SBATCH --job-name=Example_run

sambamba sort -t 20 some_bam.bam &
MainScript=$!
python cpu_ram_log.py -u nickhir -o cpu_ram.log --interval 5 &
wait $MainScript
pkill -P $$

Afterwards the resulting log file can be plotted (for that matplotlib is needed). For that use the visualize_log.py script which needs 2 arguments:

  • -i which specifies the path to the log file which was created with cpu_ram_log.py
  • -o which specifies the path to the resulting PDF which contains a plot for the CPU and RAM usage.

Example:

python visualize_log.py -i cpu_ram.log -o cpu_ram_visualization.pdf

The resulting plots will look somewhat like this: exampleImage

Monitor and log Network and Disks statistics in MegaBytes per second.

iometrics Monitor and log Network and Disks statistics in MegaBytes per second. Install pip install iometrics Usage Pytorch-lightning integration from

Leo Gallucci 17 May 03, 2022
Debugging-friendly exceptions for Python

Better tracebacks This is a more helpful version of Python's built-in exception message: It shows more code context and the current values of nearby v

Clemens Korndörfer 1.2k Dec 28, 2022
Python bindings for g3log

g3logPython Python bindings for g3log This library provides python3 bindings for g3log + g3sinks (currently logrotate, syslog, and a color-terminal ou

4 May 21, 2021
Vibrating-perimeter - Simple helper mod that logs how fast you are mining together with a simple buttplug.io script to control a vibrator

Vibrating Perimeter This project consists of a small minecraft helper mod that writes too a log file and a script that reads said log. Currently it on

Heart[BOT] 0 Nov 20, 2022
A colored formatter for the python logging module

Log formatting with colors! colorlog.ColoredFormatter is a formatter for use with Python's logging module that outputs records using terminal colors.

Sam Clements 778 Dec 26, 2022
metovlogs is a very simple logging library

metovlogs is a very simple logging library. Setup is one line, then you can use it as a drop-in print replacement. Sane and useful log format out of the box. Best for small or early projects.

Azat Akhmetov 1 Mar 01, 2022
Json Formatter for the standard python logger

This library is provided to allow standard python logging to output log data as json objects. With JSON we can make our logs more readable by machines and we can stop writing custom parsers for syslo

Zakaria Zajac 1.4k Jan 04, 2023
Monitor creation, deletion and changes to LDAP objects live during your pentest or system administration!

LDAP Monitor Monitor creation, deletion and changes to LDAP objects live during your pentest or system administration! With this tool you can quickly

Podalirius 500 Dec 28, 2022
Greppin' Logs: Leveling Up Log Analysis

This repo contains sample code and example datasets from Jon Stewart and Noah Rubin's presentation at the 2021 SANS DFIR Summit titled Greppin' Logs. The talk was centered around the idea that Forens

Stroz Friedberg 20 Sep 14, 2022
Fancy console logger and wise assistant within your python projects

Fancy console logger and wise assistant within your python projects. Made to save tons of hours for common routines.

BoB 5 Apr 01, 2022
dash-manufacture-spc-dashboard is a dashboard for monitoring read-time process quality along manufacture production line

In our solution based on plotly, dash and influxdb, the user will firstly generate the specifications for different robots, and then a wide range of interactive visualizations for different machines

Dequn Teng 1 Feb 13, 2022
A cool logging replacement for Python.

Welcome to Logbook Travis AppVeyor Supported Versions Latest Version Test Coverage Logbook is a nice logging replacement. It should be easy to setup,

1.4k Nov 11, 2022
This is a DemoCode for parsing through large log files and triggering an email whenever there's an error.

LogFileParserDemoCode This is a DemoCode for parsing through large log files and triggering an email whenever there's an error. There are a total of f

2 Jan 06, 2022
Yaml - Loggers are like print() statements

Upgrade your print statements Loggers are like print() statements except they also include loads of other metadata: timestamp msg (same as print!) arg

isaac peterson 38 Jul 20, 2022
A new kind of Progress Bar, with real time throughput, eta and very cool animations!

alive-progress :) A new kind of Progress Bar, with real-time throughput, eta and very cool animations! Ever found yourself in a remote ssh session, do

Rogério Sampaio de Almeida 4k Dec 30, 2022
Ultimate Logger - A Discord bot that logs lots of events in a channel written in python

Ultimate Logger - A Discord bot that logs lots of events in a channel written in python

Luca 2 Mar 27, 2022
A lightweight logging library for python applications

cakelog a lightweight logging library for python applications This is a very small logging library to make logging in python easy and simple. config o

2 Jan 05, 2022
Beautifully colored, quick and simple Python logging

Python Quick Logging | QLogging Beautifully colored, quick and simple Python logging. This logger is based on Python logging package Screenshots: Term

45 Sep 25, 2022
蓝鲸日志平台(BK-LOG)是为解决分布式架构下日志收集、查询困难的一款日志产品,基于业界主流的全文检索引擎

蓝鲸日志平台(BK-LOG)是为解决分布式架构下日志收集、查询困难的一款日志产品,基于业界主流的全文检索引擎,通过蓝鲸智云的专属 Agent 进行日志采集,提供多种场景化的采集、查询功能。

腾讯蓝鲸 102 Dec 22, 2022
Docker container log aggregation with Elasticsearch, Kibana & Filebeat

Epilog Dead simple container log aggregation with ELK stack Preface Epilog aims to demonstrate a language-agnostic, non-invasive, and straightfo

Redowan Delowar 23 Oct 26, 2022