Monitor and log Network and Disks statistics in MegaBytes per second.

Overview

iometrics

Python

Monitor and log Network and Disks statistics in MegaBytes per second.

Install

pip install iometrics

Usage

Pytorch-lightning integration

from pytorch_lightning import Trainer
from iometrics.pytorch_lightning.callbacks import NetworkAndDiskStatsMonitor

net_disk_stats = NetworkAndDiskStatsMonitor()

trainer = Trainer(callbacks=[net_disk_stats])

Metrics generated

  • network/recv_MB_per_sec – Received MB/s on all relevant network interfaces as a SUM.
  • network/sent_MB_per_sec – Sent MB/s on all relevant network interfaces as a SUM.
  • disk/util% – Disk utilization percentage as the average of all disk devices.
  • disk/read_MB_per_sec – Disks read MB/s as the sum of all disk devices.
  • disk/writ_MB_per_sec – Disks written MB/s as the sum of all disk devices.
  • disk/io_read_count_per_sec – Disks read I/O operations per second as the sum of all disk devices.
  • disk/io_writ_count_per_sec – Disks written I/O operations per second as the sum of all disk devices.

Screen shot

Pure-Python implementation (zero dependencies)

Quick check

python -c 'from iometrics.example import usage; usage()'

Example output

|        Network (MBytes/s)       | Disk Util |            Disk MBytes          |           Disk I/O          |
|     Received    |     Sent      |     %     |    MB/s Read    |  MB/s Written |     I/O Read    | I/O Write |
|   val  |   avg  |  val  |  avg  | val | avg |  val   |  avg   |  val  |  avg  |   val  |   avg  | val | avg |
| ------:| ------:| -----:| -----:| ---:| ---:| ------:| ------:| -----:| -----:| ------:| ------:| ---:| ---:|
|    4.6 |    3.5 |   0.1 |   0.1 |  49 |   2 |   52.8 |    1.1 |   0.0 |   0.9 |    211 |      4 |   5 |  18 |
|    4.1 |    3.5 |   0.1 |   0.1 |  61 |   3 |   60.4 |    2.4 |  40.3 |   1.7 |    255 |     10 | 149 |  21 |

Full code

import time
from iometrics import NetworkMetrics, DiskMetrics
from iometrics.example import DUAL_METRICS_HEADER
net  = NetworkMetrics()
disk = DiskMetrics()
for i in range(100):
    time.sleep(1)
    net.update_stats()
    disk.update_stats()
    if i % 15 == 0:
        print(DUAL_METRICS_HEADER)
    row = (
        f"| {net.mb_recv_ps.val:6.1f} | {net.mb_recv_ps.avg:6.1f} "
        f"| {net.mb_sent_ps.val:5.1f} | {net.mb_sent_ps.avg:5.1f} "
        f"| {int(disk.io_util.val):3d} | {int(disk.io_util.avg):3d} "
        f"| {disk.mb_read.val:6.1f} | {disk.mb_read.avg:6.1f} "
        f"| {disk.mb_writ.val:5.1f} | {disk.mb_writ.avg:5.1f} "
        f"| {int(disk.io_read.val):4d} | {int(disk.io_read.avg):4d} "
        f"| {int(disk.io_writ.val):3d} | {int(disk.io_writ.avg):3d} "
        f"|"
    )
    print(row)

Contributing

See CONTRIBUTING.md

You might also like...
loghandler allows you to easily log messages to multiple endpoints.

loghandler loghandler allows you to easily log messages to multiple endpoints. Using Install loghandler via pip pip install loghandler In your code im

A simple package that allows you to save inputs & outputs as .log files

wolf_dot_log A simple package that allows you to save inputs & outputs as .log files pip install wolf_dot_log pip3 install wolf_dot_log |Instructions|

This is a wonderful simple python tool used to store the keyboard log.

Keylogger This is a wonderful simple python tool used to store the keyboard log. Record your keys. It will capture passwords and credentials in a comp

Python script to scan log files/system for unauthorized access around system

checkLogs Python script to scan log files/system for unauthorized access around Linux systems Table of contents General info Getting started Usage Gen

ClusterMonitor - a very simple python script which monitors and records the CPU and RAM consumption of submitted cluster jobs
ClusterMonitor - a very simple python script which monitors and records the CPU and RAM consumption of submitted cluster jobs

ClusterMonitor A very simple python script which monitors and records the CPU and RAM consumption of submitted cluster jobs. Usage To start recording

Robust and effective logging for Python 2 and 3.
Robust and effective logging for Python 2 and 3.

Robust and effective logging for Python 2 and 3.

Soda SQL Data testing, monitoring and profiling for SQL accessible data.

Soda SQL Data testing, monitoring and profiling for SQL accessible data. What does Soda SQL do? Soda SQL allows you to Stop your pipeline when bad dat

A Fast, Extensible Progress Bar for Python and CLI
A Fast, Extensible Progress Bar for Python and CLI

tqdm tqdm derives from the Arabic word taqaddum (تقدّم) which can mean "progress," and is an abbreviation for "I love you so much" in Spanish (te quie

Rich is a Python library for rich text and beautiful formatting in the terminal.
Rich is a Python library for rich text and beautiful formatting in the terminal.

Rich 中文 readme • lengua española readme • Läs på svenska Rich is a Python library for rich text and beautiful formatting in the terminal. The Rich API

Comments
  • Add Docker Support

    Add Docker Support

    Change Summary

    Add Docker Support

    Related issue number

    https://github.com/elgalu/iometrics/issues/3

    Checklist

    • [ ] Unit tests for the changes exist
    • [ ] Tests pass on CI and coverage is maintained
    • [x] Documentation reflects the changes where applicable
    opened by m0rt3z4gh 0
  • Not working properly in containers (Docker)

    Not working properly in containers (Docker)

    Network metrics properly reported in the Host:

    |        Network (MBytes/s)       | Disk Util |            Disk MBytes          |             Disk I/O            |
    |     Received    |     Sent      |     %     |    MB/s Read    |  MB/s Written |     I/O Read    |   I/O Write   |
    |   val  |   avg  |  val  |  avg  | val | avg |  val   |  avg   |  val  |  avg  |   val  |   avg  |  val  |  avg  |
    | ------:| ------:| -----:| -----:| ---:| ---:| ------:| ------:| -----:| -----:| ------:| ------:| -----:| -----:|
    |  417.2 |  417.2 |   3.0 |   3.0 |   8 |   8 |    0.9 |    0.9 | 370.5 | 370.5 |    220 |    220 | 11897 | 11897 |
    |  335.1 |  376.1 |   2.4 |   2.7 |  10 |   9 |    1.4 |    1.1 | 275.5 | 323.0 |    335 |    278 |  9168 | 10532 |
    |  434.2 |  395.5 |   3.2 |   2.9 |   9 |   9 |    3.9 |    2.1 | 169.9 | 272.0 |    653 |    403 | 11450 | 10838 |
    

    Network metrics are not visible in the container Disclaimer: the timeline extracts are not perfectly synchronized

    |        Network (MBytes/s)       | Disk Util |            Disk MBytes          |             Disk I/O            |
    |     Received    |     Sent      |     %     |    MB/s Read    |  MB/s Written |     I/O Read    |   I/O Write   |
    |   val  |   avg  |  val  |  avg  | val | avg |  val   |  avg   |  val  |  avg  |   val  |   avg  |  val  |  avg  |
    | ------:| ------:| -----:| -----:| ---:| ---:| ------:| ------:| -----:| -----:| ------:| ------:| -----:| -----:|
    |    0.0 |    0.0 |   0.0 |   0.0 |   7 |   9 |    0.9 |    1.3 |  82.4 | 556.4 |    216 |    310 | 20093 |  9335 |
    |    0.0 |    0.0 |   0.0 |   0.0 |   9 |   9 |   24.9 |    2.0 | 375.4 | 550.8 |    783 |    325 |  3604 |  9156 |
    |    0.0 |    0.0 |   0.0 |   0.0 |  10 |   9 |   60.5 |    3.8 | 103.9 | 537.2 |   1315 |    355 | 12689 |  9263 |
    

    To reproduce

    1. generate network I/O outside the container (in the host)
    2. start a container without any special flags, without any admin or additional capabilities

    TODO

    Find a fix with the least possible privileges

    bug 
    opened by elgalu 0
  • Features/azdo pipeline

    Features/azdo pipeline

    Change Summary

    Related issue number

    Checklist

    • [ ] Unit tests for the changes exist
    • [ ] Tests pass on CI and coverage is maintained
    • [ ] Documentation reflects the changes where applicable
    opened by ugurgural 0
Releases(0.0.8)
Owner
Leo Gallucci
Leo is the creator of docker-selenium which aims to get Selenium running as simple and fast as possible
Leo Gallucci
GTK and Python based, system performance and usage monitoring tool

System Monitoring Center GTK3 and Python 3 based, system performance and usage monitoring tool. Features: Detailed system performance and usage usage

Hakan Dündar 649 Jan 03, 2023
Json Formatter for the standard python logger

This library is provided to allow standard python logging to output log data as json objects. With JSON we can make our logs more readable by machines and we can stop writing custom parsers for syslo

Zakaria Zajac 1.4k Jan 04, 2023
loghandler allows you to easily log messages to multiple endpoints.

loghandler loghandler allows you to easily log messages to multiple endpoints. Using Install loghandler via pip pip install loghandler In your code im

Mathias V. Nielsen 2 Dec 04, 2021
A demo of Prometheus+Grafana for monitoring an ML model served with FastAPI.

ml-monitoring Jeremy Jordan This repository provides an example setup for monitoring an ML system deployed on Kubernetes.

Jeremy Jordan 176 Jan 01, 2023
Soda SQL Data testing, monitoring and profiling for SQL accessible data.

Soda SQL Data testing, monitoring and profiling for SQL accessible data. What does Soda SQL do? Soda SQL allows you to Stop your pipeline when bad dat

Soda Data Monitoring 51 Jan 01, 2023
Colored terminal output for Python's logging module

coloredlogs: Colored terminal output for Python's logging module The coloredlogs package enables colored terminal output for Python's logging module.

Peter Odding 496 Dec 30, 2022
A new kind of Progress Bar, with real time throughput, eta and very cool animations!

alive-progress :) A new kind of Progress Bar, with real-time throughput, eta and very cool animations! Ever found yourself in a remote ssh session, do

Rogério Sampaio de Almeida 4k Dec 30, 2022
A simple, transparent, open-source key logger, written in Python, for tracking your own key-usage statistics.

A simple, transparent, open-source key logger, written in Python, for tracking your own key-usage statistics, originally intended for keyboard layout optimization.

Ga68 56 Jan 03, 2023
A python library used to interact with webots robocup game web logs

A python library used to interact with webots robocup game web logs

Hamburg Bit-Bots 2 Nov 05, 2021
Ransomware leak site monitoring

RansomWatch RansomWatch is a ransomware leak site monitoring tool. It will scrape all of the entries on various ransomware leak sites, store the data

Zander Work 278 Dec 31, 2022
This is a DemoCode for parsing through large log files and triggering an email whenever there's an error.

LogFileParserDemoCode This is a DemoCode for parsing through large log files and triggering an email whenever there's an error. There are a total of f

2 Jan 06, 2022
Debugging-friendly exceptions for Python

Better tracebacks This is a more helpful version of Python's built-in exception message: It shows more code context and the current values of nearby v

Clemens Korndörfer 1.2k Dec 28, 2022
Vibrating-perimeter - Simple helper mod that logs how fast you are mining together with a simple buttplug.io script to control a vibrator

Vibrating Perimeter This project consists of a small minecraft helper mod that writes too a log file and a script that reads said log. Currently it on

Heart[BOT] 0 Nov 20, 2022
A very basic esp32-based logic analyzer capable of sampling digital signals at up to ~3.2MHz.

A very basic esp32-based logic analyzer capable of sampling digital signals at up to ~3.2MHz.

Davide Della Giustina 43 Dec 27, 2022
Python logging made (stupidly) simple

Loguru is a library which aims to bring enjoyable logging in Python. Did you ever feel lazy about configuring a logger and used print() instead?... I

13.7k Jan 02, 2023
The new Python SDK for Sentry.io

sentry-python - Sentry SDK for Python This is the next line of the Python SDK for Sentry, intended to replace the raven package on PyPI. from sentry_s

Sentry 1.4k Dec 31, 2022
A watchdog and logger to Discord for hosting ScPrime servers.

ScpDog A watchdog and logger to Discord for hosting ScPrime servers. Designed to work on Linux servers. This is only capable of sending the logs from

Keagan Landfried 3 Jan 10, 2022
Ultimate Logger - A Discord bot that logs lots of events in a channel written in python

Ultimate Logger - A Discord bot that logs lots of events in a channel written in python

Luca 2 Mar 27, 2022
The easy way to send notifications

See changelog for recent changes Got an app or service and you want to enable your users to use notifications with their provider of choice? Working o

Or Carmi 2.4k Dec 25, 2022
Fuzzy-logger - Fuzzy project is here Log all your pc's actions Simple and free to use Security of datas !

Fuzzy-logger - ➡️⭐ Fuzzy ⭐ project is here ! ➡️ Log all your pc's actions ! ➡️ Simple and free to use ➡️ Security of datas !

natrix_dev 2 Oct 02, 2022