ioztat is a storage load analysis tool for OpenZFS

Related tags

Miscellaneousioztat
Overview

ioztat

ioztat is a storage load analysis tool for OpenZFS. It provides iostat-like statistics at an individual dataset/zvol level.

The statistics offered are read and write operations per second, read and write throughput per second, and the average size (in kilobytes) of read and write operations issued in the current reporting interval. Viewing these statistics at the individual dataset level allows system administrators to identify storage "hot spots" in larger multi-tenant systems--particularly those with many VMs or containers operating essentially independent workloads.

This sample output shows activity which has taken place in the most recent second, on a the ssd zpool of a ZFS virtualization host:

[email protected]:~# ioztat -y -c1 ssd
dataset                                         w/s      wMB/s        r/s      rMB/s   wareq-sz   rareq-sz
ssd                                            0.00       0.00       0.00       0.00       0.00       0.00
   images                                      0.00       0.00       0.00       0.00       0.00       0.00
      DC1                                     17.96       0.10       3.99       0.05       5.66      12.29
      DC2                                     21.95       0.14       0.00       0.00       6.59       0.00
      QB                                       2.00       0.01       0.00       0.00       7.17       0.00
      SAP-TC                                   2.99       0.02       0.00       0.00       6.83       0.00
      SAP4-WIN2019                             2.99       0.05       0.00       0.00      17.07       0.00
      nagios                                   0.00       0.00       0.00       0.00       0.00       0.00
      qemu                                     0.00       0.00       0.00       0.00       0.00       0.00
         autostart                             0.00       0.00       0.00       0.00       0.00       0.00
   iso                                         0.00       0.00       0.00       0.00       0.00       0.00
   unsnapped                                   0.00       0.00       0.00       0.00       0.00       0.00
      rp9                                      0.00       0.00       0.00       0.00       0.00       0.00

For the most part, ioztat behaves the same way that the system standard iostat tool does, with similar arguments.

usage: ioztat [-h] [-s SORT] [-i INTERVAL] [-c COUNT] [-y] [-b]
              pool [pool ...]

iostat for ZFS datasets

positional arguments:
  pool         ZFS pool

optional arguments:
  -h, --help   show this help message and exit
  -s SORT      Sort by: name / wps / wMBps / rps / rMBps
  -i INTERVAL  Time between each report
  -c COUNT     Number of reports generated
  -y           Skip the initial "summary" report
  -b           Use binary (power-of-two) prefixes

The only required argument is the name of at least one zpool to monitor. Without any other arguments, ioztat first prints a summary record showing activity per dataset since the most recent system boot, then prints a new record showing the most recent activity once per second. The -i argument can be used to change the report interval, and the -c argument can be used to limit ioztat to a certain number of intervals before exiting.

For those who wish a continually-updated, easy to read summary of pool activity, watch -n1 ioztat poolname -c1 -y will suit nicely.

Comments
  • Rework formatting

    Rework formatting

    This makes ioztat output more closely resemble zpool iostat:

                                  operations      bandwidth         opsize
    dataset                       read   write    read   write    read   write
    --------------------------  ------  ------  ------  ------  ------  ------
    rpool
       ROOT
          default                   23       0   62.6K       0    2.7K     759
       cvs                           0       0    1.6K     892   24.2K  108.4K
       home                          0       0       0       0       0       0
          freaky                     3       2   29.7K    8.3K   10.2K    3.9K
    

    This also adds -e for exact values (-p in zpool iostat, which we're currently using elsewhere) and -H for scripted mode that drops the header and separates fields by tabs.

    The intent is for this to eventually fix #23, probably by adjusting the space for dataset.

    opened by Freaky 12
  • Painting the bikeshed

    Painting the bikeshed

    Should bandwidth be changed to throughput? This is more correct, but deviates from zpool-iostat(8).

    Should we default to binary mode, since that's all zpool-iostat(8) supports?

    Should we default to a single iteration? This is what iostat(1) and zpool-iostat(8) both do. They also accept interval and count as positional arguments.

    Is + as an ellipsis for truncated dataset names reasonable, or should it be ... despite the additional space it uses?

    opened by Freaky 8
  • Overwrite old reports in terminal by default

    Overwrite old reports in terminal by default

    I thought that this script put a little too much into the terminal. This PR makes it so that only the most recent report is visible, which can be disabled with the new -n argument.

    opened by Aluminite 4
  • No warning if zfs version too low (was: doesn't find datasets on Debian Buster, zfs 0.7.12-2)

    No warning if zfs version too low (was: doesn't find datasets on Debian Buster, zfs 0.7.12-2)

    ioztat -Hey as suggested by the man page doesn't return anything.

    ioztat -SIn rpool, also suggested by the man page, gives "dataset not found: 'rpool'" (rpool of course exists).

    Is there a way to further debug this? Am I missing a dependency?

    opened by steffen-AEI 3
  • Make numeric precision variable

    Make numeric precision variable

    Matching ZFS number formatting behaviour, use between 0 and 2 decimal places according to available space, and further reduce the column size to again match zpool-iostat and make more room for dataset names.

    This includes the careful rounding behaviour of the previous implementation, which I believe ZFS lacks.

    opened by Freaky 3
  • Fix -y when not combined with -c

    Fix -y when not combined with -c

    Fix an error when args.count is None. While I'm here also fix a count of zero that was ignored due to zero being falsey.

    Simplify the logic slightly by changing skipsummary from a bool to a generic skip count.

    opened by Freaky 3
  • Output is insensitive to terminal width

    Output is insensitive to terminal width

    The current output width is 106 columns, which by happy coincidence is precisely the width of my usual terminals.

    This coincidence seems unlikely to be universal, and some sensitivity to this would seem to be in order.

    opened by Freaky 3
  • Fix and simplify overwrite mode

    Fix and simplify overwrite mode

    This fixes overwrite mode by replacing the line counting and iterated cursor repositioning with a clear screen and placing the cursor at 1,1 - exactly what GNU watch does.

    I've also done some tidying, hoisting some iteration and filtering logic into dedicated generator functions which helps to further simplify the main loop.

    opened by Freaky 3
  • python3 shebang--does this work in FreeBSD?

    python3 shebang--does this work in FreeBSD?

    @Freaky, I discovered after merging your patches today that ioztat does not work on the Python2.7 that my Ubuntu workstation at the home office provides with "env python". I had to update the shebang to "env python3" to get it to work again.

    This should be fine for Ubuntu, but does it work in FreeBSD-land as well? I don't know if there's a python3 executable in your environment, or whether there is only python and you get what you get.

    opened by jimsalterjrs 3
  • Format name with all intermediates for disambiguation

    Format name with all intermediates for disambiguation

    Fixes #5

    This replaces the "just-the-rightmost-dataset-name-component" logic by "components-of-the-name-unique-after-the-previous-one". Output with default options looks like

    dataset                                         w/s      wMB/s        r/s      rMB/s   wareq-sz   rareq-sz
    zroot                                          0.00       0.00       0.00       0.00       0.00       0.00
       ROOT/default                                0.00       0.00       3.88       0.00       0.00       0.16
       tmp                                         0.00       0.00       0.00       0.00       0.00       0.00
    

    (e.g. here instead of default being indented two, and ROOT being invisible, we get the dataset-name-components that distinguish it from zroot, the previously-printed name)

    opened by adriaandegroot 3
  • Unmounted datasets lead to confusing gaps in output

    Unmounted datasets lead to confusing gaps in output

    Stats for unmounted datasets aren't exported, so you can end up with incomplete hierarchies in ioztat output. For example here rpool and rpool/ROOT aren't mounted, leaving rpool/ROOT/default without the path leading up to it:

    dataset                                         w/s      wMB/s        r/s      rMB/s   wareq-sz   rareq-sz
          default                                  0.00       0.00       0.97       0.00       0.00       0.05
    

    Similarly, rpool/usr/local/poudriere/ports/default misses the mountpoint-less poudriere/ports segments:

       usr                                         0.00       0.00       0.97       0.00       0.00       0.01
          local                                    0.00       0.00       0.00       0.00       0.00       0.00
                   default                         0.00       0.00       0.00       0.00       0.00       0.00
    
    opened by Freaky 3
  • using dataclasses

    using dataclasses

    Hi,

    I started of my own version of this, but found yours basically the last moment … My name of choice was objset, but the poll is finished I guess.

    I used dataclasses - ported here. And updated the fields to match current OpenZFS git.

    opened by commonism 7
  • no activity reported for zfs replication

    no activity reported for zfs replication

    is it expected that ioztat does not seem to report activity for zfs replication ?

    if this is not a bug but by design, maybe it's worth mentioning in the

    opened by devZer0 0
  • Packages

    Packages

    Produce .deb/.rpm/etc packages. sanoid is already doing this. I'm currently using sanoid package in latest ubuntu release.

    Getting these included will take some time but is worthwhile I think.

    enhancement help wanted 
    opened by morganchristiansson 2
Releases(v2.0.1)
  • v2.0.1(Mar 4, 2022)

  • v2.0.0(Mar 3, 2022)

    Added

    • An iostat(8) man page
    • CHANGELOG.md, this Changelog
    • -e and -H flags for use in scripting [#26]
    • -I flag for totals since last update instead of per-second
    • -o flag to overwrite prior reports [#9] [#20]
    • -P and -p flags to override dataset name display [#19]
    • -S flag for including child dataset statistics in parents [#32]
    • -T d|u flag for adding a timestamp to each report [#25]
    • -V, --version flags [#13] [#14]
    • -x flag for extended statistics, including unlink counts under -xx [#33]
    • interval and count positional arguments [#31]

    Changed

    • Shebang line is now /usr/bin/env python3 [#18]
    • Formatting has changed from resembling GNU iostat(8) to ZFS zpool-iostat(8) [#26]
    • Header is now printed periodically on a tty unless -N is specified [#26]
    • Display is clamped to the terminal width by truncating dataset names if necessary [#26]
    • Sort field names have been changed, with fallbacks for compatibility [#26]
    • Sort options are now case-insensitive [#26]
    • count now defaults to 1 unless an interval is specified [#31]
    • dataset is now an optional argument [#20]
    • Binary (1024-based) formatting is now default, with new -D flag for decimal
    • Average I/O sizes are now hidden beyind -x flag by default to reduce clutter [#33]
    • Exit with an error if a requested dataset does not exist or is not mounted

    Removed

    • -b flag. Binary mode is now the default to match other iostat tools.

    Fixed

    • count and interval must now be positive [#31]
    • WIFSIGNALED() status is now propagated properly to caller
    • Sleep interval now adjusts to compensate for runtime
    • Unhandled BrokenPipeError exception on SIGPIPE
    • Unhandled FileNotFoundError exception on Linux if a dataset is destroyed while enumerating files in /proc
    • Unhandled CalledProcessError exception on FreeBSD if a requested pool does not exist when using sysctl(8) fallback
    Source code(tar.gz)
    Source code(zip)
  • v1.1.0(Jan 20, 2022)

    Thanks to the efforts of @Freaky and @adriaandegroot, ioztat has new features, improved output readability, and now works on FreeBSD as well as Linux!

    New features include initial selection at the dataset level rather than pool level, optional suppression of zero-activity datasets with -z, and optional suppression of recursion into child datasets with -n.

    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(Jan 12, 2022)

    The first release of ioztat builds on efforts from the Reddit r/zfs community, including:

    u/55rzs (initial creation) u/d1722825 (substantial refactoring and cleanup) u/mercenary_sysadmin (addition of the -y flag to allow for easy use with the GNU watch command)

    Enjoy!

    Source code(tar.gz)
    Source code(zip)
Owner
Jim Salter
Jim Salter
A nonebot2 plugin, send news information in a picture form.

A nonebot2 plugin, send news information in a picture form.

幼稚园园长 7 Nov 18, 2022
A promo calculator for sports betting odds.

Sportbetter Calculation Toolkit Parlay Calculator This is a quick parlay calculator that considers some of the common promos offered. It is used to id

Luke Bhan 1 Sep 08, 2022
A TODO-list tool written in Python

PyTD A TODO-list tool written in Python. Its goal is to provide a stable posibility to get a good view over all your TODOs motivate you to actually fi

1 Feb 12, 2022
TinyBar - Tiny MacOS menu bar utility to track price dynamics for assets on TinyMan.org

📃 About A simple MacOS menu bar app to display current coins from most popular Liquidity Pools on TinyMan.org

Al 8 Dec 23, 2022
JPMC Virtual Experience

This repository contains the submitted patch files along with raw files of the various tasks assigned by JPMorgan Chase & Co. through its Software Engineering Virtual Experience Program on Forage (fo

Vardhini K 1 Dec 05, 2021
AKSWINPOSTINIT -- AKS Windows node post provisioning initialization

AKSWINPOSTINIT -- AKS Windows node post provisioning initialization Features This is a tool that provides one-time powershell script initilization for

Ping He 3 Nov 25, 2021
Python script for changing the SSH banner content with other content

Banner-changer-py Python script for changing the SSH banner content with other content. The Script will take the content of a specified file range and

2 Nov 23, 2021
Python project that aims to discover CDP neighbors and map their Layer-2 topology within a shareable medium like Visio or Draw.io.

Python project that aims to discover CDP neighbors and map their Layer-2 topology within a shareable medium like Visio or Draw.io.

3 Feb 11, 2022
An a simple sistem code in python

AMS OS An a simple code in python ⁕¿What is AMS OS? AMS OS is an a simple sistem code writed in python. This code helps you with the cotidian task, yo

1 Nov 10, 2021
All kinds of programs are accepted here, raise a genuine PR, and claim a PR, Make 4 successful PR's and get the Stickers and T-Shirt from hacktoberfest 2021

this repository is excluded from hacktoberfest Hacktoberfest-2021 This repository aims to help code beginners with their first successful pull request

34 Sep 11, 2022
Python Repository for Bachelor Ski Sign.

BachelorSkiSign Python Repository for Bachelor Ski Sign. This application reads data from https://bachelorapi.azurewebsites.net/ It is written in Ciru

Winston 1 Jan 04, 2022
Org agenda in the console

This Python script reads an org agenda file (i.e. a regular org file with some active dates) and displays an interactive and colored year calendar with detailed information for each day when the mous

Nicolas P. Rougier 113 Jan 03, 2023
[CVPR 2020] Rethinking Class-Balanced Methods for Long-Tailed Visual Recognition from a Domain Adaptation Perspective

Rethinking Class-Balanced Methods for Long-Tailed Visual Recognition from a Domain Adaptation Perspective [Arxiv] This is PyTorch implementation of th

Abdullah Jamal 22 Nov 19, 2022
Repositório de código de curso de Djavue ministrado na Python Brasil 2021

djavue-python-brasil Repositório de código de curso de Djavue ministrado na Python Brasil 2021 Completamente baseado no curso Djavue. A diferença está

Buser 15 Dec 26, 2022
CaskDB is a disk-based, embedded, persistent, key-value store based on the Riak's bitcask paper, written in Python.

CaskDB - Disk based Log Structured Hash Table Store CaskDB is a disk-based, embedded, persistent, key-value store based on the Riak's bitcask paper, w

886 Dec 27, 2022
Free Data Engineering course!

Data Engineering Zoomcamp Register in DataTalks.Club's Slack Join the #course-data-engineering channel The videos are published to DataTalks.Club's Yo

DataTalksClub 7.3k Dec 30, 2022
Animation retargeting tool for Autodesk Maya. Retargets mocap to a custom rig with a few clicks.

Animation Retargeting Tool for Maya A tool for transferring animation data between rigs or transfer raw mocap from a skeleton to a custom rig. (The sc

Joaen 62 Dec 19, 2022
OTP-Bomber - An otp from MPL ID app, which can be spammed

OTP-Bomber An otp from MPL ID app, which can be spammed Note: Only available on

5 Oct 29, 2022
Active Transport Analytics Model: A new strategic transport modelling and data visualization framework

{ATAM} Active Transport Analytics Model Active Transport Analytics Model (“ATAM”

ATAM Analytics 2 Dec 21, 2022
The presented desktop application was made to solve 1d schrodinger eqation

schrodinger_equation_1d_solver The presented desktop application was made to solve 1d schrodinger eqation. It implements Numerov's algorithm (step by

Artem Kashapov 2 Dec 29, 2021