Visual disk-usage analyser for docker images

Related tags

DevOps Toolspythondockerdisk-usage
Overview

whaler

PyPI versions PyPI versions

What? A command-line tool for visually investigating the disk usage of docker images

Why? Large images are slow to move and expensive to store. They cost developer productivity by lengthening devops tasks and often contain unnecessary data

Who is this for? Primarily for engineers working with images containing Python packages.

User Stories

This tool should allow you to answer questions such as:

  1. Which file types are occupying the most disk space?
  2. Which are my largest Python packages?
  3. What are my unknown causes of high disk usage?

Quick start

pip install whaler

Run against a local directory

➜ whaler .venv
Running bash -c cd .venv && du -a -k
Done. Serving output at http://localhost:8000 (ctrl+c to exit)
Running python3 -m http.server 8000 --directory=_whaler/html

Run against a docker image

The tool will pull the image first if it is not present.

whaler --image='hl:latest' /
Running docker run --rm --entrypoint=du --workdir=/ hl:latest -a -k
Ignoring what seems to be non-fatal error(s):
du: cannot access './proc/1/task/1/fd/4': No such file or directory
du: cannot access './proc/1/task/1/fdinfo/4': No such file or directory
du: cannot access './proc/1/fd/3': No such file or directory
du: cannot access './proc/1/fdinfo/3': No such file or directory


Done. Serving output at http://localhost:8000 (ctrl+c to exit)
Running python3 -m http.server 8000 --directory=_whaler/html

HTML Report

Play with a hosted demo

Limitations

  1. Platform: whaler uses du to gather disk usage data. It must be present in your docker image
  2. Scale: I have tested the web UI with up to 500,000 file system nodes with du output of up to ~100MB.

Alternatives/Complements to this tool:

  1. Whaler can tell you what is taking up space in the final layer of your Docker image, but you may have intermediate layers which are contributing to the image size. For diving through the layers, use dive
    • Related: read up on multi-stage builds to understand how to mitigate the problem of intermediate layers bloating your image.
  2. For investigating disk usage in non-docker directories, Disk Inventory X is a great tool on OS X which I have based whaler on.
  3. GoogleContainerTools/distroless for base images not containing standard linux distro contents

See also

  1. a good blog post on making small python images

Developing

See .github/workflows/test.yml for the development platform and setup.

For UI, see whaler-ui

You might also like...
Lima is an alternative to using Docker Desktop on your Mac.
Lima is an alternative to using Docker Desktop on your Mac.

lima-xbar-plugin Table of Contents Description Installation Dependencies Lima is an alternative to using Docker Desktop on your Mac. Description This

Joe Block 68 Dec 22, 2022
Build Netbox as a Docker container

netbox-docker The Github repository houses the components needed to build Netbox as a Docker container. Images are built using this code and are relea

Farshad Nick 1 Dec 18, 2021
Some automation scripts to setup a deployable development database server (with docker).

Postgres-Docker Database Initializer This is a simple automation script that will create a Docker Postgres database with a custom username, password,

Pysogge 1 Nov 11, 2021
A job launching library for docker, EC2, GCP, etc.

doodad A library for packaging dependencies and launching scripts (with a focus on python) on different platforms using Docker. Currently supported pl

Justin Fu 55 Aug 27, 2022
A repository containing a short tutorial for Docker (with Python).
A repository containing a short tutorial for Docker (with Python).

Docker Tutorial for IFT 6758 Lab In this repository, we examine the advtanges of virtualization, what Docker is and how we can deploy simple programs

Arka Mukherjee 0 Dec 14, 2021
Bitnami Docker Image for Python using snapshots for the system packages repositories

Python Snapshot packaged by Bitnami What is Python Snapshot? Python is a programming language that lets you work quickly and integrate systems more ef

Bitnami 1 Jan 13, 2022
Hatch plugin for Docker containers

hatch-containers CI/CD Package Meta This provides a plugin for Hatch that allows

Ofek Lev 11 Dec 30, 2022
Create pinned requirements.txt inside a Docker image using pip-tools
Create pinned requirements.txt inside a Docker image using pip-tools

Pin your Python dependencies! pin-requirements.py is a script that lets you pin your Python dependencies inside a Docker container. Pinning your depen

null 4 Aug 18, 2022
This repository contains useful docker-swarm-tools.

docker-swarm-tools This repository contains useful docker-swarm-tools. swarm-guardian This Docker image is intended to be used in a multihost docker e

NeuroForge GmbH & Co. KG 4 Jan 12, 2022
Comments
  • Bounding box on selected node barely visible

    Bounding box on selected node barely visible

    This is due to the canvas not being fully re-drawn upon selection.

    The solution is probably to ensure the canvas can be re-drawn quickly and creating a fresh render every time

    bug 
    opened by alex-treebeard 0
Releases(0.1.2)
Owner
Treebeard Technologies
Helping with Data Infra
Treebeard Technologies
GitHub Repository https://treebeardtech.github.io/whaler/
DC/OS - The Datacenter Operating System

DC/OS - The Datacenter Operating System The easiest way to run microservices, big data, and containers in production. What is DC/OS? Like traditional

DC/OS 2.3k Jan 06, 2023
Ganeti is a virtual machine cluster management tool built on top of existing virtualization technologies such as Xen or KVM and other open source software.

Ganeti 3.0 =========== For installation instructions, read the INSTALL and the doc/install.rst files. For a brief introduction, read the ganeti(7) m

395 Jan 04, 2023
Let's learn how to build, release and operate your containerized applications to Amazon ECS and AWS Fargate using AWS Copilot.

🚀 Welcome to AWS Copilot Workshop In this workshop, you'll learn how to build, release and operate your containerised applications to Amazon ECS and

Donnie Prakoso 15 Jul 14, 2022
The low-level, core functionality of boto 3.

botocore A low-level interface to a growing number of Amazon Web Services. The botocore package is the foundation for the AWS CLI as well as boto3. On

the boto project 1.2k Jan 03, 2023
Travis CI testing a Dockerfile based on Palantir's remix of Apache Cassandra, testing IaC, and testing integration health of Debian

Testing Palantir's remix of Apache Cassandra with Snyk & Travis CI This repository is to show Travis CI testing a Dockerfile based on Palantir's remix

Montana Mendy 1 Dec 20, 2021
Let's Git - Version Control & Open Source Homework

Let's Git - Version Control & Open Source Homework Welcome to this homework for our MOOC: Let's Git! We hope you will learn a lot and have fun working

1 Dec 05, 2021
Cado Response Integration with Amazon GuardDuty using AWS Lambda

Cado Response Integration with Amazon GuardDuty using AWS Lambda This repository contains a simple example where: An alert is triggered by GuardDuty T

Cado Security 4 Mar 02, 2022
Create pinned requirements.txt inside a Docker image using pip-tools

Pin your Python dependencies! pin-requirements.py is a script that lets you pin your Python dependencies inside a Docker container. Pinning your depen

4 Aug 18, 2022
Learning and experimenting with Kubernetes

Kubernetes Experiments This repository contains code that I'm using to learn and experiment with Kubernetes. 1. Environment setup minikube kubectl doc

Richard To 10 Dec 02, 2022
Some automation scripts to setup a deployable development database server (with docker).

Postgres-Docker Database Initializer This is a simple automation script that will create a Docker Postgres database with a custom username, password,

Pysogge 1 Nov 11, 2021
A simple python application for running a CI pipeline locally This app currently supports GitLab CI scripts

🏃 Simple Local CI Runner 🏃 A simple python application for running a CI pipeline locally This app currently supports GitLab CI scripts ⚙️ Setup Inst

Tom Stowe 0 Jan 11, 2022
HB Case Study

HB Case Study Envoy Proxy It is a modern Layer7(App) and Layer3(TCP) proxy Incredibly modernized version of reverse proxies like NGINX, HAProxy It is

Ilker Ispir 1 Oct 22, 2021
Azure plugins for Feast (FEAture STore)

Feast on Azure This project provides resources to enable running a feast feature store on Azure. Feast Azure Provider The Feast Azure provider acts li

Microsoft Azure 70 Dec 31, 2022
Convenient tool to manage multiple VMs at once using libvirt

Convenient tool to manage multiple VMs at once using libvirt Installing To install the tool and its dependencies: pip install -e . Getting completion

Cedric Bosdonnat 13 Nov 11, 2022
A repository containing a short tutorial for Docker (with Python).

Docker Tutorial for IFT 6758 Lab In this repository, we examine the advtanges of virtualization, what Docker is and how we can deploy simple programs

Arka Mukherjee 0 Dec 14, 2021
A Python Implementation for Git for learning

A pure Python implementation for Git based on Buliding Git

shidenggui 42 Jul 13, 2022
Chef-like functionality for Fabric

/ / ___ ___ ___ ___ | | )| |___ | | )|___) |__ |__/ | __/ | | / |__ -- Chef-like functionality for Fabric About Fabric i

Sébastien Pierre 1.3k Dec 21, 2022
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Apache Airflow Apache Airflow (or simply Airflow) is a platform to programmatically author, schedule, and monitor workflows. When workflows are define

The Apache Software Foundation 28.6k Jan 01, 2023
Ingress patch example by Kustomize

Ingress patch example by Kustomize

Jinu 10 Nov 14, 2022
A cron monitoring tool written in Python & Django

Healthchecks Healthchecks is a cron job monitoring service. It listens for HTTP requests and email messages ("pings") from your cron jobs and schedule

Healthchecks 5.8k Jan 02, 2023