A small example project for efficiently configuring a Python application with YAMLs and the CLI.
A frequent requirement for productive Python application is that they are configurable via configuration files and/or the command-line-interface (CLI). This allows you to change the behavior of your application without touching the source code, e.g. configuring another database URL or the logging verbosity. For the CLI-part, [argparse] or Click is often used and with PyYAML configuration files can be easily read, so where is the problem?
The CLI and configuration file of a Python application have many things in common, i.e., both
- configure the runtime behaviour of your application,
- need to implement validations, e.g. is the port an integer above 1024,
- need to be consistent and mergeable, i.e. a CLI flag should be named like the YAML key and if both are passed the CLI overwrites the YAML configuration.
Thus implementing the CLI and configuration by a YAML file seperated from each other, leads often to code duplication and inconsistent behavior, not to mention the enormous amount of work that must be done to get this right.
With this in mind, Facebook implemented the Hydra library, which allows you to do hierarchical configuration by composition and override it through config files and the command-line. This repository serves as an example project set up to demonstrate the most important features. We also show how Hydra can be used in conjunction with pydantic, which extends the validation capabilities of OmegaConf that is used internally by Hydra.
Read this blog post if you want to know more.
If you want to play around with this project, just clone it and set up the necessary environment:
- create an environment
hydra-example-projectwith the help of conda:conda env create -f environment.yml - activate the new environment with:
conda activate hydra-example-project
and you are all set to run hydra-test.
Glad you asked. This project was set up using the miraculous PyScaffold with the dsproject extension. The setup was as simple as:
putup -f hydra-example-project --dsproject -p my_pkgIn order to have the CLI command hydra-test, we changed in setup.cfg the following lines:
# Add here console scripts like:
console_scripts =
hydra-test = my_pkg.cli:mainRemember to run pip install -e . after any changes to setup.cfg.
├── AUTHORS.md <- List of developers and maintainers.
├── CHANGELOG.md <- Changelog to keep track of new features and fixes.
├── CONTRIBUTING.md <- Guidelines for contributing to this project.
├── Dockerfile <- Build a docker container with `docker build .`.
├── LICENSE.txt <- License as chosen on the command-line.
├── README.md <- The top-level README for developers.
├── configs <- Directory for configurations of model & application.
├── data
│ ├── external <- Data from third party sources.
│ ├── interim <- Intermediate data that has been transformed.
│ ├── processed <- The final, canonical data sets for modeling.
│ └── raw <- The original, immutable data dump.
├── docs <- Directory for Sphinx documentation in rst or md.
├── environment.yml <- The conda environment file for reproducibility.
├── models <- Trained and serialized models, model predictions,
│ or model summaries.
├── notebooks <- Jupyter notebooks. Naming convention is a number (for
│ ordering), the creator's initials and a description,
│ e.g. `1.0-fw-initial-data-exploration`.
├── pyproject.toml <- Build configuration. Don't change! Use `pip install -e .`
│ to install for development or to build `tox -e build`.
├── references <- Data dictionaries, manuals, and all other materials.
├── reports <- Generated analysis as HTML, PDF, LaTeX, etc.
│ └── figures <- Generated plots and figures for reports.
├── scripts <- Analysis and production scripts which import the
│ actual PYTHON_PKG, e.g. train_model.
├── setup.cfg <- Declarative configuration of your project.
├── setup.py <- [DEPRECATED] Use `python setup.py develop` to install for
│ development or `python setup.py bdist_wheel` to build.
├── src
│ └── my_pkg <- Actual Python package where the main functionality goes.
├── tests <- Unit tests which can be run with `pytest`.
├── .coveragerc <- Configuration for coverage reports of unit tests.
├── .isort.cfg <- Configuration for git hook that sorts imports.
└── .pre-commit-config.yaml <- Configuration of pre-commit git hooks.
This project has been set up using PyScaffold 4.1.4 and the dsproject extension 0.7.