DeepOBS: A Deep Learning Optimizer Benchmark Suite

Related tags

Deep LearningDeepOBS
Overview

DeepOBS - A Deep Learning Optimizer Benchmark Suite

DeepOBS

PyPI version Documentation Status License: MIT

DeepOBS is a benchmarking suite that drastically simplifies, automates and improves the evaluation of deep learning optimizers.

It can evaluate the performance of new optimizers on a variety of real-world test problems and automatically compare them with realistic baselines.

DeepOBS automates several steps when benchmarking deep learning optimizers:

  • Downloading and preparing data sets.
  • Setting up test problems consisting of contemporary data sets and realistic deep learning architectures.
  • Running the optimizers on multiple test problems and logging relevant metrics.
  • Reporting and visualization the results of the optimizer benchmark.

DeepOBS Output

This branch contains the beta of version 1.2.0 with TensorFlow and PyTorch support. It is currently in a pre-release state. Not all features are implemented and most notably we currently don't provide baselines for this version.

The full documentation of this beta version is available on readthedocs: https://deepobs-with-pytorch.readthedocs.io/

The paper describing DeepOBS has been accepted for ICLR 2019 and can be found here: https://openreview.net/forum?id=rJg6ssC5Y7

If you find any bugs in DeepOBS, or find it hard to use, please let us know. We are always interested in feedback and ways to improve DeepOBS.

Installation

pip install -e git+https://github.com/fsschneider/[email protected]#egg=DeepOBS

We tested the package with Python 3.6, TensorFlow version 1.12, Torch version 1.1.0 and Torchvision version 0.3.0. Other versions might work, and we plan to expand compatibility in the future.

Further tutorials and a suggested protocol for benchmarking deep learning optimizers can be found on https://deepobs-with-pytorch.readthedocs.io/

Comments
  • Request: Share the hyper-parameters found in the grid search

    Request: Share the hyper-parameters found in the grid search

    To lessen the burden of re-running the benchmark, would it be possible to publish the optimal hyper-parameters somewhere?

    By-reusing those hyper-parameters, one would avoid the most computationally-demanding part of reproducing the results (by 1-2 orders of magnitude).

    opened by jotaf98 2
  • Add functionality to skip existing runs, plotting modes, some refactoring

    Add functionality to skip existing runs, plotting modes, some refactoring

    • Adding parameter skip_if_exists to runner.run
      • Default value is set such that the current behavior is maintained
      • By setting to True, runs that already have a .json output file will not be executed again
    • Possible extensions
      • Make skip_if_exists arg-parsable
    opened by f-dangel 2
  • KeyError: 'optimizer_hyperparams'

    KeyError: 'optimizer_hyperparams'

    (Apologies for creating multiple issues in a row -- it seemed more clean to keep them separate.)

    I downloaded the data from DeepOBS_Baselines, and attempted to run example_analyze_pytorch.py. Unfortunately DeepOBS seems to look for keys in the JSON files that don't exist:

    $ python example_analyze_pytorch.py
    /users/user/Research/deepobs/deepobs/analyzer/shared_utils.py:144: RuntimeWarning: Metric valid_accu
    racies does not exist for testproblem quadratic_deep. We now use fallback metric valid_losses
      default_metric), RuntimeWarning)
    /users/user/Research/deepobs/deepobs/analyzer/shared_utils.py:229: RuntimeWarning: All settings for
    /scratch/local/ssd/user/data/deepobs/quadratic_deep/SGD on test problem quadratic_deep have the same
     number of seeds runs. Mode 'most' does not make sense and we use the fallback mode 'final'
      .format(optimizer_path, testproblem_name), RuntimeWarning)
    {'Performance': 127.96759578159877, 'Speed': 'N.A.', 'Hyperparameters': {'lr': 0.01, 'momentum': 0.9
    9, 'nesterov': False}, 'Training Parameters': {}}
    /users/user/Research/deepobs/deepobs/analyzer/shared_utils.py:144: RuntimeWarning: Metric valid_accu
    racies does not exist for testproblem quadratic_deep. We now use fallback metric valid_losses
      default_metric), RuntimeWarning)
    /users/user/Research/deepobs/deepobs/analyzer/shared_utils.py:229: RuntimeWarning: All settings for
    /scratch/local/ssd/user/data/deepobs/quadratic_deep/SGD on test problem quadratic_deep have the same
     number of seeds runs. Mode 'most' does not make sense and we use the fallback mode 'final'
      .format(optimizer_path, testproblem_name), RuntimeWarning)
    /users/user/Research/deepobs/deepobs/analyzer/shared_utils.py:150: RuntimeWarning: Cannot fallback t
    o metric valid_losses for optimizer MomentumOptimizer on testproblem quadratic_deep. Will now fallba
    ck to metric test_losses
      testproblem_name), RuntimeWarning)
    /users/user/miniconda3/lib/python3.7/site-packages/numpy/core/_methods.py:193: RuntimeWarning: inva$
    id value encountered in subtract
      x = asanyarray(arr - arrmean)
    /users/user/miniconda3/lib/python3.7/site-packages/numpy/lib/function_base.py:3949: RuntimeWarning:
    invalid value encountered in multiply
      x2 = take(ap, indices_above, axis=axis) * weights_above
    Traceback (most recent call last):
      File "example_analyze_pytorch.py", line 17, in <module>
        analyzer.plot_optimizer_performance(result_path, reference_path=base + '/deepobs/baselines/quad$
    atic_deep/MomentumOptimizer')
      File "/users/user/Research/deepobs/deepobs/analyzer/analyze.py", line 514, in plot_optimizer_perfo
    rmance
        which=which)
      File "/users/user/Research/deepobs/deepobs/analyzer/analyze.py", line 462, in _plot_optimizer_perf
    ormance
        optimizer_path, mode, metric)
      File "/users/user/Research/deepobs/deepobs/analyzer/shared_utils.py", line 206, in create_setting_
    analyzer_ranking
        setting_analyzers = _get_all_setting_analyzer(optimizer_path)
      File "/users/user/Research/deepobs/deepobs/analyzer/shared_utils.py", line 184, in _get_all_settin
    g_analyzer
        setting_analyzers.append(SettingAnalyzer(sett_path))
      File "/users/user/Research/deepobs/deepobs/analyzer/shared_utils.py", line 260, in __init__
        self.aggregate = aggregate_runs(path)
      File "/users/user/Research/deepobs/deepobs/analyzer/shared_utils.py", line 101, in aggregate_runs
        aggregate['optimizer_hyperparams'] = json_data['optimizer_hyperparams']
    KeyError: 'optimizer_hyperparams'
    

    One of the JSON files in question looks like this (data points snipped for brevity):

    {
    "train_losses": [353.9337594168527, 347.5994306291853, 331.35902622767856, 307.2468915666853, ... 97.28871154785156, 91.45470428466797, 96.45774841308594, 86.27237701416016],
    "optimizer": "MomentumOptimizer",
    "testproblem": "quadratic_deep",
    "weight_decay": null,
    "batch_size": 128,
    "num_epochs": 100,
    "learning_rate": 1e-05,
    "lr_sched_epochs": null,
    "lr_sched_factors": null,
    "random_seed": 42,
    "train_log_interval": 1,
    "hyperparams": {"momentum": 0.99, "use_nesterov": false}
    }
    

    The obvious key seems to be hyperparams as opposed to optimizer_hyperparams; this occurs only for some JSON files.

    Edit: Having fixed this, there is a further key error on training_params. Perhaps these were generated with different versions of the package.

    opened by jotaf98 3
  • Installation error / unmentioned dependency

    Installation error / unmentioned dependency "bayes_opt"

    Attempting to install by following the documentation's instructions, after installing all the mentioned dependencies with conda, results in the following error:

    (base) [email protected]:~$ pip install -e git+https://github.com/abahde/[email protected]#egg=DeepOBS
    Obtaining DeepOBS from git+https://github.com/abahde/[email protected]#egg=DeepOBS
      Cloning https://github.com/abahde/DeepOBS.git (to revision master) to ./src/deepobs
      Running command git clone -q https://github.com/abahde/DeepOBS.git /users/user/src/deepobs
        ERROR: Complete output from command python setup.py egg_info:
        ERROR: Traceback (most recent call last):
          File "<string>", line 1, in <module>
          File "/users/user/src/deepobs/setup.py", line 5, in <module>
            from deepobs import __version__
          File "/users/user/src/deepobs/deepobs/__init__.py", line 5, in <module>
            from . import analyzer
          File "/users/user/src/deepobs/deepobs/analyzer/__init__.py", line 2, in <module>
            from . import analyze
          File "/users/user/src/deepobs/deepobs/analyzer/analyze.py", line 12, in <module>
            from ..tuner.tuner_utils import generate_tuning_summary
          File "/users/user/src/deepobs/deepobs/tuner/__init__.py", line 4, in <module>
            from .bayesian import GP
          File "/users/user/src/deepobs/deepobs/tuner/bayesian.py", line 3, in <module>
            from bayes_opt import UtilityFunction
        ModuleNotFoundError: No module named 'bayes_opt'
        ----------------------------------------
    ERROR: Command "python setup.py egg_info" failed with error code 1 in /users/user/src/deepobs/
    

    Is this bayes_opt package really necessary? It seems a bit tangential to the package's purpose (or at most optional).

    Edit: It turns out that bayesian-optimization has relatively few requirements so this is not a big issue; perhaps just the docs need updating.

    As an aside, it might be possible to suggest a single conda command that installs everything: conda install -c conda-forge seaborn matplotlib2tikz bayesian-optimization.

    opened by jotaf98 0
  • Wall-clock time plots

    Wall-clock time plots

    Optimizers can have very different runtimes per iteration, especially 2nd-order ones.

    This means that sometimes, despite promises of "faster" convergence, the wall-clock time taken to converge is disappointingly larger.

    Is there any chance DeepOBS could implement wall-clock time plots, in addition to per-epoch ones? (E.g. X axis in minutes or hours.)

    opened by jotaf98 4
  • Improve estimate_runtime()

    Improve estimate_runtime()

    There are a couple of improvements that I suggest:

    • [ ] Return the results not as a string, but as a dict or an object.
    • [ ] (Maybe, think about that) Include the ability to test multiple optimizers simultaneously.
    • [ ] Report standard deviation and individual runtimes for SGD.
    • [ ] Add a function that generates a figure, similar to https://github.com/ludwigbald/probprec/blob/master/code/exp_perf_prec/analyze.py
    opened by ludwigbald 0
  • Implement validation set split also for TensorFlow

    Implement validation set split also for TensorFlow

    In PyTorch we split the validation set from the training set randomly. It has the size of the test set. The validation performance is used by the tuner and analyzer to obtain the best instance. This split should be implemented in the TensorFlow data sets as well. We have already prepared the test problem and the runner implementations for this change. The only change that needs to be done to the runner is marked in the code with a ToDo flag.

    bug enhancement 
    opened by abahde 0
Releases(v1.2.0-beta)
  • v1.2.0-beta(Sep 17, 2019)

    Draft of release notes:

    • A PyTorch implementation (though not for all test problems yet)
    • A refactored Analyzer module (more flexibility and interpretability)
    • A Tuning module that automates the tuning process
    • Some minor improvements of the TensorFlow code (important bugfix: fmnist_mlp now really uses F-MNIST and not MNIST)
    • For the PyTorch code a validation set metric for each test problem. However, so far, the TensorFlow code comes without validation sets.
    • Runners now break from training if the loss becomes NaN.
    • Runners now return the output dictionary.
    • Additional training parameters can be passed as kwargs to the run() method.
    • Numpy is now also seeded.
    • Small and large benchmark sets are now global variables in DeepOBS.
    • Default test problem settings are now a global variable in DeepOBS.
    • JSON output is now dumped in human readable format.
    • Accuracy is now only printed if available.
    • Simplified Runner API.
    • Learning Rate Schedule Runner is now an extra class.
    Source code(tar.gz)
    Source code(zip)
Owner
Aaron Bahde
Graduate student at the University of Tübingen, Methods of Machine Learning
Aaron Bahde
Model-based reinforcement learning in TensorFlow

Bellman Website | Twitter | Documentation (latest) What does Bellman do? Bellman is a package for model-based reinforcement learning (MBRL) in Python,

46 Nov 09, 2022
Codes of paper "Unseen Object Amodal Instance Segmentation via Hierarchical Occlusion Modeling"

Unseen Object Amodal Instance Segmentation (UOAIS) Seunghyeok Back, Joosoon Lee, Taewon Kim, Sangjun Noh, Raeyoung Kang, Seongho Bak, Kyoobin Lee This

GIST-AILAB 92 Dec 13, 2022
The UI as a mobile display for OP25

OP25 Mobile Control Head A 'remote' control head that interfaces with an OP25 instance. We take advantage of some data end-points left exposed for the

Sarah Rose Giddings 13 Dec 28, 2022
Place holder for HOPE: a human-centric and task-oriented MT evaluation framework using professional post-editing

HOPE: A Task-Oriented and Human-Centric Evaluation Framework Using Professional Post-Editing Towards More Effective MT Evaluation Place holder for dat

Lifeng Han 1 Apr 25, 2022
Discriminative Condition-Aware PLDA

DCA-PLDA This repository implements the Discriminative Condition-Aware Backend described in the paper: L. Ferrer, M. McLaren, and N. Brümmer, "A Speak

Luciana Ferrer 31 Aug 05, 2022
Implement of homography net by pytorch

HomographyNet Implement of homography net by pytorch Brief Introduction This project is based on the work Homography-Net: @article{detone2016deep, t

ronghao_CN 4 May 19, 2022
Code release for "COTR: Correspondence Transformer for Matching Across Images"

COTR: Correspondence Transformer for Matching Across Images This repository contains the inference code for COTR. We plan to release the training code

UBC Computer Vision Group 360 Jan 06, 2023
a basic code repository for basic task in CV(classification,detection,segmentation)

basic_cv a basic code repository for basic task in CV(classification,detection,segmentation,tracking) classification generate dataset train predict de

1 Oct 15, 2021
Pytorch implementation of the paper: "SAPNet: Segmentation-Aware Progressive Network for Perceptual Contrastive Image Deraining"

SAPNet This repository contains the official Pytorch implementation of the paper: "SAPNet: Segmentation-Aware Progressive Network for Perceptual Contr

11 Oct 17, 2022
This is a simple backtesting framework to help you test your crypto currency trading. It includes a way to download and store historical crypto data and to execute a trading strategy.

You can use this simple crypto backtesting script to ensure your trading strategy is successful Minimal setup required and works well with static TP a

Andrei 154 Sep 12, 2022
Torch code for our CVPR 2018 paper "Residual Dense Network for Image Super-Resolution" (Spotlight)

Residual Dense Network for Image Super-Resolution This repository is for RDN introduced in the following paper Yulun Zhang, Yapeng Tian, Yu Kong, Bine

Yulun Zhang 494 Dec 30, 2022
U-Time: A Fully Convolutional Network for Time Series Segmentation

U-Time & U-Sleep Official implementation of The U-Time [1] model for general-purpose time-series segmentation. The U-Sleep [2] model for resilient hig

Mathias Perslev 176 Dec 19, 2022
Repository for "Space-Time Correspondence as a Contrastive Random Walk" (NeurIPS 2020)

Space-Time Correspondence as a Contrastive Random Walk This is the repository for Space-Time Correspondence as a Contrastive Random Walk, published at

A. Jabri 239 Dec 27, 2022
LAVT: Language-Aware Vision Transformer for Referring Image Segmentation

LAVT: Language-Aware Vision Transformer for Referring Image Segmentation Where we are ? 12.27 目前和原论文仍有1%左右得差距,但已经力压很多SOTA了 ckpt__448_epoch_25.pth mIoU

zichengsaber 60 Dec 11, 2022
PatrickStar enables Larger, Faster, Greener Pretrained Models for NLP. Democratize AI for everyone.

PatrickStar: Parallel Training of Large Language Models via a Chunk-based Memory Management Meeting PatrickStar Pre-Trained Models (PTM) are becoming

Tencent 633 Dec 28, 2022
PyTorch implementation for "Mining Latent Structures with Contrastive Modality Fusion for Multimedia Recommendation"

MIRCO PyTorch implementation for paper: Latent Structures Mining with Contrastive Modality Fusion for Multimedia Recommendation Dependencies Python 3.

Big Data and Multi-modal Computing Group, CRIPAC 9 Dec 08, 2022
PyTorch Implementation of ByteDance's Cross-speaker Emotion Transfer Based on Speaker Condition Layer Normalization and Semi-Supervised Training in Text-To-Speech

Cross-Speaker-Emotion-Transfer - PyTorch Implementation PyTorch Implementation of ByteDance's Cross-speaker Emotion Transfer Based on Speaker Conditio

Keon Lee 114 Jan 08, 2023
CLIPort: What and Where Pathways for Robotic Manipulation

CLIPort CLIPort: What and Where Pathways for Robotic Manipulation Mohit Shridhar, Lucas Manuelli, Dieter Fox CoRL 2021 CLIPort is an end-to-end imitat

246 Dec 11, 2022
A graph-to-sequence model for one-step retrosynthesis and reaction outcome prediction.

Graph2SMILES A graph-to-sequence model for one-step retrosynthesis and reaction outcome prediction. 1. Environmental setup System requirements Ubuntu:

29 Nov 18, 2022
Le dataset des images du projet d'IA de 2021

face-mask-dataset-ilc-2021 Le dataset des images du projet d'IA de 2021, Indiquez vos id git dans la issue pour les droits TL;DR: Choisir 200 images J

7 Nov 15, 2021