Active and Sample-Efficient Model Evaluation

Overview

Active Testing: Sample-Efficient Model Evaluation

Hi, good to see you here! 👋

This is code for "Active Testing: Sample-Efficient Model Evaluation".

Please cite our paper, if you find this helpful:

@article{kossen2021active,
  title={{A}ctive {T}esting: {S}ample-{E}fficient {M}odel {E}valuation},
  author={Kossen, Jannik and Farquhar, Sebastian and Gal, Yarin and Rainforth, Tom},
  journal={arXiv:2103.05331},
  year={2021}
}

animation

Setup

The requirements.txt can be used to set up a python environment for this codebase. You can do this, for example, with conda:

conda create -n isactive python=3.8
conda activate isactive
pip install -r requirements.txt

Reproducing the Experiments

  • To reproduce a figure of the paper, first run the appropriate experiments
sh reproduce/experiments/figure-X.sh
  • And then create the plots with the Jupyter Notebook at
notebooks/plots_paper.ipynb
  • (The notebook let's you conveniently select which plots to recreate.)

  • Which should put plots into notebooks/plots/.

  • In the above, replace X by

    • 123 for Figures 1, 2, 3
    • 4 for Figure 4
    • 5 for Figure 5
    • 6 for Figure 6
    • 7 for Figure 7
  • Other notes

    • Synthetic data experiments do not require GPUs and should run on pretty much all recent hardware.
    • All other plots, realistically speaking, require GPUs.
    • We are also happy to share a 4 GB file with results from all experiments presented in the paper.
    • You may want to produce plots 7 and 8 for other experiment setups than the one in the paper, i.e. ones you already have computed.
    • Some experiments, e.g. those for Figures 4 or 6, may run a really long time on a single GPU. It may be good to
      • execute the scripts in the sh-files in parallel on multiple GPUs.
      • start multiple runs in parallel and then combine experiments. (See below).
      • end the runs early / decrease number of total runs (this can be very reasonable -- look at the config files in conf/paper to modify this property)
    • If you want to understand the code, below we give a good strategy for approaching it. (Also start with synthetic data experiments. They have less complex code!)

Running A Custom Experiment

  • main.py is the main entry point into this code-base.

    • It executes a a total of n_runs active testing experiments for a fixed setup.
    • Each experiment:
      • Trains (or loads) one main model.
      • This model can then be evaluated with a variety of acquisition strategies.
      • Risk estimates are then computed for points/weights from all acquisition strategies for all risk estimators.
  • This repository uses Hydra to manage configs.

    • Look at conf/config.yaml or one of the experiments in conf/... for default configs and hyperparameters.
    • Experiments are autologged and results saved to ./output/.
  • See notebooks/eplore_experiment.ipynb for some example code on how to evaluate custom experiments.

    • The evaluations use activetesting.visualize.Visualiser which implements visualisation methods.
    • Give it a path to an experiment in output/path/to/experiment and explore the methods.
    • If you want to combine data from multiple runs, give it a list of paths.
    • I prefer to load this in Jupyter Notebooks, but hey, everybody's different.
  • A guide to the code

    • main.py runs repeated experiments and orchestrates the whole shebang.
      • It iterates through all n_runs and acquisition strategies.
    • experiment.py handles a single experiment.
      • It combines the model, dataset, acquisition strategy, and risk estimators.
    • datasets.py, aquisition.py, loss.py, risk_estimators.py all contain exactly what you would expect!
    • hoover.py is a logging module.
    • models/ contains all models, scikit-learn and pyTorch.
      • In sk2torch.py we have some code that wraps torch models in a way that lets them be used as scikit-learn models from the outside.

And Finally

Thanks for stopping by!

If you find anything wrong with the code, please contact us.

We are happy to answer any questions related to the code and project.

Owner
Jannik Kossen
PhD Student at OATML Oxford
Jannik Kossen
Anomaly Localization in Model Gradients Under Backdoor Attacks Against Federated Learning

Federated_Learning This repo provides a federated learning framework that allows to carry out backdoor attacks under varying conditions. This is a ker

Arçelik ARGE Açık Kaynak Yazılım Organizasyonu 0 Nov 30, 2021
SciPy fixes and extensions

scipyx SciPy is large library used everywhere in scientific computing. That's why breaking backwards-compatibility comes as a significant cost and is

Nico Schlömer 16 Jul 17, 2022
GUI for a Vocal Remover that uses Deep Neural Networks.

GUI for a Vocal Remover that uses Deep Neural Networks.

4.4k Jan 07, 2023
PyTorch-Multi-Style-Transfer - Neural Style and MSG-Net

PyTorch-Style-Transfer This repo provides PyTorch Implementation of MSG-Net (ours) and Neural Style (Gatys et al. CVPR 2016), which has been included

Hang Zhang 906 Jan 04, 2023
The repository contain code for building compiler using puthon.

Building Compiler This is a python implementation of JamieBuild's "Super Tiny Compiler" Overview JamieBuilds developed a wonderfully educative compile

Shyam Das Shrestha 1 Nov 21, 2021
Official PyTorch implementation of UACANet: Uncertainty Aware Context Attention for Polyp Segmentation

UACANet: Uncertainty Aware Context Attention for Polyp Segmentation Official pytorch implementation of UACANet: Uncertainty Aware Context Attention fo

Taehun Kim 85 Dec 14, 2022
High performance, easy-to-use, and scalable machine learning (ML) package, including linear model (LR), factorization machines (FM), and field-aware factorization machines (FFM) for Python and CLI interface.

What is xLearn? xLearn is a high performance, easy-to-use, and scalable machine learning package that contains linear model (LR), factorization machin

Chao Ma 3k Jan 03, 2023
Semantically Contrastive Learning for Low-light Image Enhancement

Semantically Contrastive Learning for Low-light Image Enhancement Here, we propose an effective semantically contrastive learning paradigm for Low-lig

48 Dec 16, 2022
The source code of the ICCV2021 paper "PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering"

Website | ArXiv | Get Start | Video PIRenderer The source code of the ICCV2021 paper "PIRenderer: Controllable Portrait Image Generation via Semantic

Ren Yurui 261 Jan 09, 2023
A curated list of the latest breakthroughs in AI (in 2021) by release date with a clear video explanation, link to a more in-depth article, and code.

2021: A Year Full of Amazing AI papers- A Review 📌 A curated list of the latest breakthroughs in AI by release date with a clear video explanation, l

Louis-François Bouchard 2.9k Dec 31, 2022
An Unpaired Sketch-to-Photo Translation Model

Unpaired-Sketch-to-Photo-Translation We have released our code at https://github.com/rt219/Unsupervised-Sketch-to-Photo-Synthesis This project is the

38 Oct 28, 2022
Pixel-wise segmentation on VOC2012 dataset using pytorch.

PiWiSe Pixel-wise segmentation on the VOC2012 dataset using pytorch. FCN SegNet PSPNet UNet RefineNet For a more complete implementation of segmentati

Bodo Kaiser 378 Dec 30, 2022
CondenseNet: Light weighted CNN for mobile devices

CondenseNets This repository contains the code (in PyTorch) for "CondenseNet: An Efficient DenseNet using Learned Group Convolutions" paper by Gao Hua

Shichen Liu 690 Nov 30, 2022
Shallow Convolutional Neural Networks for Human Activity Recognition using Wearable Sensors

-IEEE-TIM-2021-1-Shallow-CNN-for-HAR [IEEE TIM 2021-1] Shallow Convolutional Neural Networks for Human Activity Recognition using Wearable Sensors All

Wenbo Huang 1 May 17, 2022
Codes for TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localization.

TS-CAM: Token Semantic Coupled Attention Map for Weakly SupervisedObject Localization This is the official implementaion of paper TS-CAM: Token Semant

vasgaowei 112 Jan 02, 2023
Code accompanying the paper "ProxyFL: Decentralized Federated Learning through Proxy Model Sharing"

ProxyFL Code accompanying the paper "ProxyFL: Decentralized Federated Learning through Proxy Model Sharing" Authors: Shivam Kalra*, Junfeng Wen*, Jess

Layer6 Labs 14 Dec 06, 2022
PyTorch implementation of Constrained Policy Optimization

PyTorch implementation of Constrained Policy Optimization (CPO) This repository has a simple to understand and use implementation of CPO in PyTorch. A

Sapana Chaudhary 25 Dec 08, 2022
Pytorch port of Google Research's LEAF Audio paper

leaf-audio-pytorch Pytorch port of Google Research's LEAF Audio paper published at ICLR 2021. This port is not completely finished, but the Leaf() fro

Dennis Fedorishin 80 Oct 31, 2022
A Simple Example for Imitation Learning with Dataset Aggregation (DAGGER) on Torcs Env

Imitation Learning with Dataset Aggregation (DAGGER) on Torcs Env This repository implements a simple algorithm for imitation learning: DAGGER. In thi

Hao 66 Nov 23, 2022
CNNs for Sentence Classification in PyTorch

Introduction This is the implementation of Kim's Convolutional Neural Networks for Sentence Classification paper in PyTorch. Kim's implementation of t

Shawn Ng 956 Dec 19, 2022