Code accompanying the paper "How Tight Can PAC-Bayes be in the Small Data Regime?"

Last update: Dec 21, 2021

Overview

How Tight Can PAC-Bayes be in the Small Data Regime?

This is the code to reproduce all experiments for the following paper:

@inproceedings{Foong:2021:How_Tight_Can_PAC-Bayes_Be,
    title = {How Tight Can {PAC}-{Bayes} Be in the Small Data Regime?},
    year = {2021},
    author = {Andrew Y. K. Foong and Wessel P. Bruinsma and David R. Burt and Richard E. Turner},
    booktitle = {Advances in Neural Information Processing Systems},
    volume = {35},
    eprint = {https://arxiv.org/abs/2106.03542},
}

Every experiment creates a folder in _experiments. The names of the files in those folders should be self-explanatory.

Installation

First, create and activate a virtual environment for Python 3.8.

virtualenv venv -p python3.8 
source venv/bin/activate

Then install an appropriate GPU-accelerated version of PyTorch.

Finally, install the requirements for the project.

pip install -e .

You should now be able to run the below commands.

Generating Datasets

In order to generate the synthetic 1D datasets used, run these commands from inside classification_1d:

python gen_data.py --class_scheme balanced --num_context 30 --name 30-context --num_train_batches 5000 --num_test_batches 64
python gen_data.py --class_scheme balanced --num_context 60 --name 60-context --num_train_batches 5000 --num_test_batches 64

The generated datasets will be in pacbayes/_data_caches

Theory Experiments

See Figure 2 in Section 3 and Appendix G.

python theory_experiments.py --setting det1-1
python theory_experiments.py --setting det1-2
python theory_experiments.py --setting det2-1
python theory_experiments.py --setting det2-1

python theory_experiments.py --setting stoch1
python theory_experiments.py --setting stoch2
python theory_experiments.py --setting stoch3

python theory_experiments.py --setting random --random-seed 1 --random-better-bound maurer
python theory_experiments.py --setting random --random-seed 6 --random-better-bound catoni

GNP Classification Experiments

See Figure 3 and 4 in Section 4 and Appendices I and J. The numbers from the graphs can be found in eval_metrics_no_post_opt.txt (without post optimisation) eval_metrics_post_opt.txt (with post optimisation).

MODEL_NONDDP=maurer MODEL_DDP=maurer-ddp NUM_CONTEXT=30 ./run_GNP_prop_024.sh
MODEL_NONDDP=maurer MODEL_DDP=maurer-ddp NUM_CONTEXT=30 ./run_GNP_prop_68.sh
MODEL_NONDDP=catoni MODEL_DDP=catoni-ddp NUM_CONTEXT=30 ./run_GNP_prop_024.sh
MODEL_NONDDP=catoni MODEL_DDP=catoni-ddp NUM_CONTEXT=30 ./run_GNP_prop_68.sh
MODEL_NONDDP=convex-nonseparable MODEL_DDP=convex-nonseparable-ddp NUM_CONTEXT=30 ./run_GNP_prop_024.sh
MODEL_NONDDP=convex-nonseparable MODEL_DDP=convex-nonseparable-ddp NUM_CONTEXT=30 ./run_GNP_prop_68.sh
MODEL_NONDDP=kl-val MODEL_DDP=kl-val NUM_CONTEXT=30 ./run_GNP_prop_024.sh
MODEL_NONDDP=kl-val MODEL_DDP=kl-val NUM_CONTEXT=30 ./run_GNP_prop_68.sh
MODEL_NONDDP=maurer-optimistic MODEL_DDP=maurer-optimistic-ddp NUM_CONTEXT=30 ./run_GNP_prop_024.sh
MODEL_NONDDP=maurer-optimistic MODEL_DDP=maurer-optimistic-ddp NUM_CONTEXT=30 ./run_GNP_prop_68.sh
MODEL_NONDDP=maurer-inv MODEL_DDP=maurer-inv-ddp NUM_CONTEXT=30 ./run_GNP_prop_024.sh
MODEL_NONDDP=maurer-inv MODEL_DDP=maurer-inv-ddp NUM_CONTEXT=30 ./run_GNP_prop_68.sh
MODEL_NONDDP=maurer-inv-optimistic MODEL_DDP=maurer-inv-optimistic-ddp NUM_CONTEXT=30 ./run_GNP_prop_024.sh
MODEL_NONDDP=maurer-inv-optimistic MODEL_DDP=maurer-inv-optimistic-ddp NUM_CONTEXT=30 ./run_GNP_prop_68.sh

MODEL_NONDDP=maurer MODEL_DDP=maurer-ddp NUM_CONTEXT=60 ./run_GNP_prop_024.sh
MODEL_NONDDP=maurer MODEL_DDP=maurer-ddp NUM_CONTEXT=60 ./run_GNP_prop_68.sh
MODEL_NONDDP=catoni MODEL_DDP=catoni-ddp NUM_CONTEXT=60 ./run_GNP_prop_024.sh
MODEL_NONDDP=catoni MODEL_DDP=catoni-ddp NUM_CONTEXT=60 ./run_GNP_prop_68.sh
MODEL_NONDDP=convex-nonseparable MODEL_DDP=convex-nonseparable-ddp NUM_CONTEXT=60 ./run_GNP_prop_024.sh
MODEL_NONDDP=convex-nonseparable MODEL_DDP=convex-nonseparable-ddp NUM_CONTEXT=60 ./run_GNP_prop_68.sh
MODEL_NONDDP=kl-val MODEL_DDP=kl-val NUM_CONTEXT=60 ./run_GNP_prop_024.sh
MODEL_NONDDP=kl-val MODEL_DDP=kl-val NUM_CONTEXT=60 ./run_GNP_prop_68.sh
MODEL_NONDDP=maurer-optimistic MODEL_DDP=maurer-optimistic-ddp NUM_CONTEXT=60 ./run_GNP_prop_024.sh
MODEL_NONDDP=maurer-optimistic MODEL_DDP=maurer-optimistic-ddp NUM_CONTEXT=60 ./run_GNP_prop_68.sh
MODEL_NONDDP=maurer-inv MODEL_DDP=maurer-inv-ddp NUM_CONTEXT=60 ./run_GNP_prop_024.sh
MODEL_NONDDP=maurer-inv MODEL_DDP=maurer-inv-ddp NUM_CONTEXT=60 ./run_GNP_prop_68.sh
MODEL_NONDDP=maurer-inv-optimistic MODEL_DDP=maurer-inv-optimistic-ddp NUM_CONTEXT=60 ./run_GNP_prop_024.sh
MODEL_NONDDP=maurer-inv-optimistic MODEL_DDP=maurer-inv-optimistic-ddp NUM_CONTEXT=60 ./run_GNP_prop_68.sh

MLP Classification Experiments

See Appendix J. The numbers from the graphs can be found in eval_metrics_no_post_opt.txt (without post optimisation) eval_metrics_post_opt.txt (with post optimisation).

MODEL_NONDDP=catoni MODEL_DDP=catoni-ddp NUM_CONTEXT=30 ./run_MLP.sh
MODEL_NONDDP=kl-val MODEL_DDP=kl-val NUM_CONTEXT=30 ./run_MLP.sh

MODEL_NONDDP=catoni MODEL_DDP=catoni-ddp NUM_CONTEXT=60 ./run_MLP.sh
MODEL_NONDDP=kl-val MODEL_DDP=kl-val NUM_CONTEXT=60 ./run_MLP.sh

Code accompanying the paper "How Tight Can PAC-Bayes be in the Small Data Regime?"

Related tags

Overview

How Tight Can PAC-Bayes be in the Small Data Regime?

Installation

Generating Datasets

Theory Experiments

GNP Classification Experiments

MLP Classification Experiments

Owner

Repositório para arquivos sobre o Módulo 1 do curso Top Coders da Let's Code + Safra

Makes patches from huge resolution .svs slide files using openslide

[CVPR'21 Oral] Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning

Go from graph data to a secure and interactive visual graph app in 15 minutes. Batteries-included self-hosting of graph data apps with Streamlit, Graphistry, RAPIDS, and more!

Replication Package for AequeVox:Automated Fariness Testing for Speech Recognition Systems

Official implementation of FCL-taco2: Fast, Controllable and Lightweight version of Tacotron2 @ ICASSP 2021

PenguinSpeciesPredictionML - Basic model to predict Penguin species based on beak size and sex.

PyTorch Implementation of Fully Convolutional Networks. (Training code to reproduce the original result is available.)

The implementation for "Comprehensive Knowledge Distillation with Causal Intervention".

Next-gen Rowhammer fuzzer that uses non-uniform, frequency-based patterns.

An Easy-to-use, Modular and Prolongable package of deep-learning based Named Entity Recognition Models.

A highly efficient and modular implementation of Gaussian Processes in PyTorch

We present a framework for training multi-modal deep learning models on unlabelled video data by forcing the network to learn invariances to transformations applied to both the audio and video streams.

Pytorch implementation for "Open Compound Domain Adaptation" (CVPR 2020 ORAL)

DrWhy is the collection of tools for eXplainable AI (XAI). It's based on shared principles and simple grammar for exploration, explanation and visualisation of predictive models.

Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. It can use GPUs and perform efficient symbolic differentiation.

Colour detection is necessary to recognize objects, it is also used as a tool in various image editing and drawing apps.

Code reproduce for paper "Vehicle Re-identification with Viewpoint-aware Metric Learning"

Algorithmic encoding of protected characteristics and its implications on disparities across subgroups

DynaTune: Dynamic Tensor Program Optimization in Deep Neural Network Compilation