Parameter-ensemble-differential-evolution - Shows how to do parameter ensembling using differential evolution.

Last update: May 04, 2022

Overview

Ensembling parameters with differential evolution

This repository shows how to ensemble parameters of two trained neural networks using differential evolution. The steps followed are as follows:

Train two networks (architecturally same) on the same dataset (CIFAR-10 used here) but from two different random initializations.
Ensemble their weights using the following formulae:
```
w_t = w_o * ema + (1 - ema) * w_p
```
w_o and w_p represents the learned of a neural network.
Randomly initialize a network (same architecture as above) and populate its parameters w_t using the above formulae.

ema is usually chosen by the developer in an empirical manner. This project uses differential evolution to find it.

Below are the top-1 accuracies (on CIFAR-10 test set) of two individually trained two models along with their ensembled variant:

Model one: 63.23%
Model two: 63.42%
Ensembled: 63.35%

With the more conventional average prediction ensembling, I was able to get to 64.92%. This is way better than what I got by ensembling the parameters. Nevertheless, the purpose of this project was to just try out an idea.

Reproducing the results

Ensure the requirements.txt is satisfied. Then train two models with ensuring your working directory is at the root of this project:

$ git clone https://github.com/sayakpaul/parameter-ensemble-differential-evolution
$ cd parameter-ensemble-differential-evolution
$ pip install -qr requirements.txt
$ for i in `seq 1 2`; python train.py; done

Then just follow the ensemble-parameters.ipynb notebook. You can also use the networks I trained. Instructions are available inside the notebook.

Parameter-ensemble-differential-evolution - Shows how to do parameter ensembling using differential evolution.

Related tags

Overview

Ensembling parameters with differential evolution

Reproducing the results

References

You might also like...

Neural Ensemble Search for Performant and Calibrated Predictions

An Ensemble of CNN (Python 3.5.1 Tensorflow 1.3 numpy 1.13)

zeus is a Python implementation of the Ensemble Slice Sampling method.

Pytorch implementation of SenFormer: Efficient Self-Ensemble Framework for Semantic Segmentation

Ensemble Knowledge Guided Sub-network Search and Fine-tuning for Filter Pruning

This Jupyter notebook shows one way to implement a simple first-order low-pass filter on sampled data in discrete time.

A fast Evolution Strategy implementation in Python

Code for the paper Task Agnostic Morphology Evolution.

Pytorch implementation of FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks

Releases(v0.1.0)

v0.1.0(Jan 2, 2022)

Owner

Sayak Paul

This is RFA-Toolbox, a simple and easy-to-use library that allows you to optimize your neural network architectures using receptive field analysis (RFA) and create graph visualizations of your architecture.

VolumeGAN - 3D-aware Image Synthesis via Learning Structural and Textural Representations

Transport Mode detection - can detect the mode of transport with the help of features such as acceeration,jerk etc

Split your patch similarly to `git add -p` but supporting multiple buckets

Evaluating AlexNet features at various depths

This is the official repository of XVFI (eXtreme Video Frame Interpolation)

Official PyTorch Implementation of Rank & Sort Loss [ICCV2021]

Integrated physics-based and ligand-based modeling.

Semi-Supervised Learning for Fine-Grained Classification

TensorFlow Implementation of Unsupervised Cross-Domain Image Generation

A Python framework for conversational search

Optimizers-visualized - Visualization of different optimizers on local minimas and saddle points.

CvT-ASSD: Convolutional vision-Transformerbased Attentive Single Shot MultiBox Detector (ICTAI 2021 CCF-C 会议)The 33rd IEEE International Conference on Tools with Artificial Intelligence

Info and sample codes for "NTU RGB+D Action Recognition Dataset"

An implementation of Equivariant e2 convolutional kernals into a convolutional self attention network, applied to radio astronomy data.

Ground truth data for the Optical Character Recognition of Historical Classical Commentaries.

The code for SAG-DTA: Prediction of Drug–Target Affinity Using Self-Attention Graph Network.

NUANCED is a user-centric conversational recommendation dataset that contains 5.1k annotated dialogues and 26k high-quality user turns.

Episodic-memory - Ego4D Episodic Memory Benchmark

Pgn2tex - Scripts to convert pgn files to latex document. Useful to build books or pdf from pgn studies