Neuron Merging: Compensating for Pruned Neurons (NeurIPS 2020)

Overview

Neuron Merging: Compensating for Pruned Neurons

Pytorch implementation of Neuron Merging: Compensating for Pruned Neurons, accepted at 34th Conference on Neural Information Processing Systems (NeurIPS 2020).

Requirements

To install requirements:

conda env create -f ./environment.yml

Python environment & main libraries:

  • python 3.8
  • pytorch 1.5.0
  • scikit-learn 0.22.1
  • torchvision 0.6.0

LeNet-300-100

To test LeNet-300-100 model on FashionMNIST, run:

bash scripts/LeNet_300_100_FashionMNIST.sh -t [model type] -c [criterion] -r [pruning ratio]

You can use three arguments for this script:

  • model type: original | prune | merge
  • pruning criterion : l1-norm | l2-norm | l2-GM
  • pruning ratio : 0.0 ~ 1.0

For example, to test the model after pruning 50% of the neurons with $l_1$-norm criterion, run:

bash scripts/LeNet_300_100_FashionMNIST.sh -t prune -c l1-norm -r 0.5

To test the model after merging , run:

bash scripts/LeNet_300_100_FashionMNIST.sh -t merge -c l1-norm -r 0.5

VGG-16

To test VGG-16 model on CIFAR-10, run:

bash scripts/VGG16_CIFAR10.sh -t [model type] -c [criterion]

You can use two arguments for this script

  • model type: original | prune | merge
  • pruning criterion: l1-norm | l2-norm | l2-GM

As a pretrained model on CIFAR-100 is not included, you must train it first. To train VGG-16 on CIFAR-100, run:

bash scripts/VGG16_CIFAR100_train.sh

All the hyperparameters are as described in the supplementary material.

After training, to test VGG-16 model on CIFAR-100, run:

bash scripts/VGG16_CIFAR100.sh -t [model type] -c [criterion]

You can use two arguments for this script

  • model type: original | prune | merge
  • pruning criterion: l1-norm | l2-norm | l2-GM

ResNet

To test ResNet-56 model on CIFAR-10, run:

bash scripts/ResNet56_CIFAR10.sh -t [model type] -c [criterion] -r [pruning ratio]

You can use three arguments for this script

  • model type: original | prune | merge
  • pruning method : l1-norm | l2-norm | l2-GM
  • pruning ratio : 0.0 ~ 1.0

To test WideResNet-40-4 model on CIFAR-10, run:

bash scripts/WideResNet_40_4_CIFAR10.sh -t [model type] -c [criterion] -r [pruning ratio]

You can use three arguments for this script

  • model type: original | prune | merge
  • pruning method : l1-norm | l2-norm | l2-GM
  • pruning ratio : 0.0 ~ 1.0

Results

Our model achieves the following performance on (without fine-tuning) :

Image classification of LeNet-300-100 on FashionMNIST

Baseline Accuracy : 89.80%

Pruning Ratio Prune ($l_1$-norm) Merge
50% 88.40% 88.69%
60% 85.17% 86.92%
70% 71.26% 82.75%
80% 66.76 80.02%

Image classification of VGG-16 on CIFAR-10

Baseline Accuracy : 93.70%

Criterion Prune Merge
$l_1$-norm 88.70% 93.16%
$l_2$-norm 89.14% 93.16%
$l_2$-GM 87.85% 93.10%

Citation

@inproceedings{kim2020merging,
  title     = {Neuron Merging: Compensating for Pruned Neurons},
  author    = {Kim, Woojeong and Kim, Suhyun and Park, Mincheol and Jeon, Geonseok},
  booktitle = {Advances in Neural Information Processing Systems 33},
  year      = {2020}
}
Owner
Woojeong Kim
Woojeong Kim
Cooperative multi-agent reinforcement learning for high-dimensional nonequilibrium control

Cooperative multi-agent reinforcement learning for high-dimensional nonequilibrium control Official implementation of: Cooperative multi-agent reinfor

0 Nov 16, 2021
This is the solution for 2nd rank in Kaggle competition: Feedback Prize - Evaluating Student Writing.

Feedback Prize - Evaluating Student Writing This is the solution for 2nd rank in Kaggle competition: Feedback Prize - Evaluating Student Writing. The

Udbhav Bamba 41 Dec 14, 2022
SlideGraph+: Whole Slide Image Level Graphs to Predict HER2 Status in Breast Cancer

SlideGraph+: Whole Slide Image Level Graphs to Predict HER2 Status in Breast Cancer A novel graph neural network (GNN) based model (termed SlideGraph+

28 Dec 24, 2022
public repo for ESTER dataset and modeling (EMNLP'21)

Project / Paper Introduction This is the project repo for our EMNLP'21 paper: https://arxiv.org/abs/2104.08350 Here, we provide brief descriptions of

PlusLab 19 Oct 27, 2022
Official code release for "Learned Spatial Representations for Few-shot Talking-Head Synthesis" ICCV 2021

Official code release for "Learned Spatial Representations for Few-shot Talking-Head Synthesis" ICCV 2021

Moustafa Meshry 16 Oct 05, 2022
A pre-trained language model for social media text in Spanish

RoBERTuito A pre-trained language model for social media text in Spanish READ THE FULL PAPER Github Repository RoBERTuito is a pre-trained language mo

25 Dec 29, 2022
Official code for "InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization" (ICLR 2020, spotlight)

InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization Authors: Fan-yun Sun, Jordan Hoffm

Fan-Yun Sun 232 Dec 28, 2022
Dynamic Token Normalization Improves Vision Transformers

Dynamic Token Normalization Improves Vision Transformers This is the PyTorch implementation of the paper Dynamic Token Normalization Improves Vision T

Wenqi Shao 20 Oct 09, 2022
Autolfads-tf2 - A TensorFlow 2.0 implementation of Latent Factor Analysis via Dynamical Systems (LFADS) and AutoLFADS

autolfads-tf2 A TensorFlow 2.0 implementation of LFADS and AutoLFADS. Installati

Systems Neural Engineering Lab 11 Oct 29, 2022
Official Implementation of Swapping Autoencoder for Deep Image Manipulation (NeurIPS 2020)

Swapping Autoencoder for Deep Image Manipulation Taesung Park, Jun-Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei A. Efros, Richard Zhang UC

449 Dec 27, 2022
Generate images from texts. In Russian

ruDALL-E Generate images from texts pip install rudalle==1.1.0rc0 🤗 HF Models: ruDALL-E Malevich (XL) ruDALL-E Emojich (XL) (readme here) ruDALL-E S

AI Forever 1.6k Dec 31, 2022
Convert onnx models to pytorch.

onnx2torch onnx2torch is an ONNX to PyTorch converter. Our converter: Is easy to use – Convert the ONNX model with the function call convert; Is easy

ENOT 264 Dec 30, 2022
Text Generation by Learning from Demonstrations

Text Generation by Learning from Demonstrations The README was last updated on March 7, 2021. The repo is based on fairseq (v0.9.?). Paper arXiv Prere

38 Oct 21, 2022
repro_eval is a collection of measures to evaluate the reproducibility/replicability of system-oriented IR experiments

repro_eval repro_eval is a collection of measures to evaluate the reproducibility/replicability of system-oriented IR experiments. The measures were d

IR Group at Technische Hochschule Köln 9 May 25, 2022
A multilingual version of MS MARCO passage ranking dataset

mMARCO A multilingual version of MS MARCO passage ranking dataset This repository presents a neural machine translation-based method for translating t

75 Dec 27, 2022
Self-Guided Contrastive Learning for BERT Sentence Representations

Self-Guided Contrastive Learning for BERT Sentence Representations This repository is dedicated for releasing the implementation of the models utilize

Taeuk Kim 16 Dec 04, 2022
Multi-query Video Retreival

Multi-query Video Retreival

Princeton Visual AI Lab 17 Nov 22, 2022
Optimized primitives for collective multi-GPU communication

NCCL Optimized primitives for inter-GPU communication. Introduction NCCL (pronounced "Nickel") is a stand-alone library of standard communication rout

NVIDIA Corporation 2k Jan 09, 2023
Pytorch implementation of Masked Auto-Encoder

Masked Auto-Encoder (MAE) Pytorch implementation of Masked Auto-Encoder: Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick

Jiyuan 22 Dec 13, 2022
This is a simple backtesting framework to help you test your crypto currency trading. It includes a way to download and store historical crypto data and to execute a trading strategy.

You can use this simple crypto backtesting script to ensure your trading strategy is successful Minimal setup required and works well with static TP a

Andrei 154 Sep 12, 2022