🔪 Elimination based Lightweight Neural Net with Pretrained Weights

Overview

ELimNet

ELimNet: Eliminating Layers in a Neural Network Pretrained with Large Dataset for Downstream Task

  • Removed top layers from pretrained EfficientNetB0 and ResNet18 to construct lightweight CNN model with less than 1M #params.
  • Assessed on Trash Annotations in Context(TACO) Dataset sampled for 6 classes with 20,851 images.
  • Compared performance with lightweight models generated with Optuna's Neural Architecture Search(NAS) constituted with same convolutional blocks.

Quickstart

Installation

# clone the repository
git clone https://github.com/snoop2head/elimnet

# fetch image dataset and unzip
!wget -cq https://aistages-prod-server-public.s3.amazonaws.com/app/Competitions/000081/data/data.zip
!unzip ./data.zip -d ./

Train

# finetune on the dataset with pretrained model
python train.py --model ./model/efficientnet_b0.yaml

# finetune on the dataset with ElimNet
python train.py --model ./model/efficientnet_b0_elim_3.yaml

Inference

# inference with the lastest ran model
python inference.py --model_dir ./exp/latest/

Performance

Performance is compared with (1) original pretrained model and (2) Optuna NAS constructed models with no pretrained weights.

  • Indicates that top convolutional layers eliminated pretrained CNN models outperforms empty Optuna NAS models generated with same convolutional blocks.
  • Suggests that eliminating top convolutional layers creates lightweight model that shows similar(or better) classifcation performance with original pretrained model.
  • Reduces parameters to 7%(or less) of its original parameters while maintaining(or improving) its performance. Saves inference time by 20% or more by eliminating top convolutional layters.

ELimNet vs Pretrained Models (Train)

[100 epochs] # of Parameters # of Layers Train Validation Test F1
Pretrained EfficientNet B0 4.0M 352 Loss: 0.43
Acc: 81.23%
F1: 0.84
Loss: 0.469
Acc: 82.17%
F1: 0.76
0.7493
EfficientNet B0 Elim 2 0.9M 245 Loss:0.652
Acc: 87.22%
F1: 0.84
Loss: 0.622
Acc: 87.22%
F1: 0.77
0.7603
EfficientNet B0 Elim 3 0.30M 181 Loss: 0.602
Acc: 78.17%
F1: 0.74
Loss: 0.661
Acc: 77.41%
F1: 0.74
0.7349
Resnet18 11.17M 69 Loss: 0.578
Acc: 78.90%
F1: 0.76
Loss: 0.700
Acc: 76.17%
F1: 0.719
-
Resnet18 Elim 2 0.68M 37 Loss: 0.447
Acc: 83.73%
F1: 0.71
Loss: 0.712
Acc: 75.42%
F1: 0.71
-

ELimNet vs Pretrained Models (Inference)

# of Parameters # of Layers CPU times (sec) CUDA time (sec) Test Inference Time (sec)
Pretrained EfficientNet B0 4.0M 352 3.9s 4.0s 105.7s
EfficientNet B0 Elim 2 0.9M 245 4.1s 13.0s 83.4s
EfficientNet B0 Elim 3 0.30M 181 3.0s 9.0s 73.5s
Resnet18 11.17M 69 - - -
Resnet18 Elim 2 0.68M 37 - - -

ELimNet vs Empty Optuna NAS Models (Train)

[100 epochs] # of Parameters # of Layers Train Valid Test F1
Empty MobileNet V3 4.2M 227 Loss 0.925
Acc: 65.18%
F1: 0.58
Loss 0.993
Acc: 62.83%
F1: 0.56
-
Empty EfficientNet B0 1.3M 352 Loss 0.867
Acc: 67.28%
F1: 0.61
Loss 0.898
Acc: 66.80%
F1: 0.61
0.6337
Empty DWConv & InvertedResidualv3 NAS 0.08M 66 - Loss: 0.766
Acc: 71.71%
F1: 0.68
0.6740
Empty MBConv NAS 0.33M 141 Loss: 0.786
Acc: 70.72%
F1: 0.66
Loss: 0.866
Acc: 68.09%
F1: 0.62
0.6245
Resnet18 Elim 2 0.68M 37 Loss: 0.447
Acc: 83.73%
F1: 0.71
Loss: 0.712
Acc: 75.42%
F1: 0.71
-
EfficientNet B0 Elim 3 0.30M 181 Loss: 0.602
Acc: 78.17%
F1: 0.74
Loss: 0.661
Acc: 77.41%
F1: 0.74
0.7603

ELimNet vs Empty Optuna NAS Models (Inference)

# of Parameters # of Layers CPU times (sec) CUDA time (sec) Test Inference Time (sec)
Empty MobileNet V3 4.2M 227 4 13 -
Empty EfficientNet B0 1.3M 352 3.780 3.782 68.4s
Empty DWConv &
InvertedResidualv3 NAS
0.08M 66 1 3.5 61.1s
Empty MBConv NAS 0.33M 141 2.14 7.201 67.1s
Resnet18 Elim 2 0.68M 37 - - -
EfficientNet B0 Elim 3 0.30M 181 3.0s 9s 73.5s

Background & WiP

Background

Work in Progress

  • Will test the performance of replacing convolutional blocks with pretrained weights with a single convolutional layer without pretrained weights.
  • Will add ResNet18's inference time data and compare with Optuna's NAS constructed lightweight model.
  • Will test on pretrained MobileNetV3, MnasNet on torchvision with elimination based lightweight model architecture search.
  • Will be applied on other small datasets such as Fashion MNIST dataset and Plant Village dataset.

Others

  • "Empty" stands for model with no pretrained weights.
  • "EfficientNet B0 Elim 2" means 2 convolutional blocks have been eliminated from pretrained EfficientNet B0. Number next to "Elim" annotates how many convolutional blocks have been removed.
  • Table's performance illustrates best performance out of 100 epochs of finetuning on TACO Dataset.

Authors

Owner
snoop2head
break, compose, display
snoop2head
[SDM 2022] Towards Similarity-Aware Time-Series Classification

SimTSC This is the PyTorch implementation of SDM2022 paper Towards Similarity-Aware Time-Series Classification. We propose Similarity-Aware Time-Serie

Daochen Zha 49 Dec 27, 2022
Pansharpening by convolutional neural networks in the full resolution framework

Z-PNN: Zoom Pansharpening Neural Network Pansharpening by convolutional neural networks in the full resolution framework is a deep learning method for

20 Nov 24, 2022
PyTorch implementation of Soft-DTW: a Differentiable Loss Function for Time-Series in CUDA

Soft DTW Loss Function for PyTorch in CUDA This is a Pytorch Implementation of Soft-DTW: a Differentiable Loss Function for Time-Series which is batch

Keon Lee 76 Dec 20, 2022
A demonstration of using a live Tensorflow session to create an interactive face-GAN explorer.

Streamlit Demo: The Controllable GAN Face Generator This project highlights Streamlit's new hash_func feature with an app that calls on TensorFlow to

Streamlit 257 Dec 31, 2022
[ICLR 2022] Contact Points Discovery for Soft-Body Manipulations with Differentiable Physics

CPDeform Code and data for paper Contact Points Discovery for Soft-Body Manipulations with Differentiable Physics at ICLR 2022 (Spotlight). @InProceed

(Lester) Sizhe Li 29 Nov 29, 2022
Bling's Object detection tool

BriVL for Building Applications This repo is used for illustrating how to build applications by using BriVL model. This repo is re-implemented from fo

chuhaojin 47 Nov 01, 2022
[ICCV 2021] HRegNet: A Hierarchical Network for Large-scale Outdoor LiDAR Point Cloud Registration

HRegNet: A Hierarchical Network for Large-scale Outdoor LiDAR Point Cloud Registration Introduction The repository contains the source code and pre-tr

Intelligent Sensing, Perception and Computing Group 55 Dec 14, 2022
High-resolution networks and Segmentation Transformer for Semantic Segmentation

High-resolution networks and Segmentation Transformer for Semantic Segmentation Branches This is the implementation for HRNet + OCR. The PyTroch 1.1 v

HRNet 2.8k Jan 07, 2023
Approaches to modeling terrain and maps in python

topography 🌎 Contains different approaches to modeling terrain and topographic-style maps in python Features Inverse Distance Weighting (IDW) A given

John Gutierrez 1 Aug 10, 2022
Implementation of the GBST block from the Charformer paper, in Pytorch

Charformer - Pytorch Implementation of the GBST (gradient-based subword tokenization) module from the Charformer paper, in Pytorch. The paper proposes

Phil Wang 105 Dec 26, 2022
This repository is for Competition for ML_data class

This repository is for Competition for ML_data class. Based on mmsegmentatoin,mainly using swin transformer to completed the competition.

jianlong 2 Oct 23, 2022
Code for the paper BERT might be Overkill: A Tiny but Effective Biomedical Entity Linker based on Residual Convolutional Neural Networks

Biomedical Entity Linking This repo provides the code for the paper BERT might be Overkill: A Tiny but Effective Biomedical Entity Linker based on Res

Tuan Manh Lai 24 Oct 24, 2022
SubOmiEmbed: Self-supervised Representation Learning of Multi-omics Data for Cancer Type Classification

SubOmiEmbed: Self-supervised Representation Learning of Multi-omics Data for Cancer Type Classification

Sayed Hashim 3 Nov 15, 2022
Code for the paper 'A High Performance CRF Model for Clothes Parsing'.

Clothes Parsing Overview This code provides an implementation of the research paper: A High Performance CRF Model for Clothes Parsing Edgar Simo-S

Edgar Simo-Serra 119 Nov 21, 2022
Unofficial implementation (replicates paper results!) of MINER: Multiscale Implicit Neural Representations in pytorch-lightning

MINER_pl Unofficial implementation of MINER: Multiscale Implicit Neural Representations in pytorch-lightning. 📖 Ref readings Laplacian pyramid explan

AI葵 51 Nov 28, 2022
Lex Rosetta: Transfer of Predictive Models Across Languages, Jurisdictions, and Legal Domains

Lex Rosetta: Transfer of Predictive Models Across Languages, Jurisdictions, and Legal Domains This is an accompanying repository to the ICAIL 2021 pap

4 Dec 16, 2021
Code for Multiple Instance Active Learning for Object Detection, CVPR 2021

Language: 简体中文 | English Introduction This is the code for Multiple Instance Active Learning for Object Detection, CVPR 2021. Installation A Linux pla

Tianning Yuan 269 Dec 21, 2022
Multi-layer convolutional LSTM with Pytorch

Convolution_LSTM_pytorch Thanks for your attention. I haven't got time to maintain this repo for a long time. I recommend this repo which provides an

Zijie Zhuang 733 Dec 30, 2022
A modification of Daniel Russell's notebook merged with Katherine Crowson's hq-skip-net changes

Edits made to this repo by Katherine Crowson I have added several features to this repository for use in creating higher quality generative art (featu

Paul Fishwick 10 May 07, 2022
Unofficial & improved implementation of NeRF--: Neural Radiance Fields Without Known Camera Parameters

[Unofficial code-base] NeRF--: Neural Radiance Fields Without Known Camera Parameters [ Project | Paper | Official code base ] ⬅️ Thanks the original

Jianfei Guo 239 Dec 22, 2022