ElimNet: Eliminating Layers in a Neural Network Pretrained with Large Dataset for Downstream Task

📂 Please refer to README.pdf for further information.

• Removed top layers from pretrained EfficientNetB0 and ResNet18 to construct lightweight CNN model with less than 1M #params.
• Assessed on Trash Annotations in Context(TACO) Dataset sampled for 6 classes with 20,851 images.
• Compared performance with lightweight models generated with Optuna's Neural Architecture Search(NAS) constituted with same convolutional blocks.
• It is speculated that such elimination method will work on neural networks with residual connections, according to the paper of Veit et al(2016) where it shows residual networks behave like ensembles of networks. Refer to Figure 5 attached below, where error when deleting layers linearly increase rather than exponentially increasing.

# clone the repository
git clone https://github.com/snoop2head/elimnet

# fetch image dataset and unzip
!wget -cq https://aistages-prod-server-public.s3.amazonaws.com/app/Competitions/000081/data/data.zip
!unzip ./data.zip -d ./

# finetune on the dataset with pretrained model
python train.py --model ./model/efficientnet/efficientnet_b0.yaml

# finetune on the dataset with ElimNet
python train.py --model ./model/efficientnet/efficientnet_b0_elim_3.yaml

# inference with the lastest ran model
python inference.py --model_dir ./exp/latest/

Performance is compared with (1) original pretrained model and (2) Optuna NAS constructed models with no pretrained weights.

• Indicates that top convolutional layers eliminated pretrained CNN models outperforms empty Optuna NAS models generated with same convolutional blocks.
• Suggests that eliminating top convolutional layers creates lightweight model that shows similar(or better) classifcation performance with original pretrained model.
• Reduces parameters to 7%(or less) of its original parameters while maintaining(or improving) its performance. Saves inference time by 20% or more by eliminating top convolutional layters.

• NLP tasks are usually downstream tasks of finetuning large pretrained transformers models(i.e. BERT, RoBERTa, XLNet).
• Removing top transformers layers may yield 40% reduction in size while preserving up to 98.2% of the performance.
• Likewise, for computer vision's classification task, removing convolutional top layers from pretrained models are tested.

• Will test the performance of replacing convolutional blocks with pretrained weights with a single convolutional layer without pretrained weights.
• Will add ResNet18's inference time data and compare with Optuna's NAS constructed lightweight model.
• Will test on pretrained MobileNetV3, MnasNet on torchvision with elimination based lightweight model architecture search.
• Will be applied on other small datasets such as Fashion MNIST dataset and Plant Village dataset.

• "Empty" stands for model with no pretrained weights.
• "EfficientNet B0 Elim 2" means 2 convolutional blocks have been eliminated from pretrained EfficientNet B0. Number next to "Elim" annotates how many convolutional blocks have been removed.
• Table's performance illustrates best performance out of 100 epochs of finetuning on TACO Dataset.

• @hihellohowareyou
• @lkm2835
• @shawnhyeonsoo
• @snoop2head
• @JoonHong-Kim
• @jjonhwa
• @kimyeondu