PyTorch implementation for "Sharpness-aware Quantization for Deep Neural Networks".

Last update: Dec 19, 2022

Related tags

Deep Learning SAQ

Overview

Sharpness-aware Quantization for Deep Neural Networks

Recent Update

2021.11.23: We release the source code of SAQ.

Setup the environments

Clone the repository locally:

git clone https://github.com/zhuang-group/SAQ

Install pytorch 1.8+, tensorboard and prettytable

conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
pip install tensorboard
pip install prettytable

Data preparation

ImageNet

Download the ImageNet 2012 dataset from here, and prepare the dataset based on this script.
Change the dataset path in link_imagenet.py and link the ImageNet-100 by

python link_imagenet.py

CIFAR-100

Download the CIFAR-100 dataset from here.

After downloading ImageNet and CIFAR-100, the file structure should look like:

dataset
├── imagenet
    ├── train
    │   ├── class1
    │   │   ├── img1.jpeg
    │   │   ├── img2.jpeg
    │   │   └── ...
    │   ├── class2
    │   │   ├── img3.jpeg
    │   │   └── ...
    │   └── ...
    └── val
        ├── class1
        │   ├── img4.jpeg
        │   ├── img5.jpeg
        │   └── ...
        ├── class2
        │   ├── img6.jpeg
        │   └── ...
        └── ...
├── cifar100
    ├── cifar-100-python
    │   ├── meta
    │   ├── test
    │   ├── train
    │   └── ...
    └── ...

Training

Fixed-precision quantization

Download the pre-trained full-precision models from the model zoo.
Train low-precision models.

To train low-precision ResNet-20 on CIFAR-100, run:

sh script/train_qsam_cifar_r20.sh

To train low-precision ResNet-18 on ImageNet, run:

sh script/train_qsam_imagenet_r18.sh

Mixed-precision quantization

Download the pre-trained full-precision models from the model zoo.
Train the configuration generator.

To train the configuration generator of ResNet-20 on CIFAR-100, run:

sh script/train_generator_cifar_r20.sh

To train the configuration generator on ImageNet, run:

sh script/train_generator_imagenet_r18.sh

After training the configuration generator, run following commands to fine-tune the resulting models with the obtained bitwidth configurations on CIFAR-100 and ImageNet.

sh script/finetune_cifar_r20.sh

sh script/finetune_imagenet_r18.sh

Results on CIFAR-100

Network	Method	Bitwidth	BOPs (M)	Top-1 Acc. (%)	Top-5 Acc. (%)
ResNet-20	SAQ	4	674.6	68.7	91.2
ResNet-20	SAMQ	MP	659.3	68.7	91.2
ResNet-20	SAQ	3	392.1	67.7	90.8
ResNet-20	SAMQ	MP	374.4	68.6	91.2
MobileNetV2	SAQ	4	1508.9	75.6	93.7
MobileNetV2	SAMQ	MP	1482.1	75.5	93.6
MobileNetV2	SAQ	3	877.1	74.4	93.2
MobileNetV2	SAMQ	MP	869.5	75.5	93.7

Results on ImageNet

Network	Method	Bitwidth	BOPs (G)	Top-1 Acc. (%)	Top-5 Acc. (%)
ResNet-18	SAQ	4	34.7	71.3	90.0
ResNet-18	SAMQ	MP	33.7	71.4	89.9
ResNet-18	SAQ	2	14.4	67.1	87.3
MobileNetV2	SAQ	4	5.3	70.2	89.4
MobileNetV2	SAMQ	MP	5.3	70.3	89.4

License

This repository is released under the Apache 2.0 license as found in the LICENSE file.

Acknowledgement

This repository has adopted codes from SAM, ASAM and ESAM, we thank the authors for their open-sourced code.

You might also like...

Objective of the repository is to learn and build machine learning models using Pytorch. 30DaysofML Using Pytorch

30 Days Of Machine Learning Using Pytorch Objective of the repository is to learn and build machine learning models using Pytorch. List of Algorithms

119 Nov 24, 2022

Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

1.4k Jan 1, 2023

Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

Amazon Forest Computer Vision Satellite Image tagging code using PyTorch / Keras Here is a sample of images we had to work with Source: https://www.ka

360 Dec 10, 2022

The Incredible PyTorch: a curated list of tutorials, papers, projects, communities and more relating to PyTorch.

This is a curated list of tutorials, projects, libraries, videos, papers, books and anything related to the incredible PyTorch. Feel free to make a pu

9.2k Jan 2, 2023

Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

Amazon Forest Computer Vision Satellite Image tagging code using PyTorch / Keras Here is a sample of images we had to work with Source: https://www.ka

359 Jan 5, 2023

A bunch of random PyTorch models using PyTorch's C++ frontend

PyTorch Deep Learning Models using the C++ frontend Gettting started Clone the repo 1. https://github.com/mrdvince/pytorchcpp 2. cd fashionmnist or

0 Jul 13, 2021

PyTorch Autoencoders - Implementing a Variational Autoencoder (VAE) Series in Pytorch.

PyTorch Autoencoders Implementing a Variational Autoencoder (VAE) Series in Pytorch. Inspired by this repository Model List check model paper conferen

8 Nov 21, 2022

PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices.

PyTorch-LIT PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices. With

157 Dec 11, 2022

A general framework for deep learning experiments under PyTorch based on pytorch-lightning

torchx Torchx is a general framework for deep learning experiments under PyTorch based on pytorch-lightning. TODO list gan-like training wrapper text

6 Mar 17, 2022

Comments

Quantize_first_last_layer

Hi! I noticed that in your code, you set bits_weights=8 and bits_activations=32 for first layer as default, it's not what is claimed in your paper " For the first and last layers of all quantized models, we quantize both weights and activations to 8-bit. " And I see an accuracy drop if I adjust the bits_activations to 8 for the first layer, could u please explain what is the reason? Thanks!

opened by mmmiiinnnggg 0
代码问题请求帮助

你好，带佬的代码写的很好，有部分代码不太懂，想请教一下， parser.add_argument( "--arch_bits", type=lambda s: [float(item) for item in s.split(",")] if len(s) != 0 else "", default=" ", help="bits configuration of each layer",

if len(args.arch_bits) != 0: if args.wa_same_bit: set_wae_bits(model, args.arch_bits) elif args.search_w_bit: set_w_bits(model, args.arch_bits) else: set_bits(model, args.arch_bits) show_bits(model) logger.info("Set arch bits to: {}".format(args.arch_bits)) logger.info(model) 这个arch_bits主要是做什么的呢，卡在这里有段时间了

opened by LKAMING97 0

PyTorch implementation for "Sharpness-aware Quantization for Deep Neural Networks".

Related tags

Overview

Sharpness-aware Quantization for Deep Neural Networks

Recent Update

Setup the environments

Data preparation

ImageNet

CIFAR-100

Training

Fixed-precision quantization

Mixed-precision quantization

Results on CIFAR-100

Results on ImageNet

License

Acknowledgement

You might also like...

Objective of the repository is to learn and build machine learning models using Pytorch. 30DaysofML Using Pytorch

Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

The Incredible PyTorch: a curated list of tutorials, papers, projects, communities and more relating to PyTorch.

Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

A bunch of random PyTorch models using PyTorch's C++ frontend

PyTorch Autoencoders - Implementing a Variational Autoencoder (VAE) Series in Pytorch.

PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices.

A general framework for deep learning experiments under PyTorch based on pytorch-lightning

Comments

Quantize_first_last_layer

代码问题请求帮助

Releases(v0.1.1)

v0.1.1(Nov 23, 2021)

v0.1(Nov 23, 2021)

Owner

Zhuang AI Group

Datasets, Transforms and Models specific to Computer Vision

ByteTrack: Multi-Object Tracking by Associating Every Detection Box

Python library to receive live stream events like comments and gifts in realtime from TikTok LIVE.

System Combination for Grammatical Error Correction Based on Integer Programming

Text Extraction Formulation + Feedback Loop for state-of-the-art WSD (EMNLP 2021)

Winners of the Facebook Image Similarity Challenge

A new play-and-plug method of controlling an existing generative model with conditioning attributes and their compositions.

Hierarchical-Bayesian-Defense - Towards Adversarial Robustness of Bayesian Neural Network through Hierarchical Variational Inference (Openreview)

Code for the USENIX 2017 paper: kAFL: Hardware-Assisted Feedback Fuzzing for OS Kernels

This is my codes that can visualize the psnr image in testing videos.

RIFE: Real-Time Intermediate Flow Estimation for Video Frame Interpolation

Implementation of " SESS: Self-Ensembling Semi-Supervised 3D Object Detection" (CVPR2020 Oral)

Defocus Map Estimation and Deblurring from a Single Dual-Pixel Image

A highly modular PyTorch framework with a focus on Neural Architecture Search (NAS).

Feature board for ERPNext

Implementation of MeMOT - Multi-Object Tracking with Memory - in Pytorch

A real world application of a Recurrent Neural Network on a binary classification of time series data

Fully Convolutional Networks for Semantic Segmentation by Jonathan Long*, Evan Shelhamer*, and Trevor Darrell. CVPR 2015 and PAMI 2016.

Official repository for MixFaceNets: Extremely Efficient Face Recognition Networks

My 1st place solution at Kaggle Hotel-ID 2021

Fully Convolutional Networks for Semantic Segmentation by Jonathan Long, Evan Shelhamer, and Trevor Darrell. CVPR 2015 and PAMI 2016.