PyTorch implementation for "Sharpness-aware Quantization for Deep Neural Networks".

Last update: Dec 19, 2022

Related tags

Deep Learning SAQ

Overview

Sharpness-aware Quantization for Deep Neural Networks

Recent Update

2021.11.23: We release the source code of SAQ.

Setup the environments

Clone the repository locally:

git clone https://github.com/zhuang-group/SAQ

Install pytorch 1.8+, tensorboard and prettytable

conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
pip install tensorboard
pip install prettytable

Data preparation

ImageNet

Download the ImageNet 2012 dataset from here, and prepare the dataset based on this script.
Change the dataset path in link_imagenet.py and link the ImageNet-100 by

python link_imagenet.py

CIFAR-100

Download the CIFAR-100 dataset from here.

After downloading ImageNet and CIFAR-100, the file structure should look like:

dataset
├── imagenet
    ├── train
    │   ├── class1
    │   │   ├── img1.jpeg
    │   │   ├── img2.jpeg
    │   │   └── ...
    │   ├── class2
    │   │   ├── img3.jpeg
    │   │   └── ...
    │   └── ...
    └── val
        ├── class1
        │   ├── img4.jpeg
        │   ├── img5.jpeg
        │   └── ...
        ├── class2
        │   ├── img6.jpeg
        │   └── ...
        └── ...
├── cifar100
    ├── cifar-100-python
    │   ├── meta
    │   ├── test
    │   ├── train
    │   └── ...
    └── ...

Training

Fixed-precision quantization

Download the pre-trained full-precision models from the model zoo.
Train low-precision models.

To train low-precision ResNet-20 on CIFAR-100, run:

sh script/train_qsam_cifar_r20.sh

To train low-precision ResNet-18 on ImageNet, run:

sh script/train_qsam_imagenet_r18.sh

Mixed-precision quantization

Download the pre-trained full-precision models from the model zoo.
Train the configuration generator.

To train the configuration generator of ResNet-20 on CIFAR-100, run:

sh script/train_generator_cifar_r20.sh

To train the configuration generator on ImageNet, run:

sh script/train_generator_imagenet_r18.sh

After training the configuration generator, run following commands to fine-tune the resulting models with the obtained bitwidth configurations on CIFAR-100 and ImageNet.

sh script/finetune_cifar_r20.sh

sh script/finetune_imagenet_r18.sh

Results on CIFAR-100

Network	Method	Bitwidth	BOPs (M)	Top-1 Acc. (%)	Top-5 Acc. (%)
ResNet-20	SAQ	4	674.6	68.7	91.2
ResNet-20	SAMQ	MP	659.3	68.7	91.2
ResNet-20	SAQ	3	392.1	67.7	90.8
ResNet-20	SAMQ	MP	374.4	68.6	91.2
MobileNetV2	SAQ	4	1508.9	75.6	93.7
MobileNetV2	SAMQ	MP	1482.1	75.5	93.6
MobileNetV2	SAQ	3	877.1	74.4	93.2
MobileNetV2	SAMQ	MP	869.5	75.5	93.7

Results on ImageNet

Network	Method	Bitwidth	BOPs (G)	Top-1 Acc. (%)	Top-5 Acc. (%)
ResNet-18	SAQ	4	34.7	71.3	90.0
ResNet-18	SAMQ	MP	33.7	71.4	89.9
ResNet-18	SAQ	2	14.4	67.1	87.3
MobileNetV2	SAQ	4	5.3	70.2	89.4
MobileNetV2	SAMQ	MP	5.3	70.3	89.4

License

This repository is released under the Apache 2.0 license as found in the LICENSE file.

Acknowledgement

This repository has adopted codes from SAM, ASAM and ESAM, we thank the authors for their open-sourced code.

You might also like...

Objective of the repository is to learn and build machine learning models using Pytorch. 30DaysofML Using Pytorch

30 Days Of Machine Learning Using Pytorch Objective of the repository is to learn and build machine learning models using Pytorch. List of Algorithms

119 Nov 24, 2022

Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

1.4k Jan 1, 2023

Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

Amazon Forest Computer Vision Satellite Image tagging code using PyTorch / Keras Here is a sample of images we had to work with Source: https://www.ka

360 Dec 10, 2022

The Incredible PyTorch: a curated list of tutorials, papers, projects, communities and more relating to PyTorch.

This is a curated list of tutorials, projects, libraries, videos, papers, books and anything related to the incredible PyTorch. Feel free to make a pu

9.2k Jan 2, 2023

Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

Amazon Forest Computer Vision Satellite Image tagging code using PyTorch / Keras Here is a sample of images we had to work with Source: https://www.ka

359 Jan 5, 2023

A bunch of random PyTorch models using PyTorch's C++ frontend

PyTorch Deep Learning Models using the C++ frontend Gettting started Clone the repo 1. https://github.com/mrdvince/pytorchcpp 2. cd fashionmnist or

0 Jul 13, 2021

PyTorch Autoencoders - Implementing a Variational Autoencoder (VAE) Series in Pytorch.

PyTorch Autoencoders Implementing a Variational Autoencoder (VAE) Series in Pytorch. Inspired by this repository Model List check model paper conferen

8 Nov 21, 2022

PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices.

PyTorch-LIT PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices. With

157 Dec 11, 2022

A general framework for deep learning experiments under PyTorch based on pytorch-lightning

torchx Torchx is a general framework for deep learning experiments under PyTorch based on pytorch-lightning. TODO list gan-like training wrapper text

6 Mar 17, 2022

Comments

Quantize_first_last_layer

Hi! I noticed that in your code, you set bits_weights=8 and bits_activations=32 for first layer as default, it's not what is claimed in your paper " For the first and last layers of all quantized models, we quantize both weights and activations to 8-bit. " And I see an accuracy drop if I adjust the bits_activations to 8 for the first layer, could u please explain what is the reason? Thanks!

opened by mmmiiinnnggg 0
代码问题请求帮助

你好，带佬的代码写的很好，有部分代码不太懂，想请教一下， parser.add_argument( "--arch_bits", type=lambda s: [float(item) for item in s.split(",")] if len(s) != 0 else "", default=" ", help="bits configuration of each layer",

if len(args.arch_bits) != 0: if args.wa_same_bit: set_wae_bits(model, args.arch_bits) elif args.search_w_bit: set_w_bits(model, args.arch_bits) else: set_bits(model, args.arch_bits) show_bits(model) logger.info("Set arch bits to: {}".format(args.arch_bits)) logger.info(model) 这个arch_bits主要是做什么的呢，卡在这里有段时间了

opened by LKAMING97 0

PyTorch implementation for "Sharpness-aware Quantization for Deep Neural Networks".

Related tags

Overview

Sharpness-aware Quantization for Deep Neural Networks

Recent Update

Setup the environments

Data preparation

ImageNet

CIFAR-100

Training

Fixed-precision quantization

Mixed-precision quantization

Results on CIFAR-100

Results on ImageNet

License

Acknowledgement

You might also like...

Objective of the repository is to learn and build machine learning models using Pytorch. 30DaysofML Using Pytorch

Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

The Incredible PyTorch: a curated list of tutorials, papers, projects, communities and more relating to PyTorch.

Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

A bunch of random PyTorch models using PyTorch's C++ frontend

PyTorch Autoencoders - Implementing a Variational Autoencoder (VAE) Series in Pytorch.

PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices.

A general framework for deep learning experiments under PyTorch based on pytorch-lightning

Comments

Quantize_first_last_layer

代码问题请求帮助

Releases(v0.1.1)

v0.1.1(Nov 23, 2021)

v0.1(Nov 23, 2021)

Owner

Zhuang AI Group

MAUS: A Dataset for Mental Workload Assessment Using Wearable Sensor - Baseline system

The official implementation of the IEEE S&P`22 paper "SoK: How Robust is Deep Neural Network Image Classification Watermarking".

functorch is a prototype of JAX-like composable function transforms for PyTorch.

Measures input lag without dedicated hardware, performing motion detection on recorded or live video

Efficiently computes derivatives of numpy code.

[ICCV'21] UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction

Educational 2D SLAM implementation based on ICP and Pose Graph

This repository is the official implementation of Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (NeurIPS21).

Aws-machine-learning-university-accelerated-tab - Machine Learning University: Accelerated Tabular Data Class

Revealing and Protecting Labels in Distributed Training

Autonomous racing with the Anki Overdrive

ICON: Implicit Clothed humans Obtained from Normals

Lux AI environment interface for RLlib multi-agents

Improving adversarial robustness by a coupling rejection strategy

Pretrained models for Jax/Flax: StyleGAN2, GPT2, VGG, ResNet.

Codes for NAACL 2021 Paper "Unsupervised Multi-hop Question Answering by Question Generation"

Python script that analyses the given datasets and comes up with the best polynomial regression representation with the smallest polynomial degree possible

Learning Time-Critical Responses for Interactive Character Control

This project provides the proof of the uniqueness of the equilibrium and the global asymptotic stability.

An addon uses SMPL's poses and global translation to drive cartoon character in Blender.