Efficient Sharpness-aware Minimization for Improved Training of Neural Networks

Code for “Efficient Sharpness-aware Minimization for Improved Training of Neural Networks”

Requisite

This code is implemented in PyTorch, and we have tested the code under the following environment settings:

python = 3.8.8
torch = 1.8.0
torchvision = 0.9.0

What is in this repository

Codes for our ESAM on CIFAR10/CIFAR100 datasets.

How to use it

from utils.layer_dp_sam import ESAM
base_optimizer = torch.optim.SGD(model.parameters(),lr=args.learning_rate,momentum=0.9,weight_decay=args.weight_decay)
optimizer = ESAM(paras, base_optimizer, rho=args.rho, weight_dropout=args.weight_dropout,adaptive=args.isASAM,nograd_cutoff=args.nograd_cutoff,opt_dropout = args.opt_dropout,temperature=args.temperature)

--beta the SWP hyperparameter

--gamma the SDS hyperparameter

During training loss_fct should have reduction="none", to return instance-wise losses. defined_backward is the function used for DDP and mixed precision backward

loss_fct = torch.nn.CrossEntropyLoss(reduction="none")
def defined_backward():
    if args.fp16:
    with amp.scale_loss(loss, optimizer0) as scaled_loss:
        scaled_loss.backward()
    else:
        loss.backward()

paras = [inputs,targets,loss_fct,model,defined_backward]
optimizer.paras = paras
optimizer.step()
predictions_logits,loss = optimizer.returnthings

Example

bash run.sh

Reference Code

[1] SAM

Efficient Sharpness-aware Minimization for Improved Training of Neural Networks

Related tags

Overview

Efficient Sharpness-aware Minimization for Improved Training of Neural Networks

Requisite

What is in this repository

How to use it

Example

Reference Code

Owner

Angusdu

RetinaFace: Deep Face Detection Library in TensorFlow for Python

Official code repository of the paper Learning Associative Inference Using Fast Weight Memory by Schlag et al.

A DeepStack custom model for detecting common objects in dark/night images and videos.

Computational inteligence project on faces in the wild dataset

[ICCV 2021] Official Tensorflow Implementation for "Single Image Defocus Deblurring Using Kernel-Sharing Parallel Atrous Convolutions"

Anomaly Detection Based on Hierarchical Clustering of Mobile Robot Data

Official implementation of the ICLR 2021 paper

Data, model training, and evaluation code for "PubTables-1M: Towards a universal dataset and metrics for training and evaluating table extraction models".

[CVPR'21] Locally Aware Piecewise Transformation Fields for 3D Human Mesh Registration

Decoding the Protein-ligand Interactions Using Parallel Graph Neural Networks

MegEngine implementation of YOLOX

Negative Sample Matters: A Renaissance of Metric Learning for Temporal Grounding

Replication Code for "Self-Supervised Bug Detection and Repair" NeurIPS 2021

Source code for Task-Aware Variational Adversarial Active Learning

Few-Shot Graph Learning for Molecular Property Prediction

Official pytorch implementation of DeformSyncNet: Deformation Transfer via Synchronized Shape Deformation Spaces

Near-Optimal Sparse Allreduce for Distributed Deep Learning (published in PPoPP'22)

Pre-Trained Image Processing Transformer (IPT)

An educational resource to help anyone learn deep reinforcement learning.

An implementation demo of the ICLR 2021 paper Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks in PyTorch.