Super-Fast-Adversarial-Training

This is a PyTorch Implementation code for developing super fast adversarial training. This code is combined with below state-of-the-art technologies for accelerating adversarial attacks and defenses with Deep Neural Networks on Volta GPU architecture.

• Distributed Data Parallel [link]
• Channel Last Memory Format [link]
• Mixed Precision Training [link]
• Mixed Precision + Adversarial Attack (based on torchattacks [link])
• Faster Adversarial Training for Large Dataset [link]
• Fast Forward Computer Vision (FFCV) [link]

Citation

If you find this work helpful, please cite it as:

@software{SuperFastAT_ByungKwanLee_2022,
  author = {Byung-Kwan Lee},
  title = {Super-Fast-Adversarial-Training},
  url = {https://github.com/ByungKwanLee/Super-Fast-Adversarial-Training},
  version = {alpha},
  year = {2022}
}

Library for Fast Adversarial Attacks

This library is developed based on the well-known package of torchattacks [link] due to its simple scalability.

Under Developement (Current Available Attacks Below)

• Fast Gradient Sign Method (FGSM)
• Projected Gradient Descent (PGD)

Environment Setting

Please check below settings to successfully run this code. If not, follow step by step during filling the checklist in.

• To utilize FFCV [link], you should install it on conda virtual environment. I use python version 3.8, pytorch 1.7.1, torchvision 0.8.2, and cuda 10.1. For more different version, you can refer to PyTorch official site [link].

conda create -y -n ffcv python=3.8 cupy pkg-config compilers libjpeg-turbo opencv pytorch==1.7.1 torchvision==0.8.2 cudatoolkit=10.1 numba -c pytorch -c conda-forge

• Activate the created environment by conda

conda activate ffcv

• And, it would be better to install cudnn to more accelerate GPU. (Optional)

conda install cudnn -c conda-forge

• To install FFCV, you should download it in pip and install torchattacks [link] to run adversarial attack.

pip install ffcv torchattacks==3.1.0

• To guarantee the execution of this code, please additionally install library in requirements.txt (matplotlib, tqdm)

pip install -r requirements.txt

Available Datasets

• CIFAR-10
• CIFAR-100
• SVHN
• Tiny-ImageNet
• ImageNet

Available Baseline Models

• VGG (model/vgg.py)
• ResNet (model/resnet.py)
• WideResNet (model/wide.py)
• DenseNet (model/dense.py)

How to run

After making completion of environment settings, then you can follow how to run below.

• First, run fast_dataset_converter.py to generate dataset with .betson extension, instead of using original dataset [FFCV].

# Future import build
from __future__ import print_function

# Import built-in module
import os
import argparse

# fetch args
parser = argparse.ArgumentParser()

# parameter
parser.add_argument('--dataset', default='imagenet', type=str)
parser.add_argument('--gpu', default='0', type=str)
args = parser.parse_args()

# GPU configurations
os.environ["CUDA_VISIBLE_DEVICES"]=args.gpu

# init fast dataloader
from utils.fast_data_utils import save_data_for_beton
save_data_for_beton(dataset=args.dataset)

• Second, run fast_pretrain_standard.py(Standard Training) or fast_pretrain_adv.py (Adversarial Training)

# model parameter
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--dataset', default='imagenet', type=str)
parser.add_argument('--network', default='resnet', type=str)
parser.add_argument('--depth', default=50, type=int)
parser.add_argument('--gpu', default='0,1,2,3,4', type=str)

# learning parameter
parser.add_argument('--learning_rate', default=0.1, type=float)
parser.add_argument('--weight_decay', default=0.0002, type=float)
parser.add_argument('--batch_size', default=512, type=float)
parser.add_argument('--test_batch_size', default=128, type=float)
parser.add_argument('--epoch', default=100, type=int)

or

# model parameter
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--dataset', default='imagenet', type=str)
parser.add_argument('--network', default='resnet', type=str)
parser.add_argument('--depth', default=18, type=int)
parser.add_argument('--gpu', default='0,1,2,3,4', type=str)

# learning parameter
parser.add_argument('--learning_rate', default=0.1, type=float)
parser.add_argument('--weight_decay', default=0.0002, type=float)
parser.add_argument('--batch_size', default=1024, type=float)
parser.add_argument('--test_batch_size', default=512, type=float)
parser.add_argument('--epoch', default=60, type=int)

# attack parameter
parser.add_argument('--attack', default='pgd', type=str)
parser.add_argument('--eps', default=0.03, type=float)
parser.add_argument('--steps', default=10, type=int)

To-do

I have plans to make a variety of functions to be a standard framework for adversarial training.

• Many Compatible Adversarial Attacks and Defenses
• Super Fast Evaluation and Validating its Compatibility
• Re-Arrangement of class and function for code readability
• Providing Checkpoints per dataset and model to reduce your own time