Dynamic Divide-and-Conquer Adversarial Training for Robust Semantic Segmentation (ICCV2021)

Overview

Dynamic Divide-and-Conquer Adversarial Training for Robust Semantic Segmentation

This is a pytorch project for the paper Dynamic Divide-and-Conquer Adversarial Training for Robust Semantic Segmentation by Xiaogang Xu, Hengshuang Zhao and Jiaya Jia presented at ICCV2021.

paper link, arxiv

Introduction

Adversarial training is promising for improving the robustness of deep neural networks towards adversarial perturbations, especially on the classification task. The effect of this type of training on semantic segmentation, contrarily, just commences. We make the initial attempt to explore the defense strategy on semantic segmentation by formulating a general adversarial training procedure that can perform decently on both adversarial and clean samples. We propose a dynamic divide-and-conquer adversarial training (DDC-AT) strategy to enhance the defense effect, by setting additional branches in the target model during training, and dealing with pixels with diverse properties towards adversarial perturbation. Our dynamical division mechanism divides pixels into multiple branches automatically. Note all these additional branches can be abandoned during inference and thus leave no extra parameter and computation cost. Extensive experiments with various segmentation models are conducted on PASCAL VOC 2012 and Cityscapes datasets, in which DDC-AT yields satisfying performance under both white- and black-box attacks.

Project Setup

For multiprocessing training, we use apex, tested with pytorch 1.0.1.

First install Python 3. We advise you to install Python 3 and PyTorch with Anaconda:

conda create --name py36 python=3.6
source activate py36

Clone the repo and install the complementary requirements:

cd $HOME
git clone --recursive [email protected]:dvlab-research/Robust_Semantic_Segmentation.git
cd Robust_Semantic_Segmentation
pip install -r requirements.txt

The environment of our experiments is CUDA10.2 and TITAN V. And you should install apex for training.

Requirement

  • Hardware: 4-8 GPUs (better with >=11G GPU memory)

Train

  • Download related datasets and you should modify the relevant paths specified in folder "config"
  • Download ImageNet pre-trained models and put them under folder initmodel for weight initialization.

Cityscapes

  • Train the baseline model with no defense on Cityscapes with PSPNet
    sh tool_train/cityscapes/psp_train.sh
    
  • Train the baseline model with no defense on Cityscapes with DeepLabv3
    sh tool_train/cityscapes/aspp_train.sh
    
  • Train the model with SAT on Cityscapes with PSPNet
    sh tool_train/cityscapes/psp_train_sat.sh
    
  • Train the model with SAT on Cityscapes with DeepLabv3
    sh tool_train/cityscapes/aspp_train_sat.sh
    
  • Train the model with DDCAT on Cityscapes with PSPNet
    sh tool_train/cityscapes/psp_train_ddcat.sh
    
  • Train the model with DDCAT on Cityscapes with DeepLabv3
    sh tool_train/cityscapes/aspp_train_ddcat.sh
    

VOC2012

  • Train the baseline model with no defense on VOC2012 with PSPNet
    sh tool_train/voc2012/psp_train.sh
    
  • Train the baseline model with no defense on VOC2012 with DeepLabv3
    sh tool_train/voc2012/aspp_train.sh
    
  • Train the model with SAT on VOC2012 with PSPNet
    sh tool_train/voc2012/psp_train_sat.sh
    
  • Train the model with SAT on VOC2012 with DeepLabv3
    sh tool_train/voc2012/aspp_train_sat.sh
    
  • Train the model with DDCAT on VOC2012 with PSPNet
    sh tool_train/voc2012/psp_train_ddcat.sh
    
  • Train the model with DDCAT on VOC2012 with DeepLabv3
    sh tool_train/voc2012/aspp_train_ddcat.sh
    

You can use the tensorboardX to visualize the training loss, by

tensorboard --logdir=exp/path_to_log

Test

We provide the script for evaluation, reporting the miou on both clean and adversarial samples (the adversarial samples are obtained with attack whose n=2, epsilon=0.03 x 255, alpha=0.01 x 255)

Cityscapes

  • Evaluate the PSPNet trained with no defense on Cityscapes
    sh tool_test/cityscapes/psp_test.sh
    
  • Evaluate the PSPNet trained with SAT on Cityscapes
    sh tool_test/cityscapes/psp_test_sat.sh
    
  • Evaluate the PSPNet trained with DDCAT on Cityscapes
    sh tool_test/cityscapes/psp_test_ddcat.sh
    
  • Evaluate the DeepLabv3 trained with no defense on Cityscapes
    sh tool_test/cityscapes/aspp_test.sh
    
  • Evaluate the DeepLabv3 trained with SAT on Cityscapes
    sh tool_test/cityscapes/aspp_test_sat.sh
    
  • Evaluate the DeepLabv3 trained with DDCAT on Cityscapes
    sh tool_test/cityscapes/aspp_test_ddcat.sh
    

VOC2012

  • Evaluate the PSPNet trained with no defense on VOC2012
    sh tool_test/voc2012/psp_test.sh
    
  • Evaluate the PSPNet trained with SAT on VOC2012
    sh tool_test/voc2012/psp_test_sat.sh
    
  • Evaluate the PSPNet trained with DDCAT on VOC2012
    sh tool_test/voc2012/psp_test_ddcat.sh
    
  • Evaluate the DeepLabv3 trained with no defense on VOC2012
    sh tool_test/voc2012/aspp_test.sh
    
  • Evaluate the DeepLabv3 trained with SAT on VOC2012
    sh tool_test/voc2012/aspp_test_sat.sh
    
  • Evaluate the DeepLabv3 trained with DDCAT on VOC2012
    sh tool_test/voc2012/aspp_test_ddcat.sh
    

Pretrained Model

You can download the pretrained models from https://drive.google.com/file/d/120xLY_pGZlm3tqaLxTLVp99e06muBjJC/view?usp=sharing

Cityscapes with PSPNet

The model trained with no defense: pretrain/cityscapes/pspnet/no_defense
The model trained with SAT: pretrain/cityscapes/pspnet/sat
The model trained with DDCAT: pretrain/cityscapes/pspnet/ddcat

Cityscapes with DeepLabv3

The model trained with no defense: pretrain/cityscapes/deeplabv3/no_defense
The model trained with SAT: pretrain/cityscapes/deeplabv3/sat
The model trained with DDCAT: pretrain/cityscapes/deeplabv3/ddcat

VOC2012 with PSPNet

The model trained with no defense: pretrain/voc2012/pspnet/no_defense
The model trained with SAT: pretrain/voc2012/pspnet/sat
The model trained with DDCAT: pretrain/voc2012/pspnet/ddcat

VOC2012 with DeepLabv3

The model trained with no defense: pretrain/voc2012/deeplabv3/no_defense
The model trained with SAT: pretrain/voc2012/deeplabv3/sat
The model trained with DDCAT: pretrain/voc2012/deeplabv3/ddcat

Citation Information

If you find the project useful, please cite:

@inproceedings{xu2021ddcat,
  title={Dynamic Divide-and-Conquer Adversarial Training for Robust Semantic Segmentation},
  author={Xiaogang Xu, Hengshuang Zhao and Jiaya Jia},
  booktitle={ICCV},
  year={2021}
}

Acknowledgments

This source code is inspired by semseg.

Contributions

If you have any questions/comments/bug reports, feel free to e-mail the author Xiaogang Xu ([email protected]).

Owner
DV Lab
Deep Vision Lab
DV Lab
A simple implementation of Kalman filter in single object tracking

kalman-filter-in-single-object-tracking A simple implementation of Kalman filter in single object tracking https://www.bilibili.com/video/BV1Qf4y1J7D4

130 Dec 26, 2022
Repository for the Bias Benchmark for QA dataset.

BBQ Repository for the Bias Benchmark for QA dataset. Authors: Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Tho

ML² AT CILVR 18 Nov 18, 2022
[ICCV 2021] Excavating the Potential Capacity of Self-Supervised Monocular Depth Estimation

EPCDepth EPCDepth is a self-supervised monocular depth estimation model, whose supervision is coming from the other image in a stereo pair. Details ar

Rui Peng 110 Dec 23, 2022
Rethinking Transformer-based Set Prediction for Object Detection

Rethinking Transformer-based Set Prediction for Object Detection Here are the code for the ICCV paper. The code is adapted from Detectron2 and AdelaiD

Zhiqing Sun 62 Dec 03, 2022
Self-Supervised depth kalilia

Self-Supervised depth kalilia

24 Oct 15, 2022
Easy-to-use micro-wrappers for Gym and PettingZoo based RL Environments

SuperSuit introduces a collection of small functions which can wrap reinforcement learning environments to do preprocessing ('microwrappers'). We supp

Farama Foundation 357 Jan 06, 2023
git《Commonsense Knowledge Base Completion with Structural and Semantic Context》(AAAI 2020) GitHub: [fig1]

Commonsense Knowledge Base Completion with Structural and Semantic Context Code for the paper Commonsense Knowledge Base Completion with Structural an

AI2 96 Nov 05, 2022
Augmented CLIP - Training simple models to predict CLIP image embeddings from text embeddings, and vice versa.

Train aug_clip against laion400m-embeddings found here: https://laion.ai/laion-400-open-dataset/ - note that this used the base ViT-B/32 CLIP model. S

Peter Baylies 55 Sep 13, 2022
A novel method to tune language models. Codes and datasets for paper ``GPT understands, too''.

P-tuning A novel method to tune language models. Codes and datasets for paper ``GPT understands, too''. How to use our code We have released the code

THUDM 562 Dec 27, 2022
Reinforcement learning for self-driving in a 3D simulation

SelfDrive_AI Reinforcement learning for self-driving in a 3D simulation (Created using UNITY-3D) 1. Requirements for the SelfDrive_AI Gym You need Pyt

Surajit Saikia 17 Dec 14, 2021
Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes

Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized C

Sam Bond-Taylor 139 Jan 04, 2023
Trading environnement for RL agents, backtesting and training.

TradzQAI Trading environnement for RL agents, backtesting and training. Live session with coinbasepro-python is finaly arrived ! Available sessions: L

Tony Denion 164 Oct 30, 2022
Code for You Only Cut Once: Boosting Data Augmentation with a Single Cut

You Only Cut Once (YOCO) YOCO is a simple method/strategy of performing augmenta

88 Dec 28, 2022
Reproduce results and replicate training fo T0 (Multitask Prompted Training Enables Zero-Shot Task Generalization)

T-Zero This repository serves primarily as codebase and instructions for training, evaluation and inference of T0. T0 is the model developed in Multit

BigScience Workshop 253 Dec 27, 2022
Data manipulation and transformation for audio signal processing, powered by PyTorch

torchaudio: an audio library for PyTorch The aim of torchaudio is to apply PyTorch to the audio domain. By supporting PyTorch, torchaudio follows the

1.9k Dec 28, 2022
Code release for the ICML 2021 paper "PixelTransformer: Sample Conditioned Signal Generation".

PixelTransformer Code release for the ICML 2021 paper "PixelTransformer: Sample Conditioned Signal Generation". Project Page Installation Please insta

Shubham Tulsiani 24 Dec 17, 2022
Ἀνατομή is a PyTorch library to analyze representation of neural networks

Ἀνατομή is a PyTorch library to analyze representation of neural networks

Ryuichiro Hataya 50 Dec 05, 2022
Music Source Separation; Train & Eval & Inference piplines and pretrained models we used for 2021 ISMIR MDX Challenge.

Introduction 1. Usage (For MSS) 1.1 Prepare running environment 1.2 Use pretrained model 1.3 Train new MSS models from scratch 1.3.1 How to train 1.3.

Leo 100 Dec 25, 2022
Source code of generalized shuffled linear regression

Generalized-Shuffled-Linear-Regression Code for the ICCV 2021 paper: Generalized Shuffled Linear Regression. Authors: Feiran Li, Kent Fujiwara, Fumio

FEI 7 Oct 26, 2022
Code for "Finding Regions of Heterogeneity in Decision-Making via Expected Conditional Covariance" at NeurIPS 2021

Finding Regions of Heterogeneity in Decision-Making via Expected Conditional Covariance Justin Lim, Christina X Ji, Michael Oberst, Saul Blecker, Leor

Sontag Lab 3 Feb 03, 2022