RVT: Robust Vision Transformers

This repository contains PyTorch code for Robust Vision Transformers.

For details see Rethinking the Design Principles of Robust Vision Transformer by Xiaofeng Mao, Gege Qi, Yuefeng Chen, Yuan He and Hui Xue.

Usage

First, clone the repository locally:

git clone https://github.com/vtddggg/Robust-Vision-Transformer.git

Then, install PyTorch 1.7.0+ and torchvision 0.8.1+ and pytorch-image-models 0.3.2:

conda install -c pytorch pytorch torchvision
pip install timm==0.3.2

We use 4 nodes with 8 gpus to train RVT-Ti, RVT-S and RVT-B:

Training RVT-Ti

python -m torch.distributed.launch --nproc_per_node=8 --nnodes=4 main.py --model rvt_tiny --data-path /path/to/imagenet --output_dir output --dist-eval

Training RVT-S

python -m torch.distributed.launch --nproc_per_node=8 --nnodes=4 main.py --model rvt_small --data-path /path/to/imagenet --output_dir output --dist-eval

Training RVT-B

python -m torch.distributed.launch --nproc_per_node=8 --nnodes=4 main.py --model rvt_base --data-path /path/to/imagenet --output_dir output --batch-size 32 --dist-eval

If you want to train RVT-Ti*, RVT-S* or RVT-B*, simply add --use_mask and --use_patch_aug to enable positon-aware attention scaling and patch-wise augmentation.

This repository contains PyTorch code for Robust Vision Transformers.

Related tags

Overview

RVT: Robust Vision Transformers

Usage

Training RVT-Ti

Training RVT-S

Training RVT-B

Owner

Database Reasoning Over Text project for ACL paper

VLG-Net: Video-Language Graph Matching Networks for Video Grounding

Video Instance Segmentation using Inter-Frame Communication Transformers (NeurIPS 2021)

Robustness between the worst and average case

Deep Ensemble Learning with Jet-Like architecture

Red Team tool for exfiltrating files from a target's Google Drive that you have access to, via Google's API.

Personalized Transfer of User Preferences for Cross-domain Recommendation (PTUPCDR)

Tackling Obstacle Tower Challenge using PPO & A2C combined with ICM.

Trained on Simulated Data, Tested in the Real World

Adversarial Adaptation with Distillation for BERT Unsupervised Domain Adaptation

Lava-DL, but with PyTorch-Lightning flavour

Source code of the paper PatchGraph: In-hand tactile tracking with learned surface normals.

Object Detection and Multi-Object Tracking

VQGAN+CLIP Colab Notebook with user-friendly interface.

Black-Box-Tuning - Black-Box Tuning for Language-Model-as-a-Service

3D2Unet: 3D Deformable Unet for Low-Light Video Enhancement (PRCV2021)

Using pretrained GROVER to extract the atomic fingerprints from molecule

🔎 Super-scale your images and run experiments with Residual Dense and Adversarial Networks.

Use graph-based analysis to re-classify stocks and to improve Markowitz portfolio optimization

PyTorch implementation of: Michieli U. and Zanuttigh P., "Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations", CVPR 2021.