PyTorch implementation of the paper Dynamic Token Normalization Improves Vision Transfromers.

Last update: Oct 09, 2022

Related tags

Overview

Dynamic Token Normalization Improves Vision Transformers

This is the PyTorch implementation of the paper Dynamic Token Normalization Improves Vision Transfromers. Codea and Models will be available soon.

Dynamic Token Normalization

We design a novel normalization method, termed Dynamic Token Normalization (DTN), which inherits the advantages from LayerNorm and InstanceNorm. DTN can be seamlessly plugged into various transformer models, consistenly improving the performance.

Comparisons of top-1 accuracies on the validation set of ImageNet, by using ViT trained with LN and DTN.

Model	Top-1	Top-5
ViT-T*-LN	72.3	91.4
ViT-T*-DTN	73.2	91.7
ViT-S*-LN	80.6	95.2
ViT-S*-DTN	81.7	95.8
ViT-B*-LN	81.7	95.8
ViT-B*-DTN	82.5	96.1

Getting Started

Install PyTorch

Clone the repo:

git clone https://github.com/dtn-anonymous/DTN.git

Requirements

Install CUDA==10.1 with cudnn7 following the official installation instructions
Install PyTorch==1.7.1 and torchvision==0.8.2 with CUDA==10.1:

conda install pytorch==1.7.1 torchvision==0.8.2 cudatoolkit=10.1 -c pytorch

Install timm==0.3.2:

pip install timm==0.3.2

Data Preparation

Download the ImageNet dataset which should contain train and val directionary and the txt file for correspondings between images and labels.

Training a model from scratch

An example to train our DTN is given in DTN/scripts/train.sh. To train ViT-S* with our DTN,

cd DTN/scripts   
sh train.sh layer vit_norm_s_star configs/ViT/vit.yaml

Number of GPUs and configuration file to use can be modified in train.sh

PyTorch implementation of the paper Dynamic Token Normalization Improves Vision Transfromers.

Related tags

Overview

Dynamic Token Normalization Improves Vision Transformers

Dynamic Token Normalization

Getting Started

Requirements

Data Preparation

Training a model from scratch

Owner

Wenqi Shao

Official codes: Self-Supervised Learning by Estimating Twin Class Distribution

Awesome Deep Graph Clustering is a collection of SOTA, novel deep graph clustering methods

Markov Attention Models

Perform Linear Classification with Multi-way Data

An example of time series augmentation methods with Keras

Code to reproduce the experiments from our NeurIPS 2021 paper " The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective"

Keep CALM and Improve Visual Feature Attribution

This repository contains answers of the Shopify Summer 2022 Data Science Intern Challenge.

PyTorch implementation for the paper Visual Representation Learning with Self-Supervised Attention for Low-Label High-Data Regime

Learning Dynamic Network Using a Reuse Gate Function in Semi-supervised Video Object Segmentation.

PyTorch implementation of DreamerV2 model-based RL algorithm

Implementation of the paper "Shapley Explanation Networks"

[CVPR'21] FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Space

Robust Video Matting in PyTorch, TensorFlow, TensorFlow.js, ONNX, CoreML!

A Python multilingual toolkit for Sentiment Analysis and Social NLP tasks

High performance distributed framework for training deep learning recommendation models based on PyTorch.

Supporting code for the paper "Dangers of Bayesian Model Averaging under Covariate Shift"

Applying PVT to Semantic Segmentation

Implementation of Bagging and AdaBoost Algorithm

Learning infinite-resolution image processing with GAN and RL from unpaired image datasets, using a differentiable photo editing model.