Improving Transferability of Representations via Augmentation-Aware Self-Supervision

Last update: Sep 16, 2022

Related tags

Overview

Improving Transferability of Representations via Augmentation-Aware Self-Supervision

Accepted to NeurIPS 2021

TL;DR: Learning augmentation-aware information by predicting the difference between two augmented samples improves the transferability of representations.

Dependencies

conda create -n AugSelf python=3.8 pytorch=1.7.1 torchvision=0.8.2 cudatoolkit=10.1 ignite -c pytorch
conda activate AugSelf
pip install scipy tensorboard kornia==0.4.1 sklearn

Checkpoints

We provide ImageNet100-pretrained models in this Dropbox link.

Pretraining

We here provide SimSiam+AugSelf pretraining scripts. For training the baseline (i.e., no AugSelf), remove --ss-crop and --ss-color options. For using other frameworks like SimCLR, use the --framework option.

STL-10

CUDA_VISIBLE_DEVICES=0 python pretrain.py \
    --logdir ./logs/stl10/simsiam/aug_self \
    --framework simsiam \
    --dataset stl10 \
    --datadir DATADIR \
    --model resnet18 \
    --batch-size 256 \
    --max-epochs 200 \
    --ss-color 1.0 --ss-crop 1.0

ImageNet100

python pretrain.py \
    --logdir ./logs/imagenet100/simsiam/aug_self \
    --framework simsiam \
    --dataset imagenet100 \
    --datadir DATADIR \
    --batch-size 256 \
    --max-epochs 500 \
    --model resnet50 \
    --base-lr 0.05 --wd 1e-4 \
    --ckpt-freq 50 --eval-freq 50 \
    --ss-crop 0.5 --ss-color 0.5 \
    --num-workers 16 --distributed

Evaluation

Our main evaluation setups are linear evaluation on fine-grained classification datasets (Table 1) and few-shot benchmarks (Table 2).

linear evaluation

CUDA_VISIBLE_DEVICES=0 python transfer_linear_eval.py \
    --pretrain-data imagenet100 \
    --ckpt CKPT \
    --model resnet50 \
    --dataset cifar10 \
    --datadir DATADIR \
    --metric top1

few-shot

CUDA_VISIBLE_DEVICES=0 python transfer_few_shot.py \
    --pretrain-data imagenet100 \
    --ckpt CKPT \
    --model resnet50 \
    --dataset cub200 \
    --datadir DATADIR

Improving Transferability of Representations via Augmentation-Aware Self-Supervision

Related tags

Overview

Improving Transferability of Representations via Augmentation-Aware Self-Supervision

Dependencies

Checkpoints

Pretraining

STL-10

ImageNet100

Evaluation

linear evaluation

few-shot

Owner

hankook

PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices.

Adversarial Reweighting for Partial Domain Adaptation

Simulation of self-focusing of laser beams in condensed media

PyTorch package for the discrete VAE used for DALL·E.

Implement Decoupled Neural Interfaces using Synthetic Gradients in Pytorch

A motion detection system with RaspberryPi, OpenCV, Python

Code for the head detector (HeadHunter) proposed in our CVPR 2021 paper Tracking Pedestrian Heads in Dense Crowd.

This project is the official implementation of our accepted ICLR 2021 paper BiPointNet: Binary Neural Network for Point Clouds.

Flickr-Faces-HQ (FFHQ) is a high-quality image dataset of human faces, originally created as a benchmark for generative adversarial networks (GAN)

Seq2seq - Sequence to Sequence Learning with Keras

MM1 and MMC Queue Simulation using python - Results and parameters in excel and csv files

Tutorials, assignments, and competitions for MIT Deep Learning related courses.

Multiband spectro-radiometric satellite image analysis with K-means cluster algorithm

Towards Ultra-Resolution Neural Style Transfer via Thumbnail Instance Normalization

Official implementation of "Dynamic Anchor Learning for Arbitrary-Oriented Object Detection" (AAAI2021).

A fast, dataset-agnostic, deep visual search engine for digital art history

Code for Discriminative Sounding Objects Localization (NeurIPS 2020)

Experiments for Operating Systems Lab (ETCS-352)

Pytorch implementation of MLP-Mixer with loading pre-trained models.

Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting