Improving Transferability of Representations via Augmentation-Aware Self-Supervision

Last update: Sep 16, 2022

Related tags

Overview

Improving Transferability of Representations via Augmentation-Aware Self-Supervision

Accepted to NeurIPS 2021

TL;DR: Learning augmentation-aware information by predicting the difference between two augmented samples improves the transferability of representations.

Dependencies

conda create -n AugSelf python=3.8 pytorch=1.7.1 torchvision=0.8.2 cudatoolkit=10.1 ignite -c pytorch
conda activate AugSelf
pip install scipy tensorboard kornia==0.4.1 sklearn

Checkpoints

We provide ImageNet100-pretrained models in this Dropbox link.

Pretraining

We here provide SimSiam+AugSelf pretraining scripts. For training the baseline (i.e., no AugSelf), remove --ss-crop and --ss-color options. For using other frameworks like SimCLR, use the --framework option.

STL-10

CUDA_VISIBLE_DEVICES=0 python pretrain.py \
    --logdir ./logs/stl10/simsiam/aug_self \
    --framework simsiam \
    --dataset stl10 \
    --datadir DATADIR \
    --model resnet18 \
    --batch-size 256 \
    --max-epochs 200 \
    --ss-color 1.0 --ss-crop 1.0

ImageNet100

python pretrain.py \
    --logdir ./logs/imagenet100/simsiam/aug_self \
    --framework simsiam \
    --dataset imagenet100 \
    --datadir DATADIR \
    --batch-size 256 \
    --max-epochs 500 \
    --model resnet50 \
    --base-lr 0.05 --wd 1e-4 \
    --ckpt-freq 50 --eval-freq 50 \
    --ss-crop 0.5 --ss-color 0.5 \
    --num-workers 16 --distributed

Evaluation

Our main evaluation setups are linear evaluation on fine-grained classification datasets (Table 1) and few-shot benchmarks (Table 2).

linear evaluation

CUDA_VISIBLE_DEVICES=0 python transfer_linear_eval.py \
    --pretrain-data imagenet100 \
    --ckpt CKPT \
    --model resnet50 \
    --dataset cifar10 \
    --datadir DATADIR \
    --metric top1

few-shot

CUDA_VISIBLE_DEVICES=0 python transfer_few_shot.py \
    --pretrain-data imagenet100 \
    --ckpt CKPT \
    --model resnet50 \
    --dataset cub200 \
    --datadir DATADIR

Improving Transferability of Representations via Augmentation-Aware Self-Supervision

Related tags

Overview

Improving Transferability of Representations via Augmentation-Aware Self-Supervision

Dependencies

Checkpoints

Pretraining

STL-10

ImageNet100

Evaluation

linear evaluation

few-shot

Owner

hankook

My tensorflow implementation of "A neural conversational model", a Deep learning based chatbot

Official implementation of the paper Label-Efficient Semantic Segmentation with Diffusion Models

RSC-Net: 3D Human Pose, Shape and Texture from Low-Resolution Images and Videos

Official implementation of the ICLR 2021 paper

Compact Bilinear Pooling for PyTorch

Federated Deep Reinforcement Learning for the Distributed Control of NextG Wireless Networks.

Disease Informed Neural Networks (DINNs) — neural networks capable of learning how diseases spread, forecasting their progression, and finding their unique parameters (e.g. death rate).

Simple machine learning library / 簡單易用的機器學習套件

PyTorch implementation of the supervised learning experiments from the paper Model-Agnostic Meta-Learning (MAML)

Code for paper Adaptively Aligned Image Captioning via Adaptive Attention Time

Deep Learning & 3D Convolutional Neural Networks for Speaker Verification

CVPR2020 Counterfactual Samples Synthesizing for Robust VQA

PyTorch implementations of neural network models for keyword spotting

A sketch extractor for anime/illustration.

Improving 3D Object Detection with Channel-wise Transformer

ML model to classify between cats and dogs

Implementation EfficientDet: Scalable and Efficient Object Detection in PyTorch

Nonnegative spatial factorization for multivariate count data

Local Attention - Flax module for Jax

A Python library for Deep Probabilistic Modeling