A Broad Study on the Transferability of Visual Representations with Contrastive Learning

Last update: Nov 09, 2022

Overview

A Broad Study on the Transferability of Visual Representations with Contrastive Learning

This repository contains code for the paper: A Broad Study on the Transferability of Visual Representations with Contrastive Learning

Prerequisites

PyTorch 1.7
pytorch-lightning 1.1.5

Install the required dependencies by:

pip install -r environments/requirements.txt

How to Run

Download Datasets

The data should be located in ~/datasets/cdfsl folder. To download all the datasets:

bash data_loader/download.sh

Training

python main.py --system ${system}  --dataset ${train_dataset} --gpus -1 --model resnet50

where system is one of base_finetune(ce), moco (SelfSupCon), moco_mit (SupCon), base_plus_moco (CE+SelfSupCon), or supervised_mean2 (SupCon+SelfSupCon).

To know more about the cli arguments, see configs.py.

You can also run the training script by bash scripts/run_linear_bn.sh -m train.

Evaluation

Linear evaluation

python main.py --system linear_eval \
  --train_aug true --val_aug false \
  --dataset ${val_data}_train --val_dataset ${val_data}_test \
  --ckpt ${ckpt} --load_base --batch_size ${bs} \
  --lr ${lr} --optim_wd ${wd}  --linear_bn --linear_bn_affine false \
  --scheduler step  --step_lr_milestones ${_milestones}

You can also run the evaluation script by bash scripts/run_linear_bn.sh -m tune to hyper-parameter tune, and then bash scripts/run_linear_bn.sh -m test to do linear-evaluation on the optimal hyper-parameter.

Few-shot

python main.py --system few_shot \
    --val_dataset ${val_data} \
    --load_base --test --model ${model} \
    --ckpt ${ckpt} --num_workers 4

You can also run the evaluation script by bash scripts/run_fewshot.sh.

Full-network finetuning

python main.py --system linear_transfer \
    --dataset ${val_data}_train --val_dataset ${val_data}_test \
    --ckpt ${ckpt} --load_base \
    --batch_size ${bs} --lr ${lr} --optim_wd ${wd} \
    --scheduler step  --step_lr_milestones ${_milestones} \
    --linear_bn --linear_bn_affine false \
    --max_epochs ${max_epochs}

You can also run the evaluation script by bash scripts/run_transfer_bn.sh -m tune to hyper-parameter tune, and then bash scripts/run_transfer_bn.sh -m test to do linear-evaluation on the optimal hyper-parameter.

Pretrained models

ImageNet pretrained models can be found here
mini-ImageNet pretrained models can be found here

You can also convert our pretrained checkpoint into torchvision resnet style checkpoint by python utils/convert_to_torchvision_resnet.py -i [input ckpt] -o [output path]

Citation

If you find this repo useful for your research, please consider citing the paper:

@misc{islam2021broad,
      title={A Broad Study on the Transferability of Visual Representations with Contrastive Learning}, 
      author={Ashraful Islam and Chun-Fu Chen and Rameswar Panda and Leonid Karlinsky and Richard Radke and Rogerio Feris},
      year={2021},
      eprint={2103.13517},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Acknowledgement

cdfsl-benchmark

Comments

eurosat.zip cannot be found on google drive

eurosat.zip cannot be found on google drive with the url: https://drive.google.com/uc?id=1FYZvuBePf2tuEsEaBCsACtIHi6eFpSwe

Can you please check the url? Thank you.

opened by Cohesion97 2
How to get CKA scores between different stages in Figure 4?
Thanks for your amazing study! I have some questions about the CKA scores shown in Figure 4. Take ResNet-50 as an example, which has five stages.

Does stage 5 include the average pooling layer to output the feature of size 1x2048?

Given an input sample, for the feature after each in-between stage (1-4), do you flatten the original feature map (1 x c x h x w) to a vector (1 x chw) OR do you adopt an extra average pooling process to obtain a vector (1 x c)? I've tried to flatten the feature map after each stage but obtained a very high-dimension vector (about 1M).

(c: feature dimension; h: height; w: width) Looking forward to your reply, thanks.
opened by klfsalfjl 0

SUPERVISED-CONTRASTIVE-LEARNING-FOR-PRE-TRAINED-LANGUAGE-MODEL-FINE-TUNING - The Facebook paper about fine tuning RoBERTa with contrastive loss

"# SUPERVISED-CONTRASTIVE-LEARNING-FOR-PRE-TRAINED-LANGUAGE-MODEL-FINE-TUNING" i

28 Dec 12, 2022

PyTorch implementation code for the paper MixCo: Mix-up Contrastive Learning for Visual Representation

How to Reproduce our Results This repository contains PyTorch implementation code for the paper MixCo: Mix-up Contrastive Learning for Visual Represen

46 Dec 15, 2022

Conformer: Local Features Coupling Global Representations for Visual Recognition

Conformer: Local Features Coupling Global Representations for Visual Recognition (arxiv) This repository is built upon DeiT and timm Usage First, inst

378 Jan 8, 2023

ImageNet-CoG is a benchmark for concept generalization. It provides a full evaluation framework for pre-trained visual representations which measure how well they generalize to unseen concepts.

The ImageNet-CoG Benchmark Project Website Paper (arXiv) Code repository for the ImageNet-CoG Benchmark introduced in the paper "Concept Generalizatio

23 Oct 9, 2022

Re-implementation of the Noise Contrastive Estimation algorithm for pyTorch, following "Noise-contrastive estimation: A new estimation principle for unnormalized statistical models." (Gutmann and Hyvarinen, AISTATS 2010)

Noise Contrastive Estimation for pyTorch Overview This repository contains a re-implementation of the Noise Contrastive Estimation algorithm, implemen

42 Nov 24, 2022

Code for Efficient Visual Pretraining with Contrastive Detection

Code for DetCon This repository contains code for the ICCV 2021 paper "Efficient Visual Pretraining with Contrastive Detection" by Olivier J. Hénaff,

56 Nov 13, 2022

The official implementation of CVPR 2021 Paper: Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation.

Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation This repository is the official implementation of CVPR 2021 paper:

9 Nov 14, 2022

This repository is the official implementation of Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (NeurIPS21).

Core-tuning This repository is the official implementation of ``Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regular

18 Dec 17, 2022

improvement of CLIP features over the traditional resnet features on the visual question answering, image captioning, navigation and visual entailment tasks.

CLIP-ViL In our paper "How Much Can CLIP Benefit Vision-and-Language Tasks?", we show the improvement of CLIP features over the traditional resnet fea

310 Dec 28, 2022

A Broad Study on the Transferability of Visual Representations with Contrastive Learning

Related tags

Overview

A Broad Study on the Transferability of Visual Representations with Contrastive Learning

Prerequisites

How to Run

Download Datasets

Training

Evaluation

Linear evaluation

Few-shot

Full-network finetuning

Pretrained models

Citation

Acknowledgement

You might also like...

SUPERVISED-CONTRASTIVE-LEARNING-FOR-PRE-TRAINED-LANGUAGE-MODEL-FINE-TUNING - The Facebook paper about fine tuning RoBERTa with contrastive loss

PyTorch implementation code for the paper MixCo: Mix-up Contrastive Learning for Visual Representation

Conformer: Local Features Coupling Global Representations for Visual Recognition

ImageNet-CoG is a benchmark for concept generalization. It provides a full evaluation framework for pre-trained visual representations which measure how well they generalize to unseen concepts.

Re-implementation of the Noise Contrastive Estimation algorithm for pyTorch, following "Noise-contrastive estimation: A new estimation principle for unnormalized statistical models." (Gutmann and Hyvarinen, AISTATS 2010)

Code for Efficient Visual Pretraining with Contrastive Detection

The official implementation of CVPR 2021 Paper: Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation.

This repository is the official implementation of Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (NeurIPS21).

improvement of CLIP features over the traditional resnet features on the visual question answering, image captioning, navigation and visual entailment tasks.

Comments

eurosat.zip cannot be found on google drive

How to get CKA scores between different stages in Figure 4?

Releases(v0.1.0)

v0.1.0(Apr 20, 2021)

Owner

Ashraful Islam

Implementation for paper LadderNet: Multi-path networks based on U-Net for medical image segmentation

Backend code to use MCPI's python API to make infinite worlds with custom generation

Python Implementation of Chess Playing AI with variable difficulty

Code for the paper "Adapting Monolingual Models: Data can be Scarce when Language Similarity is High"

📚 A collection of Jupyter notebooks for learning and experimenting with OpenVINO 👓

Learning multiple gaits of quadruped robot using hierarchical reinforcement learning

Code repo for "Cross-Scale Internal Graph Neural Network for Image Super-Resolution" (NeurIPS'20)

This project provides an unsupervised framework for mining and tagging quality phrases on text corpora with pretrained language models (KDD'21).

A pytorch implementation of Reading Wikipedia to Answer Open-Domain Questions.

Official repository for ABC-GAN

StyleGAN2-ADA - Official PyTorch implementation

PyTorch implementation of VAGAN: Visual Feature Attribution Using Wasserstein GANs

StyleGAN-Human: A Data-Centric Odyssey of Human Generation

A Python training and inference implementation of Yolov5 helmet detection in Jetson Xavier nx and Jetson nano

learning and feeling SLAM together with hands-on-experiments

Only a Matter of Style: Age Transformation Using a Style-Based Regression Model

PyTorch Code for NeurIPS 2021 paper Anti-Backdoor Learning: Training Clean Models on Poisoned Data.

"SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image", Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Humphrey Shi, Zhangyang Wang

COVID-VIT: Classification of Covid-19 from CT chest images based on vision transformer models

Emotion Recognition from Facial Images