Implementation for paper "Towards the Generalization of Contrastive Self-Supervised Learning"

Last update: Nov 30, 2022

Overview

Contrastive Self-Supervised Learning on CIFAR-10

Paper

"Towards the Generalization of Contrastive Self-Supervised Learning", Weiran Huang, Mingyang Yi and Xuyang Zhao, arXiv:2111.00743, 2021.

The most critical argument we made in our paper is that the quality of data augmentation exhibits great impact on the quality of contrastive-learned encoder. The data augmentation with sharper intra-class concentration enables the model to have better generalization on downstream tasks. We verify it through a variety of experiments in this repository.

Supported methods

SimCLR
Barlow Twins
MoCo
SimSiam

Installation

pip install -r requirement.txt

Dependencies

torch==1.4.0
torchvision==0.5.0
pytorch-lightning==1.3.8
lightly==1.0.8 (important!)

Evaluation

KNN evaluation protocol. Code from here.

Results

ResNet-18 trained by SimCLR and Barlow Twins over various data augmentation combinations.

Example: python main.py --model=twins --epoch=800 --batch=512 --round=3 --augs=abcde

(a)	(b)	(c)	(d)	(e)	SimCLR	Barlow Twins
✓	✓	✓	✓	✓	89.92 ± 0.05	83.93 ± 0.57
✓	✓	✓	✓	×	88.41 ± 0.11	83.37 ± 0.43
✓	✓	✓	×	×	83.62 ± 0.19	73.70 ± 0.99
✓	✓	×	×	×	62.91 ± 0.25	49.56 ± 0.11
✓	×	×	×	×	62.37 ± 0.09	48.54 ± 0.29

Augmentation operations include:

(a) random cropping with a scaling factor chosen in [0.08, 1.0];

(b) random Gaussian blur with a probability 0.5;

(d) color distortion with a probability of 0.8 and with strength of [0.4, 0.4, 0.4, 0.1];

(e) random horizontal flipping with a probability of 0.5.

ResNet18 trained by SimCLR and Barlow Twins over various color distortion strengths.

Example: python main.py --model=simclr --epoch=800 --batch=512 --round=3 --augs=color --strength=1

Color Distortion Strength	SimCLR	Barlow Twins
1/8	73.60 ± 0.11	61.13 ± 2.81
1/4	76.25 ± 0.16	68.30 ± 0.15
1/2	78.49 ± 0.09	72.76 ± 1.50
1	82.64 ± 0.57	78.79 ± 0.54

Acknowledgement

This code is based on:

Citation

If you find our work useful in your research, please consider citing:

@misc{huang2021generalization,
      title={Towards the Generalization of Contrastive Self-Supervised Learning}, 
      author={Weiran Huang and Mingyang Yi and Xuyang Zhao},
      year={2021},
      eprint={2111.00743},
      archivePrefix={arXiv}
}

You might also like...

Implementation of Nyström Self-attention, from the paper Nyströmformer

Nyström Attention Implementation of Nyström Self-attention, from the paper Nyströmformer. Yannic Kilcher video Install $ pip install nystrom-attention

95 Jan 2, 2023

Implementation of SETR model, Original paper: Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers.

SETR - Pytorch Since the original paper (Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers.) has no official

112 Dec 16, 2022

Official implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis https://arxiv.org/abs/2011.13775

CIPS -- Official Pytorch Implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis Requirements pip install -r requi

Multimodal Lab @ Samsung AI Center Moscow

201 Dec 21, 2022

Official pytorch implementation of paper

Implementation for paper "Towards the Generalization of Contrastive Self-Supervised Learning"

Related tags

Overview

Contrastive Self-Supervised Learning on CIFAR-10

Paper

Supported methods

Installation

Dependencies

Evaluation

Results

ResNet-18 trained by SimCLR and Barlow Twins over various data augmentation combinations.

ResNet18 trained by SimCLR and Barlow Twins over various color distortion strengths.

Acknowledgement

Citation

You might also like...

Implementation of Nyström Self-attention, from the paper Nyströmformer

Implementation of SETR model, Original paper: Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers.

Official implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis https://arxiv.org/abs/2011.13775

Official pytorch implementation of paper "Image-to-image Translation via Hierarchical Style Disentanglement".

PyTorch implementation of paper "Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes", CVPR 2021

Implementation of Barlow Twins paper

Official pytorch implementation of paper "Inception Convolution with Efficient Dilation Search" (CVPR 2021 Oral).

Official implementation of our paper "LLA: Loss-aware Label Assignment for Dense Pedestrian Detection" in Pytorch.

Functional TensorFlow Implementation of Singular Value Decomposition for paper Fast Graph Learning

Releases(v0.2.1)

v0.2.1(Jun 16, 2022)

v0.2.0(Jun 4, 2022)

v0.1.0(Jun 4, 2022)

Owner

Weiran Huang

U-Net Brain Tumor Segmentation

PyTorch Implementation of the SuRP algorithm by the authors of the AISTATS 2022 paper "An Information-Theoretic Justification for Model Pruning"

RDA: Robust Domain Adaptation via Fourier Adversarial Attacking

Fight Recognition from Still Images in the Wild @ WACVW2022, Real-world Surveillance Workshop

Official Pytorch implementation for Deep Contextual Video Compression, NeurIPS 2021

Specificity-preserving RGB-D Saliency Detection

RODD: A Self-Supervised Approach for Robust Out-of-Distribution Detection

Home for cuQuantum Python & NVIDIA cuQuantum SDK C++ samples

CTF challenges and write-ups for MicroCTF 2021.

source code and pre-trained/fine-tuned checkpoint for NAACL 2021 paper LightningDOT

Automatic self-diagnosis program (python required)Automatic self-diagnosis program (python required)

PFFDTD is an open-source FDTD simulator for 3D room acoustics

IhoneyBakFileScan Modify - 批量网站备份文件扫描器，增加文件规则，优化内存占用

Implementation of BI-RADS-BERT & The Advantages of Section Tokenization.

Simple and ready-to-use tutorials for TensorFlow

(Personalized) Page-Rank computation using PyTorch

CAUSE: Causality from AttribUtions on Sequence of Events

Unofficial implementation of the ImageNet, CIFAR 10 and SVHN Augmentation Policies learned by AutoAugment using pillow

Official implementation of the ICLR 2021 paper

Does Pretraining for Summarization Reuqire Knowledge Transfer?