[AAAI-2022] Official implementations of MCL: Mutual Contrastive Learning for Visual Representation Learning

Related tags

Deep LearningMCL
Overview

Mutual Contrastive Learning for Visual Representation Learning

This project provides source code for our Mutual Contrastive Learning for Visual Representation Learning (MCL).

Installation

Requirements

Ubuntu 18.04 LTS

Python 3.8 (Anaconda is recommended)

CUDA 11.1

PyTorch 1.7.0

NCCL for CUDA 11.1

Supervised Learning on CIFAR-100 dataset

Dataset

CIFAR-100 : download

unzip to the ./data folder

Training two baseline networks

python main_cifar.py --arch resnet32 --number-net 2

More commands for training various architectures can be found in scripts/train_cifar_baseline.sh

Training two networks by MCL

python main_cifar.py --arch resnet32  --number-net 2 \
    --alpha 0.1 --gamma 1. --beta 0.1 --lam 1. 

More commands for training various architectures can be found in scripts/train_cifar_mcl.sh

Results of MCL on CIFAR-100

We perform all experiments on a single NVIDIA RTX 3090 GPU (24GB) with three runs.

Network Baseline MCL(×2) MCL(×4)
ResNet-32 70.91±0.14 72.96±0.28 74.04±0.07
ResNet-56 73.15±0.23 74.48±0.23 75.74±0.16
ResNet-110 75.29±0.16 77.12±0.20 78.82±0.14
WRN-16-2 72.55±0.24 74.56±0.11 75.79±0.07
WRN-40-2 76.89±0.29 77.51±0.42 78.84±0.22
HCGNet-A1 77.42±0.16 78.62±0.26 79.50±0.15
ShuffleNetV2 0.5× 67.39±0.35 69.55±0.22 70.92±0.28
ShuffleNetV2 1× 70.93±0.24 73.26±0.18 75.18±0.25

Training multiple networks by MCL combined with Logit distillation

python main_cifar.py --arch WRN_16_2  --number-net 4 \
    --alpha 0.1 --gamma 1. --beta 0.1 --lam 1. \
    --logit-distill

More commands for training various architectures can be found in scripts/train_cifar_mcl_logit.sh

Results of MCL combined with logit distillation on CIFAR-100

We perform all experiments on a single NVIDIA RTX 3090 GPU (24GB) with three runs.

Network Baseline MCL(×4)+Logit KD
WRN-16-2 72.55±0.24 76.34±0.22
WRN-40-2 76.89±0.29 80.02±0.45
WRN-28-4 79.17±0.29 81.68±0.31
ShuffleNetV2 1× 70.93±0.24 77.02±0.32
HCGNet-A2 79.00±0.41 82.47±0.20

Supervised Learning on ImageNet dataset

Dataset preparation

  • Download the ImageNet dataset to YOUR_IMAGENET_PATH and move validation images to labeled subfolders

  • Create a datasets subfolder and a symlink to the ImageNet dataset

$ ln -s PATH_TO_YOUR_IMAGENET ./data/

Folder of ImageNet Dataset:

data/ImageNet
├── train
├── val

Training two networks by MCL

python main_cifar.py --arch resnet18  --number-net 2 \
    --alpha 0.1 --gamma 1. --beta 0.1 --lam 1. 

More commands for training various architectures can be found in scripts/train_imagenet_mcl.sh

Results of MCL on ImageNet

We perform all experiments on a single NVIDIA Tesla V100 GPU (32GB) with three runs.

Network Baseline MCL(×2) MCL(×4)
ResNet-18 69.76 70.32 70.77
ResNet-34 73.30 74.13 74.34

Training two networks by MCL combined with logit distillation

python main_cifar.py --arch resnet18  --number-net 2 \
    --alpha 0.1 --gamma 1. --beta 0.1 --lam 1. 

More commands for training various architectures can be found in scripts/train_imagenet_mcl.sh

Results of MCL combined with logit distillation on ImageNet

We perform all experiments on a single NVIDIA Tesla V100 GPU (32GB) with three runs.

Network Baseline MCL(×4)+Logit KD
ResNet-18 69.76 70.82

Self-Supervised Learning on ImageNet dataset

Apply MCL(×2) to MoCo

python main_moco_mcl.py \
  -a resnet18 \
  --lr 0.03 \
  --batch-size 256 \
  --number-net 2 \
  --dist-url 'tcp://localhost:10001' \
  --multiprocessing-distributed \
  --world-size 1 \
  --rank 0 \
  --gpu-ids 0,1,2,3,4,5,6,7 

Linear Classification

python main_lincls.py \
  -a resnet18 \
  --lr 30.0 \
  --batch-size 256 \
  --pretrained [your checkpoint path]/checkpoint_0199.pth.tar \
  --dist-url 'tcp://localhost:10001' \
  --multiprocessing-distributed \
  --world-size 1 \
  --rank 0 \
  --gpu-ids 0,1,2,3,4,5,6,7 

Results of applying MCL to MoCo on ImageNet

We perform all experiments on 8 NVIDIA RTX 3090 GPUs with three runs.

Network Baseline MCL(×2)
ResNet-18 47.45±0.11 48.04±0.13

Citation

@inproceedings{yang2022mcl,
  title={Mutual Contrastive Learning for Visual Representation Learning},
  author={Chuanguang Yang, Zhulin An, Linhang Cai, Yongjun Xu},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  year={2022}
}
Owner
winycg
winycg
MetaBalance: Improving Multi-Task Recommendations via Adapting Gradient Magnitudes of Auxiliary Tasks

MetaBalance: Improving Multi-Task Recommendations via Adapting Gradient Magnitudes of Auxiliary Tasks Introduction This repo contains the pytorch impl

Meta Research 38 Oct 10, 2022
3DMV jointly combines RGB color and geometric information to perform 3D semantic segmentation of RGB-D scans.

3DMV 3DMV jointly combines RGB color and geometric information to perform 3D semantic segmentation of RGB-D scans. This work is based on our ECCV'18 p

Владислав Молодцов 0 Feb 06, 2022
MAT: Mask-Aware Transformer for Large Hole Image Inpainting

MAT: Mask-Aware Transformer for Large Hole Image Inpainting (CVPR2022, Oral) Wenbo Li, Zhe Lin, Kun Zhou, Lu Qi, Yi Wang, Jiaya Jia [Paper] News This

254 Dec 29, 2022
Generate pixel-style avatars with python.

face2pixel Generate pixel-style avatars with python. Run: Clone the project: git clone https://github.com/theodorecooper/face2pixel install requiremen

Theodore Cooper 2 May 11, 2022
Full Stack Deep Learning Labs

Full Stack Deep Learning Labs Welcome! Project developed during lab sessions of the Full Stack Deep Learning Bootcamp. We will build a handwriting rec

Full Stack Deep Learning 1.2k Dec 31, 2022
JAX + dataclasses

jax_dataclasses jax_dataclasses provides a wrapper around dataclasses.dataclass for use in JAX, which enables automatic support for: Pytree registrati

Brent Yi 35 Dec 21, 2022
Imbalanced Gradients: A Subtle Cause of Overestimated Adversarial Robustness

Imbalanced Gradients: A Subtle Cause of Overestimated Adversarial Robustness Code for Paper "Imbalanced Gradients: A Subtle Cause of Overestimated Adv

Hanxun Huang 11 Nov 30, 2022
An Api for Emotion recognition.

PLAYEMO Playemo was built from the ground-up with Flask, a python tool that makes it easy for developers to build APIs. Use Cases Is Python your langu

greek geek 2 Jul 16, 2022
Implementation of ETSformer, state of the art time-series Transformer, in Pytorch

ETSformer - Pytorch Implementation of ETSformer, state of the art time-series Transformer, in Pytorch Install $ pip install etsformer-pytorch Usage im

Phil Wang 121 Dec 30, 2022
Biomarker identification for COVID-19 Severity in BALF cells Single-cell RNA-seq data

scBALF Covid-19 dataset Analysis Here is the Github page that has the codes for the bioinformatics pipeline described in the paper COVID-Datathon: Bio

Nami Niyakan 2 May 21, 2022
Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translation.

Understanding Minimum Bayes Risk Decoding This repo provides code and documentation for the following paper: Müller and Sennrich (2021): Understanding

ZurichNLP 13 May 01, 2022
My take on a practical implementation of Linformer for Pytorch.

Linformer Pytorch Implementation A practical implementation of the Linformer paper. This is attention with only linear complexity in n, allowing for v

Peter 349 Dec 25, 2022
Modular Probabilistic Programming on MXNet

MXFusion | | | | Tutorials | Documentation | Contribution Guide MXFusion is a modular deep probabilistic programming library. With MXFusion Modules yo

Amazon 100 Dec 10, 2022
Video Instance Segmentation using Inter-Frame Communication Transformers (NeurIPS 2021)

Video Instance Segmentation using Inter-Frame Communication Transformers (NeurIPS 2021) Paper Video Instance Segmentation using Inter-Frame Communicat

Sukjun Hwang 81 Dec 29, 2022
Merlion: A Machine Learning Framework for Time Series Intelligence

Merlion: A Machine Learning Library for Time Series Table of Contents Introduction Installation Documentation Getting Started Anomaly Detection Foreca

Salesforce 2.8k Dec 30, 2022
Solutions of Reinforcement Learning 2nd Edition

Solutions of Reinforcement Learning, An Introduction

YIFAN WANG 1.4k Dec 30, 2022
Implementation of the paper: "SinGAN: Learning a Generative Model from a Single Natural Image"

SinGAN This is an unofficial implementation of SinGAN from someone who's been sitting right next to SinGAN's creator for almost five years. Please ref

35 Nov 10, 2022
Visualizing lattice vibration information from phonon dispersion to atoms (For GPUMD)

Phonon-Vibration-Viewer (For GPUMD) Visualizing lattice vibration information from phonon dispersion for primitive atoms. In this tutorial, we will in

Liangting 6 Dec 10, 2022
using STGCN to achieve egg classification task

EEG Classification   The task requires us to classify electroencephalography(EEG) into six categories, including human body, human face, animal body,

4 Jun 13, 2022
Graduation Project

Gesture-Detection-and-Depth-Estimation This is my graduation project. (1) In this project, I use the YOLOv3 object detection model to detect gesture i

ChaosAT 1 Nov 23, 2021