Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning, CVPR 2021

Last update: Dec 20, 2022

Overview

Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning

By Zhenda Xie*, Yutong Lin*, Zheng Zhang, Yue Cao, Stephen Lin and Han Hu.

This repo is an official implementation of "Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning" on PyTorch.

Introduction

PixPro (pixel-to-propagation) is an unsupervised visual feature learning approach by leveraging pixel-level pretext tasks. The learnt feature can be well transferred to downstream dense prediction tasks such as object detection and semantic segmentation. PixPro achieves the best transferring performance on Pascal VOC object detection (60.2 AP using C4) and COCO object detection (41.4 / 40.5 mAP using FPN / C4) with a ResNet-50 backbone.

An illustration of the proposed PixPro method.

Architecture of the PixContrast and PixPro methods.

Citation

@article{xie2020propagate,
  title={Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning},
  author={Xie, Zhenda and Lin, Yutong and Zhang, Zheng and Cao, Yue and Lin, Stephen and Hu, Han},
  conference={IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2021}
}

Main Results

PixPro pre-trained models

Epochs	Arch	Instance Branch	Download
100	ResNet-50		script \| model
400	ResNet-50		script \| model
100	ResNet-50	✔️	-
400	ResNet-50	✔️	-

Pascal VOC object detection

Faster-RCNN with C4

Method	Epochs	Arch	AP	AP₅₀	AP₇₅	Download
Scratch	-	ResNet-50	33.8	60.2	33.1	-
Supervised	100	ResNet-50	53.5	81.3	58.8	-
MoCo	200	ResNet-50	55.9	81.5	62.6	-
SimCLR	1000	ResNet-50	56.3	81.9	62.5	-
MoCo v2	800	ResNet-50	57.6	82.7	64.4	-
InfoMin	200	ResNet-50	57.6	82.7	64.6	-
InfoMin	800	ResNet-50	57.5	82.5	64.0	-
PixPro (ours)	100	ResNet-50	58.8	83.0	66.5	config \| model
PixPro (ours)	400	ResNet-50	60.2	83.8	67.7	config \| model

COCO object detection

Mask-RCNN with FPN

Method	Epochs	Arch	Schedule	bbox AP	mask AP	Download
Scratch	-	ResNet-50	1x	32.8	29.9	-
Supervised	100	ResNet-50	1x	39.7	35.9	-
MoCo	200	ResNet-50	1x	39.4	35.6	-
SimCLR	1000	ResNet-50	1x	39.8	35.9	-
MoCo v2	800	ResNet-50	1x	40.4	36.4	-
InfoMin	200	ResNet-50	1x	40.6	36.7	-
InfoMin	800	ResNet-50	1x	40.4	36.6	-
PixPro (ours)	100	ResNet-50	1x	40.8	36.8	config \| model
PixPro (ours)	100*	ResNet-50	1x	41.3	37.1	-
PixPro (ours)	400*	ResNet-50	1x	41.4	37.4	-

* Indicates methods with instance branch.

Mask-RCNN with C4

Method	Epochs	Arch	Schedule	bbox AP	mask AP	Download
Scratch	-	ResNet-50	1x	26.4	29.3	-
Supervised	100	ResNet-50	1x	38.2	33.3	-
MoCo	200	ResNet-50	1x	38.5	33.6	-
SimCLR	1000	ResNet-50	1x	38.4	33.6	-
MoCo v2	800	ResNet-50	1x	39.5	34.5	-
InfoMin	200	ResNet-50	1x	39.0	34.1	-
InfoMin	800	ResNet-50	1x	38.8	33.8	-
PixPro (ours)	100	ResNet-50	1x	40.0	34.8	config \| model
PixPro (ours)	400	ResNet-50	1x	40.5	35.3	config \| model

Getting started

Requirements

At present, we have not checked the compatibility of the code with other versions of the packages, so we only recommend the following configuration.

Python 3.7
PyTorch == 1.4.0
Torchvision == 0.5.0
CUDA == 10.1
Other dependencies

Installation

We recommand using conda env to setup the experimental environments.

# Create environment
conda create -n PixPro python=3.7 -y
conda activate PixPro

# Install PyTorch & Torchvision
conda install pytorch=1.4.0 cudatoolkit=10.1 torchvision -c pytorch -y

# Install apex
git clone https://github.com/NVIDIA/apex
cd apex
pip install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
cd ..

# Clone repo
git clone https://github.com/zdaxie/PixPro ./PixPro
cd ./PixPro

# Create soft link for data
mkdir data
ln -s ${ImageNet-Path} ./data/imagenet

# Install other requirements
pip install -r requirements.txt

Pretrain with PixPro

# Train with PixPro base for 100 epochs.
./tools/pixpro_base_r50_100ep.sh

Transfer to Pascal VOC or COCO object detection

# Convert a pre-trained PixPro model to detectron2's format
cd transfer/detection
python convert_pretrain_to_d2.py ${Input-Checkpoint(.pth)} ./output.pkl  

# Install Detectron2
python -m pip install detectron2==0.2.1 -f \
  https://dl.fbaipublicfiles.com/detectron2/wheels/cu101/torch1.4/index.html

# Create soft link for data
mkdir datasets
ln -s ${Pascal-VOC-Path}/VOC2007 ./datasets/VOC2007
ln -s ${Pascal-VOC-Path}/VOC2012 ./datasets/VOC2012
ln -s ${COCO-Path} ./datasets/coco

# Train detector with pre-trained PixPro model
# 1. Train Faster-RCNN with Pascal-VOC
python train_net.py --config-file configs/Pascal_VOC_R_50_C4_24k_PixPro.yaml --num-gpus 8 MODEL.WEIGHTS ./output.pkl
# 2. Train Mask-RCNN-FPN with COCO
python train_net.py --config-file configs/COCO_R_50_FPN_1x_PixPro.yaml --num-gpus 8 MODEL.WEIGHTS ./output.pkl
# 3. Train Mask-RCNN-C4 with COCO
python train_net.py --config-file configs/COCO_R_50_C4_1x_PixPro.yaml --num-gpus 8 MODEL.WEIGHTS ./output.pkl

# Test detector with provided fine-tuned model
python train_net.py --config-file configs/Pascal_VOC_R_50_C4_24k_PixPro.yaml --num-gpus 8 --eval-only \
  MODEL.WEIGHTS ./pixpro_base_r50_100ep_voc_md5_ec2dfa63.pth

More models and logs will be released!

Acknowledgement

Our testbed builds upon several existing publicly available codes. Specifically, we have modified and integrated the following code into this project:

Contributing to the project

Any pull requests or issues are welcomed.

Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning, CVPR 2021

Related tags

Overview

Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning

Introduction

Citation

Main Results

PixPro pre-trained models

Pascal VOC object detection

Faster-RCNN with C4

COCO object detection

Mask-RCNN with FPN

Mask-RCNN with C4

Getting started

Requirements

Installation

Pretrain with PixPro

Transfer to Pascal VOC or COCO object detection

Acknowledgement

Contributing to the project

Owner

Zhenda Xie

LAMDA: Label Matching Deep Domain Adaptation

Python wrappers to the C++ library SymEngine, a fast C++ symbolic manipulation library.

A collection of metrics for evaluating timbre dissimilarity using the TorchMetrics API

2D Time independent Schrodinger equation solver for arbitrary shape of well

A parametric soroban written with CADQuery.

Image-to-Image Translation in PyTorch

A python implementation of Physics-informed Spline Learning for nonlinear dynamics discovery

Image Processing, Image Smoothing, Edge Detection and Transforms

Facial detection, landmark tracking and expression transfer library for Windows, Linux and Mac

Code and description for my BSc Project, September 2021

This is the code for Deformable Neural Radiance Fields, a.k.a. Nerfies.

MonoRec: Semi-Supervised Dense Reconstruction in Dynamic Environments from a Single Moving Camera

Multi-Glimpse Network With Python

Training DiffWave using variational method from Variational Diffusion Models.

Benchmarking the robustness of Spatial-Temporal Models

A Transformer-Based Siamese Network for Change Detection

Code for CVPR2021 paper "Robust Reflection Removal with Reflection-free Flash-only Cues"

FL-WBC: Enhancing Robustness against Model Poisoning Attacks in Federated Learning from a Client Perspective

Code accompanying the paper "Knowledge Base Completion Meets Transfer Learning"

ToFFi - Toolbox for Frequency-based Fingerprinting of Brain Signals