[NIPS 2021] UOTA: Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration.

Last update: Jun 29, 2022

Related tags

Deep Learning uota

Overview

UOTA: Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration

This repository is the official PyTorch implementation of UOTA (Unsupervised OuTlier Arbitration).

0 Requirements

Python 3.6
PyTorch install = 1.6.0
torchvision install = 0.7.0
CUDA 10.1
Apex with CUDA extension
Other dependencies: opencv-python, scipy, pandas, numpy

1 Pretraining

We release a demo to pretrain ResNet50 on ImageNet1K with SwAV+UOTA pretrained models.

1.1 SwAV+UOTA pretrain

To train SwAV+UOTA on a single node with 4 gpus for 200 epochs, run:

DATASET_PATH="path/to/ImageNet1K/train"
EXPERIMENT_PATH="path/to/experiment"

python -m torch.distributed.launch --nproc_per_node=4 main_uota.py \
--data_path ${DATASET_PATH} \
--nmb_crops 2 6 \
--size_crops 224 96 \
--min_scale_crops 0.14 0.05 \
--max_scale_crops 1. 0.14 \
--crops_for_assign 0 1 \
--use_pil_blur true \
--epochs 200 \
--warmup_epochs 0 \
--batch_size 64 \
--base_lr 0.6 \
--final_lr 0.0006 \
--uota_tau 350. \
--epoch_uota_starts 100 \
--wd 0.000001 \
--use_fp16 true \
--dist_url "tcp://localhost:40000" \
--arch uota_r50 \
--sync_bn pytorch \
--dump_path ${EXPERIMENT_PATH}

2 Linear Evaluation

To train a linear classifier on frozen features out of deep network pretrained via various self-supervised pretraining methods, run:

DATASET_PATH="path/to/ImageNet1K"
EXPERIMENT_PATH="path/to/experiment"
LINCLS_PATH="path/to/lincls"

python -m torch.distributed.launch --nproc_per_node=4 eval_linear.py \
--data_path ${DATASET_PATH} \
--arch resnet50 \
--lr 1.2 \
--dump_path ${LINCLS_PATH} \
--pretrained ${EXPERIMENT_PATH}/swav_uota_r50_e200_pretrained.pth \
--batch_size 64 \
--num_classes 100 \

3 Results

To compare with SwAV fairly, we provide a SwAV+UOTA model with ResNet-50 architecture pretrained on ImageNet1K for 200 epochs, and release the pretrained model and the linear classier.

method	epochs	batch-size	multi-crop	ImageNet1K top-1 acc.	pretrained model	linear classifier
SwAV	200	256	2x224 + 6x96	72.7	/	/
SwAV + UOTA	200	256	2x224 + 6x96	73.5	pretrained	linear

4 Citation

@InProceedings{wang2021NeurIPS,
  title={Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration},
  author={Wang, Yu and Lin, Jingyang and Zou, Jingjing and Pan, Yingwei and Yao, Ting and Mei, Tao},
  booktitle={NeurIPS},
  year={2021},
}

You might also like...

PyTorch implementation of spectral graph ConvNets, NIPS’16

Graph ConvNets in PyTorch October 15, 2017 Xavier Bresson http://www.ntu.edu.sg/home/xbresson https://github.com/xbresson https://twitter.com/xbresson

287 Jan 4, 2023

PyTorch implementation of the Value Iteration Networks (VIN) (NIPS '16 best paper)

Value Iteration Networks in PyTorch Tamar, A., Wu, Y., Thomas, G., Levine, S., and Abbeel, P. Value Iteration Networks. Neural Information Processing

75 Nov 24, 2022

Pytorch implementation of Value Iteration Networks (NIPS 2016 best paper)

VIN: Value Iteration Networks A quick thank you A few others have released amazing related work which helped inspire and improve my own implementation

297 Dec 26, 2022

pytorch implementation of "Contrastive Multiview Coding", "Momentum Contrast for Unsupervised Visual Representation Learning", and "Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination"

Unofficial implementation: MoCo: Momentum Contrast for Unsupervised Visual Representation Learning (Paper) InsDis: Unsupervised Feature Learning via N

16 Nov 4, 2020

The official implementation of CVPR 2021 Paper: Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation.

Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation This repository is the official implementation of CVPR 2021 paper:

9 Nov 14, 2022

(JMLR'19) A Python Toolbox for Scalable Outlier Detection (Anomaly Detection)

[NIPS 2021] UOTA: Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration.

Related tags

Overview

UOTA: Improving Self-supervised Learning with Automated Unsupervised Outlier Arbitration

0 Requirements

1 Pretraining

1.1 SwAV+UOTA pretrain

2 Linear Evaluation

3 Results

4 Citation

You might also like...

PyTorch implementation of spectral graph ConvNets, NIPS’16

PyTorch implementation of the Value Iteration Networks (VIN) (NIPS '16 best paper)

Pytorch implementation of Value Iteration Networks (NIPS 2016 best paper)

pytorch implementation of "Contrastive Multiview Coding", "Momentum Contrast for Unsupervised Visual Representation Learning", and "Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination"

The official implementation of CVPR 2021 Paper: Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation.

(JMLR'19) A Python Toolbox for Scalable Outlier Detection (Anomaly Detection)

Streaming Anomaly Detection Framework in Python (Outlier Detection for Streaming Data)

A gesture recognition system powered by OpenPose, k-nearest neighbours, and local outlier factor.

Outlier Exposure with Confidence Control for Out-of-Distribution Detection

Releases(v1.0.0)

v1.0.0(Oct 26, 2021)

Owner

Anatomy of Matplotlib -- tutorial developed for the SciPy conference

Text Generation by Learning from Demonstrations

Retinal Vessel Segmentation with Pixel-wise Adaptive Filters (ISBI 2022)

OpenMMLab Computer Vision Foundation

Official implementation of Densely connected normalizing flows

Official implementation of "Synthetic Temporal Anomaly Guided End-to-End Video Anomaly Detection" (ICCV Workshops 2021: RSL-CV).

A collection of awesome resources image-to-image translation.

UT-Sarulab MOS prediction system using SSL models

DeepSpeed is a deep learning optimization library that makes distributed training easy, efficient, and effective.

the code for paper "Energy-Based Open-World Uncertainty Modeling for Confidence Calibration"

UMich 500-Level Mobile Robotics Course

Train an imgs.ai model on your own dataset

Extreme Rotation Estimation using Dense Correlation Volumes

Implementation of 'lightweight' GAN, proposed in ICLR 2021, in Pytorch. High resolution image generations that can be trained within a day or two

RM Operation can equivalently convert ResNet to VGG, which is better for pruning; and can help RepVGG perform better when the depth is large.

Code for paper "Context-self contrastive pretraining for crop type semantic segmentation"

Official implementation for Scale-Aware Neural Architecture Search for Multivariate Time Series Forecasting

A multi-scale unsupervised learning for deformable image registration

Pytorch implementation of “Recursive Non-Autoregressive Graph-to-Graph Transformer for Dependency Parsing with Iterative Refinement”

Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more