[ICCV2021] Learning to Track Objects from Unlabeled Videos

Related tags

Deep LearningUSOT
Overview

Unsupervised Single Object Tracking (USOT)

🌿 Learning to Track Objects from Unlabeled Videos

Jilai Zheng, Chao Ma, Houwen Peng and Xiaokang Yang

2021 IEEE/CVF International Conference on Computer Vision (ICCV)

Introduction

This repository implements unsupervised deep tracker USOT, which learns to track objects from unlabeled videos.

Main ideas of USOT are listed as follows.

  • Coarsely discovering moving objects from videos, with pseudo boxes precise enough for bbox regression.
  • Training a naive Siamese tracker from single-frame pairs, then gradually extending it to longer temporal spans.
  • Following cycle memory training paradigm, enabling unsupervised tracker to update online.

Results

Results of USOT and USOT* on recent tracking benchmarks.

Model VOT2016
EAO
VOT2018
EAO
VOT2020
EAO
LaSOT
AUC (%)
TrackingNet
AUC (%)
OTB100
AUC (%)
USOT 0.351 0.290 0.222 33.7 59.9 58.9
USOT* 0.402 0.344 0.219 35.8 61.5 57.4

Raw result files can be found in folder result from Google Drive.

Tutorial

Environments

The environment we utilize is listed as follows.

  • Preprocessing: Pytorch 1.1.0 + CUDA-9.0 / 10.0 (following ARFlow)
  • Train / Test / Eval: Pytorch 1.7.1 + CUDA-10.0 / 10.2 / 11.1

If you have problems for preprocessing, you can actually skip it by downloading off-the-shelf preprocessed materials.

Preparations

Assume the project root path is $USOT_PATH. You can build an environment for development with the provided script, where $CONDA_PATH denotes your anaconda path.

cd $USOT_PATH
bash ./preprocessing/install_model.sh $CONDA_PATH USOT
source activate USOT && export PYTHONPATH=$(pwd)

You can revise the CUDA toolkit version for pytorch in install_model.sh (by default 10.0).

Test and Eval

First, we provide both models utilized in our paper (USOT.pth and USOT_star.pth). You can download them in folder snapshot from Google Drive, and place them in $USOT_PATH/var/snapshot.

Next, you can link your wanted benchmark dataset (e.g. VOT2018) to $USOT_PATH/datasets_test as follows. The ground truth json files for some benchmarks (e.g VOT2018.json) can be downloaded in folder test from Google Drive, and placed also in $USOT_PATH/datasets_test.

cd $USOT_PATH && mkdir datasets_test
ln -s $your_benchmark_path ./datasets_test/VOT2018

After that, you can test the tracker on these benchmarks (e.g. VOT2018) as follows. The raw results will be placed in $USOT_PATH/var/result/VOT2018/USOT.

cd $USOT_PATH
python -u ./scripts/test_usot.py --dataset VOT2018 --resume ./var/snapshot/USOT_star.pth

The inference result can be evaluated with pysot-toolkit. Install pysot-toolkit before evaluation.

cd $USOT_PATH/lib/eval_toolkit/pysot/utils
python setup.py build_ext --inplace

Then the evaluation can be conducted as follows.

cd $USOT_PATH
python ./lib/eval_toolkit/bin/eval.py --dataset_dir datasets_test \
        --dataset VOT2018 --tracker_result_dir var/result/VOT2018 --trackers USOT

Train

First, download the pretrained backbone in folder pretrain from Google Drive into $USOT_PATH/pretrain. Note that USOT* and USOT are respectively trained from imagenet_pretrain.model and moco_v2_800.model.

Second, preprocess the raw datasets with the paradigm of DP + Flow. Refer to $USOT_PATH/preprocessing/datasets_train for details.

In fact, we have provided two shortcuts for skipping this preprocessing procedure.

  • You can directly download the generated pseudo box files (e.g. got10k_flow.json) in folder train/box_sample_result from Google Drive, and place them into the corresponding dataset preprocessing path (e.g. $USOT_PATH/preprocessing/datasets_train/got10k), in order to skip the box generation procedure.
  • You can directly download the whole cropped training dataset (e.g. got10k_flow.tar) in dataset folder from Google Drive (Coming soon) (e.g. train/GOT-10k), which enables you to skip all procedures in preprocessing.

Third, revise the config file for training as $USOT_PATH/experiments/train/USOT.yaml. Very important options are listed as follows.

  • GPUS: the gpus for training, e.g. '0,1,2,3'
  • TRAIN/PRETRAIN: the pretrained backbone, e.g. 'imagenet_pretrain.model'
  • DATASET: the folder for your cropped training instances and their pseudo annotation files, e.g. PATH: '/data/got10k_flow/crop511/', ANNOTATION: '/data/got10k_flow/train.json'

Finally, you can start the training phase with the following script. The training checkpoints will also be placed automatically in $USOT_PATH/var/snapshot.

cd $USOT_PATH
python -u ./scripts/train_usot.py --cfg experiments/train/USOT.yaml --gpus 0,1,2,3 --workers 32

We also provide a onekey script for train, test and eval.

cd $USOT_PATH
python ./scripts/onekey_usot.py --cfg experiments/train/USOT.yaml

Citation

If any parts of our paper and codes are helpful to your work, please generously citing:

@inproceedings{zheng-iccv2021-usot,
   title={Learning to Track Objects from Unlabeled Videos},
   author={Jilai Zheng and Chao Ma and Houwen Peng and Xiaokang Yang},
   booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
   year={2021}
}

Reference

We refer to the following repositories when implementing our unsupervised tracker. Thanks for their great work.

Contact

Feel free to contact me if you have any questions.

Automatic learning-rate scheduler

AutoLRS This is the PyTorch code implementation for the paper AutoLRS: Automatic Learning-Rate Schedule by Bayesian Optimization on the Fly published

Yuchen Jin 33 Nov 18, 2022
Face-Recognition-Attendence-System - This face recognition Attendence system using Python

Face-Recognition-Attendence-System I have developed this face recognition Attend

Riya Gupta 4 May 10, 2022
Contains code for Deep Kernelized Dense Geometric Matching

DKM - Deep Kernelized Dense Geometric Matching Contains code for Deep Kernelized Dense Geometric Matching We provide pretrained models and code for ev

Johan Edstedt 83 Dec 23, 2022
MATLAB codes of the book "Digital Image Processing Fourth Edition" converted to Python

Digital Image Processing Python MATLAB codes of the book "Digital Image Processing Fourth Edition" converted to Python TO-DO: Refactor scripts, curren

Merve Noyan 24 Oct 16, 2022
Gym environment for FLIPIT: The Game of "Stealthy Takeover"

gym-flipit Gym environment for FLIPIT: The Game of "Stealthy Takeover" invented by Marten van Dijk, Ari Juels, Alina Oprea, and Ronald L. Rivest. Desi

Lisa Oakley 2 Dec 15, 2021
[BMVC'21] Official PyTorch Implementation of Grounded Situation Recognition with Transformers

Grounded Situation Recognition with Transformers Paper | Model Checkpoint This is the official PyTorch implementation of Grounded Situation Recognitio

Junhyeong Cho 18 Jul 19, 2022
OcclusionFusion: realtime dynamic 3D reconstruction based on single-view RGB-D

OcclusionFusion (CVPR'2022) Project Page | Paper | Video Overview This repository contains the code for the CVPR 2022 paper OcclusionFusion, where we

Wenbin Lin 193 Dec 15, 2022
Adversarial Attacks are Reversible via Natural Supervision

Adversarial Attacks are Reversible via Natural Supervision ICCV2021 Citation @InProceedings{Mao_2021_ICCV, author = {Mao, Chengzhi and Chiquier

Computer Vision Lab at Columbia University 20 May 22, 2022
Pytorch Implementation of Adversarial Deep Network Embedding for Cross-Network Node Classification

Pytorch Implementation of Adversarial Deep Network Embedding for Cross-Network Node Classification (ACDNE) This is a pytorch implementation of the Adv

陈志豪 8 Oct 13, 2022
Official code for the paper: Deep Graph Matching under Quadratic Constraint (CVPR 2021)

QC-DGM This is the official PyTorch implementation and models for our CVPR 2021 paper: Deep Graph Matching under Quadratic Constraint. It also contain

Quankai Gao 55 Nov 14, 2022
A clean implementation based on AlphaZero for any game in any framework + tutorial + Othello/Gobang/TicTacToe/Connect4 and more

Alpha Zero General (any game, any framework!) A simplified, highly flexible, commented and (hopefully) easy to understand implementation of self-play

Surag Nair 3.1k Jan 05, 2023
This is the source code for generating the ASL-Skeleton3D and ASL-Phono datasets. Check out the README.md for more details.

ASL-Skeleton3D and ASL-Phono Datasets Generator The ASL-Skeleton3D contains a representation based on mapping into the three-dimensional space the coo

Cleison Amorim 5 Nov 20, 2022
Convert Table data to approximate values with GUI

Table_Editor Convert Table data to approximate values with GUIs... usage - Import methods for extension Tables. Imported method supposed to have only

CLJ 1 Jan 10, 2022
PERIN is Permutation-Invariant Semantic Parser developed for MRP 2020

PERIN: Permutation-invariant Semantic Parsing David Samuel & Milan Straka Charles University Faculty of Mathematics and Physics Institute of Formal an

ÚFAL 40 Jan 04, 2023
Official code for "Simpler is Better: Few-shot Semantic Segmentation with Classifier Weight Transformer. ICCV2021".

Simpler is Better: Few-shot Semantic Segmentation with Classifier Weight Transformer. ICCV2021. Introduction We proposed a novel model training paradi

Lucas 103 Dec 14, 2022
PyTorch implementation of the WarpedGANSpace: Finding non-linear RBF paths in GAN latent space (ICCV 2021)

Authors official PyTorch implementation of the "WarpedGANSpace: Finding non-linear RBF paths in GAN latent space" [ICCV 2021].

Christos Tzelepis 100 Dec 06, 2022
FreeSOLO for unsupervised instance segmentation, CVPR 2022

FreeSOLO: Learning to Segment Objects without Annotations This project hosts the code for implementing the FreeSOLO algorithm for unsupervised instanc

NVIDIA Research Projects 253 Jan 02, 2023
An MQA (Studio, originalSampleRate) identifier for lossless flac files written in Python.

An MQA (Studio, originalSampleRate) identifier for "lossless" flac files written in Python.

Daniel 10 Oct 03, 2022
Used to record WKU's utility bills on a regular basis.

WKU水电费小助手 一个用于定期记录WKU水电费的脚本 Looking for English Readme? 背景 由于WKU校园内的水电账单系统时常存在扣费延迟的现象,而补扣的费用缺乏令人信服的证明。不少学生为费用摸不着头脑,但也没有申诉的依据。为了更好地掌握水电费使用情况,留下一手证据,我开源

2 Jul 21, 2022