Real-time Object Detection for Streaming Perception, CVPR 2022

Last update: Jun 08, 2022

Overview

StreamYOLO

Real-time Object Detection for Streaming Perception

Jinrong Yang, Songtao Liu, Zeming Li, Xiaoping Li, Sun Jian
Real-time Object Detection for Streaming Perception, CVPR 2022 (Oral)
Paper

Bestsoftwarechoose

Benchmark

Model	size	velocity	sAP 0.5:0.95	sAP50	sAP75	weights	COCO pretrained weights
StreamYOLO-s	600×960	1x	29.8	50.3	29.8	github	github
StreamYOLO-m	600×960	1x	33.7	54.5	34.0	github	github
StreamYOLO-l	600×960	1x	36.9	58.1	37.5	github	github
StreamYOLO-l	600×960	2x	34.6	56.3	34.7	github	github
StreamYOLO-l	600×960	still	39.4	60.0	40.2	github	github

Quick Start

Dataset preparation

You can download Argoverse-1.1 full dataset and annotation from HERE and unzip it.

The folder structure should be organized as follows before our processing.

StreamYOLO
├── exps
├── tools
├── yolox
├── data
│   ├── Argoverse-1.1
│   │   ├── annotations
│   │       ├── tracking
│   │           ├── train
│   │           ├── val
│   │           ├── test
│   ├── Argoverse-HD
│   │   ├── annotations
│   │       ├── test-meta.json
│   │       ├── train.json
│   │       ├── val.json

The hash strings represent different video sequences in Argoverse, and ring_front_center is one of the sensors for that sequence. Argoverse-HD annotations correspond to images from this sensor. Information from other sensors (other ring cameras or LiDAR) is not used, but our framework can be also extended to these modalities or to a multi-modality setting.

Installation

# basic python libraries
conda create --name streamyolo python=3.7

pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

pip3 install yolox==0.3
git clone [email protected]:yancie-yjr/StreamYOLO.git

cd StreamYOLO/

# add StreamYOLO to PYTHONPATH and add this line to ~/.bashrc or ~/.zshrc (change the file accordingly)
ADDPATH=$(pwd)
echo export PYTHONPATH=$PYTHONPATH:$ADDPATH >> ~/.bashrc
source ~/.bashrc

# Installing `mmcv` for the official sAP evaluation:
# Please replace `{cu_version}` and ``{torch_version}`` with the versions you are currently using.
# You will get import or runtime errors if the versions are incorrect.
pip install mmcv-full==1.1.5 -f https://download.openmmlab.com/mmcv/dist/{cu_version}/{torch_version}/index.html

Reproduce our results on Argoverse-HD

Step1. Prepare COCO dataset

cd <StreamYOLO_HOME>
ln -s /path/to/your/Argoverse-1.1 ./data/Argoverse-1.1
ln -s /path/to/your/Argoverse-HD ./data/Argoverse-HD

Step2. Reproduce our results on Argoverse:

python tools/train.py -f cfgs/m_s50_onex_dfp_tal_flip.py -d 8 -b 32 -c [/path/to/your/coco_pretrained_path] -o --fp16

-d: number of gpu devices.
-b: total batch size, the recommended number for -b is num-gpu * 8.
--fp16: mixed precision training.
-c: model checkpoint path.

Offline Evaluation

We support batch testing for fast evaluation:

python tools/eval.py -f  cfgs/l_s50_onex_dfp_tal_flip.py -c [/path/to/your/model_path] -b 64 -d 8 --conf 0.01 [--fp16] [--fuse]

--fuse: fuse conv and bn.
-d: number of GPUs used for evaluation. DEFAULT: All GPUs available will be used.
-b: total batch size across on all GPUs.
-c: model checkpoint path.
--conf: NMS threshold. If using 0.001, the performance will further improve by 0.2~0.3 sAP.

Online Evaluation

We modify the online evaluation from sAP

Please use 1 V100 GPU to test the performance since other GPUs with low computing power will trigger non-real-time results!!!!!!!!

cd sAP/streamyolo
bash streamyolo.sh

Citation

Please cite the following paper if this repo helps your research:

@InProceedings{streamyolo,
    author    = {Yang, Jinrong and Liu, Songtao and Li, Zeming and Li, Xiaoping and Sun, Jian},
    title     = {Real-time Object Detection for Streaming Perception},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
    year      = {2022}
}

License

This repo is released under the Apache 2.0 license. Please see the LICENSE file for more information.

Issues

when will the readme document be completed

Hi, @GOATmessi7 @yancie-yjr great wokrs. Can you enrich the readme about datasets preparing、how to training & validation and so on. hope to finish it soon. thanks

opened by SmallMunich 1

Real-time Object Detection for Streaming Perception, CVPR 2022

Related tags

Overview

StreamYOLO

Real-time Object Detection for Streaming Perception

Benchmark

Quick Start

Citation

License

Issues

when will the readme document be completed

Releases(0.1.0rc)

0.1.0rc(Apr 26, 2022)

Owner

Jinrong Yang

Rethinking the Importance of Implementation Tricks in Multi-Agent Reinforcement Learning

LBBA-boosted WSOD

General Multi-label Image Classification with Transformers

The code for "Deep Level Set for Box-supervised Instance Segmentation in Aerial Images".

Learning cell communication from spatial graphs of cells

Experiments with differentiable stacks and queues in PyTorch

Measuring if attention is explanation with ROAR

Poisson Surface Reconstruction for LiDAR Odometry and Mapping

NumPy로 구현한 딥러닝 라이브러리입니다. (자동 미분 지원)

Combining Diverse Feature Priors

Repository for the semantic WMI loss

Python and C++ implementation of "MarkerPose: Robust real-time planar target tracking for accurate stereo pose estimation". Accepted at LXCV @ CVPR 2021.

This is the official implementation code repository of Underwater Light Field Retention : Neural Rendering for Underwater Imaging (Accepted by CVPR Workshop2022 NTIRE)

Proof of concept GnuCash Webinterface

official code for dynamic convolution decomposition

Script that attempts to force M1 macs into RGB mode when used with monitors that are defaulting to YPbPr.

Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes

Improving Calibration for Long-Tailed Recognition (CVPR2021)

Official implementation of Influence-balanced Loss for Imbalanced Visual Classification in PyTorch.

Bu repo SAHI uygulamasını mantığını öğreniyoruz.