[ICCV 2021] Relaxed Transformer Decoders for Direct Action Proposal Generation

Last update: Nov 30, 2022

Overview

RTD-Net (ICCV 2021)

This repo holds the codes of paper: "Relaxed Transformer Decoders for Direct Action Proposal Generation", accepted in ICCV 2021.

News

[2021.8.17] We release codes, checkpoint and features on THUMOS14.

Overview

This paper presents a simple and end-to-end learnable framework (RTD-Net) for direct action proposal generation, by re-purposing a Transformer-alike architecture. Thanks to the parallel decoding of multiple proposals with explicit context modeling, our RTD-Net outperforms the previous state-of-the-art methods in temporal action proposal generation task on THUMOS14 and also yields a superior performance for action detection on this dataset. In addition, free of NMS post-processing, our detection pipeline is more efficient than previous methods.

Dependencies

Python 3.7 or higher
PyTorch 1.6 or higher
Torchvision
Numpy 1.19.2

Data Preparation

To reproduce the results in THUMOS14 without further changes:

Download the data from GoogleDrive.
Place I3D_features and TEM_scores into the folder data.

Checkpoint

Dataset	[email protected]	[email protected]	[email protected]	[email protected]	checkpoint
THUMOS14	41.52	49.33	56.41	62.91	link

Training

Use train.sh to train RTD-Net.


# First stage

CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node=2 --master_port=11323 --use_env main.py --window_size 100 --batch_size 32 --stage 1 --num_queries 32 --point_prob_normalize

# Second stage for relaxation mechanism

CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node=2 --master_port=11324 --use_env main.py --window_size 100 --batch_size 32 --lr 1e-5 --stage 2 --epochs 10 --lr_drop 5 --num_queries 32 --point_prob_normalize --load outputs/checkpoint_best_sum_ar.pth

# Third stage for completeness head

CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node=2 --master_port=11325 --use_env main.py --window_size 100 --batch_size 32 --lr 1e-4 --stage 3 --epochs 20 --num_queries 32 --point_prob_normalize --load outputs/checkpoint_best_sum_ar.pth

Testing

Inference with test.sh.

CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node=2 --master_port=11325 --use_env main.py --window_size 100 --batch_size 32 --lr 1e-4 --stage 3 --epochs 20 --num_queries 32 --point_prob_normalize --eval --resume outputs/checkpoint_best_sum_ar.pth

References

We especially thank the contributors of the BSN, G-TAD and DETR for providing helpful code.

Citations

If you think our work is helpful, please feel free to cite our paper.

@InProceedings{Tan_2021_RTD,
    author    = {Tan, Jing and Tang, Jiaqi and Wang, Limin and Wu, Gangshan},
    title     = {Relaxed Transformer Decoders for Direct Action Proposal Generation},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2021},
    pages     = {13526-13535}
}

Contact

For any question, please file an issue or contact

Jing Tan: [email protected]
Jiaqi Tang: [email protected]

[ICCV 2021] Relaxed Transformer Decoders for Direct Action Proposal Generation

Related tags

Overview

RTD-Net (ICCV 2021)

News

Overview

Dependencies

Data Preparation

Checkpoint

Training

Testing

References

Citations

Contact

Owner

Multimedia Computing Group, Nanjing University

Personal project about genus-0 meshes, spherical harmonics and a cow

Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators

Lacmus is a cross-platform application that helps to find people who are lost in the forest using computer vision and neural networks.

Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more

:boar: :bear: Deep Learning based Python Library for Stock Market Prediction and Modelling

Implementation of the paper Scalable Intervention Target Estimation in Linear Models (NeurIPS 2021), and the code to generate simulation results.

Learning recognition/segmentation models without end-to-end training. 40%-60% less GPU memory footprint. Same training time. Better performance.

Torchserve server using a YoloV5 model running on docker with GPU and static batch inference to perform production ready inference.

SNIPS: Solving Noisy Inverse Problems Stochastically

Audio-Visual Generalized Few-Shot Learning with Prototype-Based Co-Adaptation

Code for 'Self-Guided and Cross-Guided Learning for Few-shot segmentation. (CVPR' 2021)'

This repository contains the source codes for the paper AtlasNet V2 - Learning Elementary Structures.

Repository providing a wide range of self-supervised pretrained models for computer vision tasks.

Pytorch reimplement of the paper "A Novel Cascade Binary Tagging Framework for Relational Triple Extraction" ACL2020. The original code is written in keras.

A modified version of DeepMind's Alphafold2 to divide CPU part (MSA and template searching) and GPU part (prediction model)

NeoDTI: Neural integration of neighbor information from a heterogeneous network for discovering new drug-target interactions

This repository provides a PyTorch implementation and model weights for HCSC (Hierarchical Contrastive Selective Coding)

This repository will be a summary and outlook on all our open, medical, AI advancements.

Easy-to-use library to boost AI inference leveraging state-of-the-art optimization techniques.

Official PyTorch implementation for paper Context Matters: Graph-based Self-supervised Representation Learning for Medical Images