Official PyTorch Implementation of "AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting".

Overview

AgentFormer

This repo contains the official implementation of our paper:

AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting
Ye Yuan, Xinshuo Weng, Yanglan Ou, Kris Kitani
ICCV 2021
[website] [paper]

Overview

Loading AgentFormer Overview

Important Note

We have recently noticed a normalization bug in the code and after fixing it, the performance of our method is worse than the original numbers reported in the ICCV paper. For comparision, please use the correct numbers in the updated arXiv version.

Installation

Environment

  • Tested OS: MacOS, Linux
  • Python >= 3.7
  • PyTorch == 1.8.0

Dependencies:

  1. Install PyTorch 1.8.0 with the correct CUDA version.
  2. Install the dependencies:
    pip install -r requirements.txt
    

Datasets

  • For the ETH/UCY dataset, we already included a converted version compatible with our dataloader under datasets/eth_ucy.
  • For the nuScenes dataset, the following steps are required:
    1. Download the orignal nuScenes dataset. Checkout the instructions here.
    2. Follow the instructions of nuScenes prediction challenge. Download and install the map expansion.
    3. Run our script to obtain a processed version of the nuScenes dataset under datasets/nuscenes_pred:
      python data/process_nuscenes.py --data_root <PATH_TO_NUSCENES>
      

Pretrained Models

  • You can download pretrained models from Google Drive or BaiduYun (password: 9rvb) to reproduce the numbers in the paper.
  • Once the agentformer_models.zip file is downloaded, place it under the root folder of this repo and unzip it:
    unzip agentformer_models.zip
    
    This will place the models under the results folder. Note that the pretrained models directly correspond to the config files in cfg.

Evaluation

ETH/UCY

Run the following command to test pretrained models for the ETH dataset:

python test.py --cfg eth_agentformer --gpu 0

You can replace eth with {hotel, univ, zara1, zara2} to test other datasets in ETH/UCY. You should be able to get the numbers reported in the paper as shown in this table:

Ours ADE FDE
ETH 0.45 0.75
Hotel 0.14 0.22
Univ 0.25 0.45
Zara1 0.18 0.30
Zara2 0.14 0.24
Avg 0.23 0.39

nuScenes

Run the following command to test pretrained models for the nuScenes dataset:

python test.py --cfg nuscenes_5sample_agentformer --gpu 0

You can replace 5sample with 10sample to compute all the metrics (ADE_5, FDE_5, ADE_10, FDE_10). You should be able to get the numbers reported in the paper as shown in this table:

ADE_5 FDE_5 ADE_10 FDE_10
Ours 1.856 3.889 1.452 2.856

Training

You can train your own models with your customized configs. Here we take the ETH dataset as an example, but you can train models for other datasets with their corresponding configs. AgentFormer requires two-stage training:

  1. Train the AgentFormer VAE model (everything but the trajectory sampler):
    python train.py --cfg user_eth_agentformer_pre --gpu 0
    
  2. Once the VAE model is trained, train the AgentFormer DLow model (trajectory sampler):
    python train.py --cfg user_eth_agentformer --gpu 0
    
    Note that you need to change the pred_cfg field in user_eth_agentformer to the config you used in step 1 (user_eth_agentformer_pre) and change the pred_epoch to the VAE model epoch you want to use.

Citation

If you find our work useful in your research, please cite our paper AgentFormer:

@inproceedings{yuan2021agent,
  title={AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting},
  author={Yuan, Ye and Weng, Xinshuo and Ou, Yanglan and Kitani, Kris},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  year={2021}
}

License

Please see the license for further details.

Owner
Ye Yuan
PhD student at Robotics Institute, CMU
Ye Yuan
A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks.

Light Gradient Boosting Machine LightGBM is a gradient boosting framework that uses tree based learning algorithms. It is designed to be distributed a

Microsoft 14.5k Jan 08, 2023
Optimized Gillespie algorithm for simulating Stochastic sPAtial models of Cancer Evolution (OG-SPACE)

OG-SPACE Introduction Optimized Gillespie algorithm for simulating Stochastic sPAtial models of Cancer Evolution (OG-SPACE) is a computational framewo

Data and Computational Biology Group UNIMIB (was BI*oinformatics MI*lan B*icocca) 0 Nov 17, 2021
Neural network for recognizing the gender of people in photos

Neural Network For Gender Recognition How to test it? Install requirements.txt file using pip install -r requirements.txt command Run nn.py using pyth

Valery Chapman 1 Sep 18, 2022
TensorFlow implementation of "Learning from Simulated and Unsupervised Images through Adversarial Training"

Simulated+Unsupervised (S+U) Learning in TensorFlow TensorFlow implementation of Learning from Simulated and Unsupervised Images through Adversarial T

Taehoon Kim 569 Dec 29, 2022
Tensorflow Tutorials using Jupyter Notebook

Tensorflow Tutorials using Jupyter Notebook TensorFlow tutorials written in Python (of course) with Jupyter Notebook. Tried to explain as kindly as po

Sungjoon 2.6k Dec 22, 2022
An AI made using artificial intelligence (AI) and machine learning algorithms (ML) .

DTech.AIML An AI made using artificial intelligence (AI) and machine learning algorithms (ML) . This is created by help of some members in my team and

1 Jan 06, 2022
Instance Semantic Segmentation List

Instance Semantic Segmentation List This repository contains lists of state-or-art instance semantic segmentation works. Papers and resources are list

bighead 87 Mar 06, 2022
A powerful framework for decentralized federated learning with user-defined communication topology

Scatterbrained Decentralized Federated Learning Scatterbrained makes it easy to build federated learning systems. In addition to traditional federated

Johns Hopkins Applied Physics Laboratory 7 Sep 26, 2022
An air quality monitoring service with a Raspberry Pi and a SDS011 sensor.

Raspberry Pi Air Quality Monitor A simple air quality monitoring service for the Raspberry Pi. Installation Clone the repository and run the following

rydercalmdown 24 Dec 09, 2022
CTC segmentation python package

CTC segmentation CTC segmentation can be used to find utterances alignments within large audio files. This repository contains the ctc-segmentation py

Ludwig Kürzinger 217 Jan 04, 2023
This repository accompanies our paper “Do Prompt-Based Models Really Understand the Meaning of Their Prompts?”

This repository accompanies our paper “Do Prompt-Based Models Really Understand the Meaning of Their Prompts?” Usage To replicate our results in Secti

Albert Webson 64 Dec 11, 2022
3D ResNet Video Classification accelerated by TensorRT

Activity Recognition TensorRT Perform video classification using 3D ResNets trained on Kinetics-400 dataset and accelerated with TensorRT P.S Click on

Akash James 39 Nov 21, 2022
tmm_fast is a lightweight package to speed up optical planar multilayer thin-film device computation.

tmm_fast tmm_fast or transfer-matrix-method_fast is a lightweight package to speed up optical planar multilayer thin-film device computation. It is es

26 Dec 11, 2022
Study of human inductive biases in CNNs and Transformers.

Are Convolutional Neural Networks or Transformers more like human vision? This repository contains the code and fine-tuned models of popular Convoluti

Shikhar Tuli 39 Dec 08, 2022
Rax is a Learning-to-Rank library written in JAX

🦖 Rax: Composable Learning to Rank using JAX Rax is a Learning-to-Rank library written in JAX. Rax provides off-the-shelf implementations of ranking

Google 247 Dec 27, 2022
Code repository for paper `Skeleton Merger: an Unsupervised Aligned Keypoint Detector`.

Skeleton Merger Skeleton Merger, an Unsupervised Aligned Keypoint Detector. The paper is available at https://arxiv.org/abs/2103.10814. A map of the r

北海若 48 Nov 14, 2022
Repository for MeshTalk supplemental material and code once the (already approved) 16 GHS captures our lab will make publicly available are released.

meshtalk This repository contains code to run MeshTalk for face animation from audio. If you use MeshTalk, please cite @inproceedings{richard2021mesht

Meta Research 221 Jan 06, 2023
OREO: Object-Aware Regularization for Addressing Causal Confusion in Imitation Learning (NeurIPS 2021)

OREO: Object-Aware Regularization for Addressing Causal Confusion in Imitation Learning (NeurIPS 2021) Video demo We here provide a video demo from co

20 Nov 25, 2022
Code for Towards Streaming Perception (ECCV 2020) :car:

sAP — Code for Towards Streaming Perception ECCV Best Paper Honorable Mention Award Feb 2021: Announcing the Streaming Perception Challenge (CVPR 2021

Martin Li 85 Dec 22, 2022
Official repository for "Deep Recurrent Neural Network with Multi-scale Bi-directional Propagation for Video Deblurring".

RNN-MBP Deep Recurrent Neural Network with Multi-scale Bi-directional Propagation for Video Deblurring (AAAI-2022) by Chao Zhu, Hang Dong, Jinshan Pan

SIV-LAB 22 Aug 31, 2022