[arXiv] What-If Motion Prediction for Autonomous Driving ❓🚗💨

Last update: Dec 29, 2022

Overview

WIMP - What If Motion Predictor

Reference PyTorch Implementation for What If Motion Prediction [PDF] [Dynamic Visualizations]

Setup

Requirements

The WIMP reference implementation and setup procedure has been tested to work with Ubuntu 16.04+ and has the following requirements:

python >= 3.7
pytorch >= 1.5.0

Installing Dependencies

Install remaining required Python dependencies using pip.
```
pip install -r requirements.txt
```
Install the Argoverse API module into the local Python environment by following steps 1, 2, and 4 in the README.

Argoverse Data

In order to set up the Argoverse dataset for training and evaluation, follow the steps below:

Download the the Argoverse Motion Forecasting v1.1 dataset and extract the compressed data subsets such that the raw CSV files are stored in the following directory structure:

├── WIMP
│   ├── src
│   ├── scripts
│   ├── data
│   │   ├── argoverse_raw
│   │   │   ├── train
│   │   │   │   ├── *.csv
│   │   │   ├── val
│   │   │   │   ├── *.csv
│   │   │   ├── test
│   │   │   │   ├── *.csv

Pre-process the raw Argoverse data into a WIMP-compatible format by running the following script. It should be noted that the Argoverse dataset is quite large and this script may take a few hours to run on a multi-threaded machine.

python scripts/run_preprocess.py --dataroot ./data/argoverse_raw/ \
--mode val --save-dir ./data/argoverse_processed --social-features \
--map-features --xy-features --normalize --extra-map-features \
--compute-all --generate-candidate-centerlines 6

Usage

For a detailed description of all possible configuration arguments, please run scripts with the -h flag.

Training

To train WIMP from scratch using a configuration similar to that reported in the paper, run a variant of the following command:

python src/main.py --mode train --dataroot ./data/argoverse_processed --IFC \
--lr 0.0001 --weight-decay 0.0 --non-linearity relu  --use-centerline-features \
--segment-CL-Encoder-Prob --num-mixtures 6 --output-conv --output-prediction \
--gradient-clipping --hidden-key-generator --k-value-threshold 10 \
--scheduler-step-size 60 90 120 150 180  --distributed-backend ddp \
--experiment-name example --gpus 4 --batch-size 25

Citing

If you've found this code to be useful, please consider citing our paper!

@article{khandelwal2020if,
  title={What-If Motion Prediction for Autonomous Driving},
  author={Khandelwal, Siddhesh and Qi, William and Singh, Jagjeet and Hartnett, Andrew and Ramanan, Deva},
  journal={arXiv preprint arXiv:2008.10587},
  year={2020}
}

Questions

This repo is maintained by William Qi and Siddhesh Khandelwal - please feel free to reach out or open an issue if you have additional questions/concerns.

We plan to clean up the codebase and add some additional utilities (possibly NuScenes data loaders and inference/visualization tools) in the near future, but don't expect to make significant breaking changes.

Comments

Pandas Error runpreprocess.py

Hello! First of all, thank you for making your code available for the readers of your great paper. I am having an issue while running run_preprocess.py. I think while reading the csv something goes wrong since my error is a pandas error. When I try to run the script, it gives me: KeyError: 'CITY_NAME' When I go to the script and give "MIA" as the CITY_NAME, just to see what happens, I receive a similar error: KeyError: 'OBJECT_TYPE' I checked the paths for the data. It seems fine. What could be the reason? Thank you!

opened by ahmetgurhan 0

Loss dimensions

Hi, thank you so much for your fantastic work.

Which is the order, and the dimensions, in this function?

def l1_ewta_loss(prediction, target, k=6, eps=1e-7, mr=2.0):
    num_mixtures = prediction.shape[1]

    target = target.unsqueeze(1).expand(-1, num_mixtures, -1, -1)
    l1_loss = nn.functional.l1_loss(prediction, target, reduction='none').sum(dim=[2, 3])

    # Get loss from top-k mixtures for each timestep
    mixture_loss_sorted, mixture_ranks = torch.sort(l1_loss, descending=False)
    mixture_loss_topk = mixture_loss_sorted.narrow(1, 0, k)

    # Aggregate loss across timesteps and batch
    loss = mixture_loss_topk.sum()
    loss = loss / target.size(0)
    loss = loss / target.size(2)
    loss = loss / k
    return loss

I am not able to obtain good results compared to NLL. I have as inputs:

predictions: batch_size x num_modes x pred_len x data_dim (e.g. 1024 x 6 x 30 x 2) gt: batch_size x pred_len x data_dim (e.g. 1024 x 30 x 2)

Is this correct?

opened by Cram3r95 0

Reproducing the Map-Free and only Social-Context Results form the Ablation Study

Hey there,

I want to reproduce the results of your ablation study, where you only used Social-Context with EWTA-Loss.

However, I habe problems training the model only with social context. What are the correct flags I need to set for preprocessing (run_preprocess.py) and for training (main.py)?

Looking forward hearing from you soon!

Best regards

SchDevel

opened by SchDevel 2
Can I get your inference/visualization code?

Hi, first of all, thanks for your awesome work and sharing that to us.

I tried to make inference/visualization code by myself, unfortunately, there were some problems.

Maybe library's mismatching, my insufficient coding skills, or something else.

So, can i get your inference/visualization code or even skeleton base code?

opened by raspbe34 3
What is the method for incomplete trajectories?
Hi, thanks for sharing your great work~ I am wondering how you deal with the incomplete trajectories problem (agents have less then 2 seconds of history).

I notice that for the neighboring agent wrt focal agent, you discard all the agents (code) if their trajectories are not complete

how would you deal with those incomplete trajectories for the focal agent? Did you use interpolation or some techniques?

Thanks!
opened by XHwind 0

Releases(1.0)

1.0(Oct 8, 2020)

Initial code release for WIMP.
Source code(tar.gz)
Source code(zip)

Owner

William Qi

Prediction @argoai

GitHub Repository

Direct application of DALLE-2 to video synthesis, using factored space-time Unet and Transformers

DALLE2 Video (wip) ** only to be built after DALLE2 image is done and replicated, and the importance of the prior network is validated ** Direct appli

105 May 15, 2022

Official Code for ICML 2021 paper "Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline"

Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline Ankit Goyal, Hei Law, Bowei Liu, Alejandro Newell, Jia Deng Internati

115 Jan 04, 2023

This is the official code of our paper "Diversity-based Trajectory and Goal Selection with Hindsight Experience Relay" (PRICAI 2021)

Diversity-based Trajectory and Goal Selection with Hindsight Experience Replay This is the official implementation of our paper "Diversity-based Traje

6 Jul 18, 2022

The source codes for ACL 2021 paper 'BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data'

BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data This repository provides the implementation details for

124 Dec 27, 2022

[arXiv] What-If Motion Prediction for Autonomous Driving ❓🚗💨

Related tags

Overview

WIMP - What If Motion Predictor

Setup

Requirements

Installing Dependencies

Argoverse Data

Usage

Training

Citing

Questions

Comments

Pandas Error runpreprocess.py

Loss dimensions

Reproducing the Map-Free and only Social-Context Results form the Ablation Study

Can I get your inference/visualization code?

What is the method for incomplete trajectories?

Releases(1.0)

1.0(Oct 8, 2020)

Owner

William Qi

Direct application of DALLE-2 to video synthesis, using factored space-time Unet and Transformers

Official Code for ICML 2021 paper "Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline"

This is the official code of our paper "Diversity-based Trajectory and Goal Selection with Hindsight Experience Relay" (PRICAI 2021)

Implementation of the paper Recurrent Glimpse-based Decoder for Detection with Transformer.

Calibrated Hyperspectral Image Reconstruction via Graph-based Self-Tuning Network.

Active window border replacement for window managers.

Machine Learning Time-Series Platform

This repo is developed for Strong Baseline For Vehicle Re-Identification in Track 2 Ai-City-2021 Challenges

DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism (SVS & TTS); AAAI 2022; Official code

利用python脚本实现微信、支付宝账单的合并，并保存到excel文件实现自动记账，可查看可视化图表。

Code for EMNLP'21 paper "Types of Out-of-Distribution Texts and How to Detect Them"

Convert ONNX model graph to Keras model format.

PyTorch implementation of our paper How robust are discriminatively trained zero-shot learning models?

In-place Parallel Super Scalar Samplesort (IPS⁴o)

CAMoE + Dual SoftMax Loss (DSL): Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss

PyTorch implementation for the Neuro-Symbolic Sudoku Solver leveraging the power of Neural Logic Machines (NLM)

PyTorch Implementation of DSB for Score Based Generative Modeling. Experiments managed using Hydra.

Trajectory Variational Autoencder baseline for Multi-Agent Behavior challenge 2022

[NeurIPS-2021] Slow Learning and Fast Inference: Efficient Graph Similarity Computation via Knowledge Distillation

The source codes for ACL 2021 paper 'BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data'