[arXiv] What-If Motion Prediction for Autonomous Driving ❓🚗💨

Last update: Dec 29, 2022

Overview

WIMP - What If Motion Predictor

Reference PyTorch Implementation for What If Motion Prediction [PDF] [Dynamic Visualizations]

Setup

Requirements

The WIMP reference implementation and setup procedure has been tested to work with Ubuntu 16.04+ and has the following requirements:

python >= 3.7
pytorch >= 1.5.0

Installing Dependencies

Install remaining required Python dependencies using pip.
```
pip install -r requirements.txt
```
Install the Argoverse API module into the local Python environment by following steps 1, 2, and 4 in the README.

Argoverse Data

In order to set up the Argoverse dataset for training and evaluation, follow the steps below:

Download the the Argoverse Motion Forecasting v1.1 dataset and extract the compressed data subsets such that the raw CSV files are stored in the following directory structure:

├── WIMP
│   ├── src
│   ├── scripts
│   ├── data
│   │   ├── argoverse_raw
│   │   │   ├── train
│   │   │   │   ├── *.csv
│   │   │   ├── val
│   │   │   │   ├── *.csv
│   │   │   ├── test
│   │   │   │   ├── *.csv

Pre-process the raw Argoverse data into a WIMP-compatible format by running the following script. It should be noted that the Argoverse dataset is quite large and this script may take a few hours to run on a multi-threaded machine.

python scripts/run_preprocess.py --dataroot ./data/argoverse_raw/ \
--mode val --save-dir ./data/argoverse_processed --social-features \
--map-features --xy-features --normalize --extra-map-features \
--compute-all --generate-candidate-centerlines 6

Usage

For a detailed description of all possible configuration arguments, please run scripts with the -h flag.

Training

To train WIMP from scratch using a configuration similar to that reported in the paper, run a variant of the following command:

python src/main.py --mode train --dataroot ./data/argoverse_processed --IFC \
--lr 0.0001 --weight-decay 0.0 --non-linearity relu  --use-centerline-features \
--segment-CL-Encoder-Prob --num-mixtures 6 --output-conv --output-prediction \
--gradient-clipping --hidden-key-generator --k-value-threshold 10 \
--scheduler-step-size 60 90 120 150 180  --distributed-backend ddp \
--experiment-name example --gpus 4 --batch-size 25

Citing

If you've found this code to be useful, please consider citing our paper!

@article{khandelwal2020if,
  title={What-If Motion Prediction for Autonomous Driving},
  author={Khandelwal, Siddhesh and Qi, William and Singh, Jagjeet and Hartnett, Andrew and Ramanan, Deva},
  journal={arXiv preprint arXiv:2008.10587},
  year={2020}
}

Questions

This repo is maintained by William Qi and Siddhesh Khandelwal - please feel free to reach out or open an issue if you have additional questions/concerns.

We plan to clean up the codebase and add some additional utilities (possibly NuScenes data loaders and inference/visualization tools) in the near future, but don't expect to make significant breaking changes.

Comments

Pandas Error runpreprocess.py

Hello! First of all, thank you for making your code available for the readers of your great paper. I am having an issue while running run_preprocess.py. I think while reading the csv something goes wrong since my error is a pandas error. When I try to run the script, it gives me: KeyError: 'CITY_NAME' When I go to the script and give "MIA" as the CITY_NAME, just to see what happens, I receive a similar error: KeyError: 'OBJECT_TYPE' I checked the paths for the data. It seems fine. What could be the reason? Thank you!

opened by ahmetgurhan 0

Loss dimensions

Hi, thank you so much for your fantastic work.

Which is the order, and the dimensions, in this function?

def l1_ewta_loss(prediction, target, k=6, eps=1e-7, mr=2.0):
    num_mixtures = prediction.shape[1]

    target = target.unsqueeze(1).expand(-1, num_mixtures, -1, -1)
    l1_loss = nn.functional.l1_loss(prediction, target, reduction='none').sum(dim=[2, 3])

    # Get loss from top-k mixtures for each timestep
    mixture_loss_sorted, mixture_ranks = torch.sort(l1_loss, descending=False)
    mixture_loss_topk = mixture_loss_sorted.narrow(1, 0, k)

    # Aggregate loss across timesteps and batch
    loss = mixture_loss_topk.sum()
    loss = loss / target.size(0)
    loss = loss / target.size(2)
    loss = loss / k
    return loss

I am not able to obtain good results compared to NLL. I have as inputs:

predictions: batch_size x num_modes x pred_len x data_dim (e.g. 1024 x 6 x 30 x 2) gt: batch_size x pred_len x data_dim (e.g. 1024 x 30 x 2)

Is this correct?

opened by Cram3r95 0

Reproducing the Map-Free and only Social-Context Results form the Ablation Study

Hey there,

I want to reproduce the results of your ablation study, where you only used Social-Context with EWTA-Loss.

However, I habe problems training the model only with social context. What are the correct flags I need to set for preprocessing (run_preprocess.py) and for training (main.py)?

Looking forward hearing from you soon!

Best regards

SchDevel

opened by SchDevel 2
Can I get your inference/visualization code?

Hi, first of all, thanks for your awesome work and sharing that to us.

I tried to make inference/visualization code by myself, unfortunately, there were some problems.

Maybe library's mismatching, my insufficient coding skills, or something else.

So, can i get your inference/visualization code or even skeleton base code?

opened by raspbe34 3
What is the method for incomplete trajectories?
Hi, thanks for sharing your great work~ I am wondering how you deal with the incomplete trajectories problem (agents have less then 2 seconds of history).

I notice that for the neighboring agent wrt focal agent, you discard all the agents (code) if their trajectories are not complete

how would you deal with those incomplete trajectories for the focal agent? Did you use interpolation or some techniques?

Thanks!
opened by XHwind 0

Releases(1.0)

1.0(Oct 8, 2020)

Initial code release for WIMP.
Source code(tar.gz)
Source code(zip)

Owner

William Qi

Prediction @argoai

GitHub Repository

The lightweight PyTorch wrapper for high-performance AI research. Scale your models, not the boilerplate.

The lightweight PyTorch wrapper for high-performance AI research. Scale your models, not the boilerplate. Website • Key Features • How To Use • Docs •

21.1k Dec 29, 2022

[SDM 2022] Towards Similarity-Aware Time-Series Classification

SimTSC This is the PyTorch implementation of SDM2022 paper Towards Similarity-Aware Time-Series Classification. We propose Similarity-Aware Time-Serie

49 Dec 27, 2022

Neural Ensemble Search for Performant and Calibrated Predictions

Neural Ensemble Search Introduction This repo contains the code accompanying the paper: Neural Ensemble Search for Performant and Calibrated Predictio

26 Dec 12, 2022

PyTorch implementation of TSception V2 using DEAP dataset

TSception This is the PyTorch implementation of TSception V2 using DEAP dataset in our paper: Yi Ding, Neethu Robinson, Su Zhang, Qiuhao Zeng, Cuntai

27 Dec 15, 2022

Relative Uncertainty Learning for Facial Expression Recognition

Relative Uncertainty Learning for Facial Expression Recognition The official implementation of the following paper at NeurIPS2021: Title: Relative Unc

35 Dec 28, 2022

Some useful blender add-ons for SMPL skeleton's poses and global translation.

Blender add-ons for SMPL skeleton's poses and trans There are two blender add-ons for SMPL skeleton's poses and trans.The first is for making an offli

154 Jan 04, 2023

The dynamics of representation learning in shallow, non-linear autoencoders

The dynamics of representation learning in shallow, non-linear autoencoders The package is written in python and uses the pytorch implementation to ML

4 Jun 08, 2022

Speeding-Up Back-Propagation in DNN: Approximate Outer Product with Memory

Approximate Outer Product Gradient Descent with Memory Code for the numerical experiment of the paper Speeding-Up Back-Propagation in DNN: Approximate

2 Mar 02, 2022

PaddlePaddle GAN library, including lots of interesting applications like First-Order motion transfer, wav2lip, picture repair, image editing, photo2cartoon, image style transfer, and so on.

English | 简体中文 PaddleGAN PaddleGAN provides developers with high-performance implementation of classic and SOTA Generative Adversarial Networks, and s

6.4k Jan 09, 2023

Categorizing comments on YouTube into different categories.

Youtube Comments Categorization This repo is for categorizing comments on a youtube video into different categories. negative (grievances, complaints,

5 Nov 26, 2022

This package is for running the semantic SLAM algorithm using extracted planar surfaces from the received detection

Semantic SLAM This package can perform optimization of pose estimated from VO/VIO methods which tend to drift over time. It uses planar surfaces extra

125 Dec 02, 2022

ImageNet Adversarial Image Evaluation

ImageNet Adversarial Image Evaluation This repository contains the code and some materials used in the experimental work presented in the following pa

11 Dec 26, 2022

Implementation of the famous Image Manipulation\Forgery Detector "ManTraNet" in Pytorch

Who has never met a forged picture on the web ? No one ! Everyday we are constantly facing fake pictures touched up in Photoshop but it is not always

77 Dec 16, 2022

ConE: Cone Embeddings for Multi-Hop Reasoning over Knowledge Graphs

ConE: Cone Embeddings for Multi-Hop Reasoning over Knowledge Graphs This is the code of paper ConE: Cone Embeddings for Multi-Hop Reasoning over Knowl

33 Dec 07, 2022

Moon-patrol - A faithful recreation of the 1983 hit classic Moon Patrol for the Atari 2600 created using the Pygame library for Python

Moon Patrol A recreation of the hit Atari 2600 game, Moon Patrol Moon Patrol is

3 Apr 20, 2022