This repository contains the official implementation of:

Hierarchical Motion Understanding via Motion Programs

full paper | short talk | long talk | project webpage

0. We start with video file and first prepare the input data

$ ffmpeg -i ${video_dir}/video.mp4 ${video_dir}/frames/%05d.jpg
$ python ${alphapose_dir}/scripts/demo_inference.py \
    --cfg ${alphapose_dir}/pretrained_models/256x192_res50_lr1e-3_1x.yaml \
    --checkpoint ${alphapose_dir}/pretrained_models/halpe26_fast_res50_256x192.pth \
    --indir ${video_dir}/frames --outdir ${video_dir}/pose_mpii_track \
    --pose_track --showbox --flip --qsize 256
$ mv ${video_dir}/pose_mpii_track/alphapose-results.json \
    ${video_dir}/alphapose-results-halpe26-posetrack.json

We packaged a demo video with necessary inputs for quickly testing our code

$ wget https://sumith1896.github.io/motion2prog/static/demo.zip
$ mv demo.zip data/  && cd data/ && unzip demo.zip && cd ..

• We need 2D pose detection results & extracted frames of video (for visualization)

• We support loading from different pose detector formats in the load function in lkeypoints.py.

• We used AlphaPose with the above commands for all pose detection results.

1. With the data prepared, you can run the synthesis with the following command:

$ python fit.py -d data/demo/276_reg -k coco -a -x -c -p 1 -w 20 --no-acc \
--stat-thres 5 --span-thres 5 --cores 9 -r 1600 -o ./visualization/static/data/demo

• The various options and their descriptions are explained in the fit.py file.

• The results can be found under ./visualization/static/data/demo.

2. We package a visualization server for visualizing the generated programs

$ cd visualization/
$ bash deploy.sh p

• Open the directed the webpage and browse the results interactively.

If you find our code or paper useful to your research, please consider citing:

@inproceedings{motion2prog2021,
    Author = {Sumith Kulal and Jiayuan Mao and Alex Aiken and Jiajun Wu},
    Title = {Hierarchical Motion Understanding via Motion Programs},
    booktitle={CVPR},
    year={2021},
}

Please open a GitHub issue or contact sumith@cs.stanford.edu for any issues or questions!

• Upload pre-processed data used in paper.

We thank Karan Chadha, Shivam Garg and Shubham Goel for helpful discussions. This work is in part supported by Magic Grant from the Brown Institute for Media Innovation, the Samsung Global Research Outreach (GRO) Program, Autodesk, Amazon Web Services, and Stanford HAI for AWS Cloud Credits.

Parts of this repo use materials from SCANimate and fit.