This repository contains the code for the paper "Hierarchical Motion Understanding via Motion Programs"

Overview

Hierarchical Motion Understanding via Motion Programs (CVPR 2021)

Paper

This repository contains the official implementation of:

Hierarchical Motion Understanding via Motion Programs

full paper | short talk | long talk | project webpage

Motion Programs example

Running motion2prog

0. We start with video file and first prepare the input data

$ ffmpeg -i ${video_dir}/video.mp4 ${video_dir}/frames/%05d.jpg
$ python AlphaPose/scripts/demo_inference.py \
    --cfg AlphaPose/pretrained_models/256x192_res50_lr1e-3_1x.yaml \
    --checkpoint AlphaPose/pretrained_models/halpe26_fast_res50_256x192.pth \
    --indir ${video_dir}/frames --outdir ${video_dir}/pose_mpii_track \
    --pose_track --showbox --flip --qsize 256
$ mv ${video_dir}/pose_mpii_track/alphapose-results.json \
    ${video_dir}/alphapose-results-halpe26-posetrack.json

We packaged a demo video with necessary inputs for quickly testing our code

$ wget https://sumith1896.github.io/motion2prog/static/demo.zip
$ mv demo.zip data/  && cd data/ && unzip demo.zip && cd ..
  • We need 2D pose detection results & extracted frames of video (for visualization)

  • We support loading from different pose detector formats in the load function in lkeypoints.py.

  • We used AlphaPose with the above commands for all pose detection results.

Run motion program synthesis pipeline

1. With the data prepared, you can run the synthesis with the following command:

$ python fit.py -d data/demo/276_reg -k coco -a -x -c -p 1 -w 20 --no-acc \
--stat-thres 5 --span-thres 5 --cores 9 -r 1600 -o ./visualization/static/data/demo
  • The various options and their descriptions are explained in the fit.py file.

  • The results can be found under ./visualization/static/data/demo.

Visualizing the synthesized programs

2. We package a visualization server for visualizing the generated programs

$ cd visualization/
$ bash deploy.sh p
  • Open the directed the webpage and browse the results interactively.

Citations

If you find our code or paper useful to your research, please consider citing:

@inproceedings{motion2prog2021,
    Author = {Sumith Kulal and Jiayuan Mao and Alex Aiken and Jiajun Wu},
    Title = {Hierarchical Motion Understanding via Motion Programs},
    booktitle={CVPR},
    year={2021},
}

Checklist

Please open a GitHub issue or contact [email protected] for any issues or questions!

  • Upload pre-processed data used in paper.
  • Add for-loop synthesis layer.

Acknowledgements

We thank Karan Chadha, Shivam Garg and Shubham Goel for helpful discussions. This work is in part supported by Magic Grant from the Brown Institute for Media Innovation, the Samsung Global Research Outreach (GRO) Program, Autodesk, Amazon Web Services, and Stanford HAI for AWS Cloud Credits.

Parts of this repo use materials from SCANimate and fit.

Owner
Sumith Kulal
Insanely passionate about Computer Science.
Sumith Kulal
Unsupervised Domain Adaptation for Nighttime Aerial Tracking (CVPR2022)

Unsupervised Domain Adaptation for Nighttime Aerial Tracking (CVPR2022) Junjie Ye, Changhong Fu, Guangze Zheng, Danda Pani Paudel, and Guang Chen. Uns

Intelligent Vision for Robotics in Complex Environment 91 Dec 30, 2022
Optimized primitives for collective multi-GPU communication

NCCL Optimized primitives for inter-GPU communication. Introduction NCCL (pronounced "Nickel") is a stand-alone library of standard communication rout

NVIDIA Corporation 2k Jan 09, 2023
Sequence to Sequence Models with PyTorch

Sequence to Sequence models with PyTorch This repository contains implementations of Sequence to Sequence (Seq2Seq) models in PyTorch At present it ha

Sandeep Subramanian 708 Dec 19, 2022
Deep motion transfer

animation-with-keypoint-mask Paper The right most square is the final result. Softmax mask (circles): \ Heatmap mask: \ conda env create -f environmen

9 Nov 01, 2022
Automatic Differentiation Multipole Moment Molecular Forcefield

Automatic Differentiation Multipole Moment Molecular Forcefield Performance notes On a single gpu, using waterbox_31ang.pdb example from MPIDplugin wh

4 Jan 07, 2022
A Genetic Programming platform for Python with TensorFlow for wicked-fast CPU and GPU support.

Karoo GP Karoo GP is an evolutionary algorithm, a genetic programming application suite written in Python which supports both symbolic regression and

Kai Staats 149 Jan 09, 2023
A general-purpose encoder-decoder framework for Tensorflow

READ THE DOCUMENTATION CONTRIBUTING A general-purpose encoder-decoder framework for Tensorflow that can be used for Machine Translation, Text Summariz

Google 5.5k Jan 07, 2023
This project generates news headlines using a Long Short-Term Memory (LSTM) neural network.

News Headlines Generator bunnysaini/Generate-Headlines Goal This project aims to generate news headlines using a Long Short-Term Memory (LSTM) neural

Bunny Saini 1 Jan 24, 2022
Atif Hassan 103 Dec 14, 2022
Blender add-on: Add to Cameras menu: View → Camera, View → Add Camera, Camera → View, Previous Camera, Next Camera

Blender add-on: Camera additions In 3D view, it adds these actions to the View|Cameras menu: View → Camera : set the current camera to the 3D view Vie

German Bauer 11 Feb 08, 2022
Code accompanying "Dynamic Neural Relational Inference" from CVPR 2020

Code accompanying "Dynamic Neural Relational Inference" This codebase accompanies the paper "Dynamic Neural Relational Inference" from CVPR 2020. This

Colin Graber 48 Dec 23, 2022
MLOps will help you to understand how to build a Continuous Integration and Continuous Delivery pipeline for an ML/AI project.

page_type languages products description sample python azure azure-machine-learning-service azure-devops Code which demonstrates how to set up and ope

1 Nov 01, 2021
Computer-Vision-Paper-Reviews - Computer Vision Paper Reviews with Key Summary along Papers & Codes

Computer-Vision-Paper-Reviews Computer Vision Paper Reviews with Key Summary along Papers & Codes. Jonathan Choi 2021 50+ Papers across Computer Visio

Jonathan Choi 2 Mar 17, 2022
Xintao 1.4k Dec 25, 2022
PyTorch implementations of the beta divergence loss.

Beta Divergence Loss - PyTorch Implementation This repository contains code for a PyTorch implementation of the beta divergence loss. Dependencies Thi

Billy Carson 7 Nov 09, 2022
Graph Self-Attention Network for Learning Spatial-Temporal Interaction Representation in Autonomous Driving

GSAN Introduction Code for paper GSAN: Graph Self-Attention Network for Learning Spatial-Temporal Interaction Representation in Autonomous Driving, wh

YE Luyao 6 Oct 27, 2022
Strongly local p-norm-cut algorithms for semi-supervised learning and local graph clustering

Strongly local p-norm-cut algorithms for semi-supervised learning and local graph clustering

Meng Liu 2 Jul 19, 2022
Official PyTorch implementation of "BlendGAN: Implicitly GAN Blending for Arbitrary Stylized Face Generation" (NeurIPS 2021)

BlendGAN: Implicitly GAN Blending for Arbitrary Stylized Face Generation Official PyTorch implementation of the NeurIPS 2021 paper Mingcong Liu, Qiang

onion 462 Dec 29, 2022
In the case of your data having only 1 channel while want to use timm models

timm_custom Description In the case of your data having only 1 channel while want to use timm models (with or without pretrained weights), run the fol

2 Nov 26, 2021
Exponential Graph is Provably Efficient for Decentralized Deep Training

Exponential Graph is Provably Efficient for Decentralized Deep Training This code repository is for the paper Exponential Graph is Provably Efficient

3 Apr 20, 2022