Official implementation of particle-based models (GNS and DPI-Net) on the Physion dataset.

Overview

Physion: Evaluating Physical Prediction from Vision in Humans and Machines [paper]

Daniel M. Bear, Elias Wang, Damian Mrowca, Felix J. Binder, Hsiao-Yu Fish Tung, R.T. Pramod, Cameron Holdaway, Sirui Tao, Kevin Smith, Fan-Yun Sun, Li Fei-Fei, Nancy Kanwisher, Joshua B. Tenenbaum, Daniel L.K. Yamins, Judith E. Fan

This is the official implementation of particle-based models (GNS and DPI-Net) on the Physion dataset. The code is built based on the original implementation of DPI-Net (https://github.com/YunzhuLi/DPI-Net).

Contact: [email protected] (Fish Tung)

Papers of GNS and DPI-Net:

** Learning to Simulate Complex Physics with Graph Networks ** [paper]

Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, Peter W. Battaglia

** Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids ** [website] [paper]

Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B. Tenenbaum, Antonio Torralba **

Demo

Rollout from our learned model (left is ground truth, right is prediction)

Dominoes Roll Contain Drape

Installation

Clone this repo:

git clone https://github.com/htung0101/DPI-Net-p.git
cd DPI-Net-p
git submodule update --init --recursive

Install Dependencies if using Conda

For Conda users, we provide an installation script:

bash ./scripts/conda_deps.sh
pip install pyyaml

To use tensorboard for training visualization

pip install tensorboardX
pip install tensorboard

Install binvox

We use binvox to transform object mesh into particles. To use binvox, please download binvox from https://www.patrickmin.com/binvox/, put it under ./bin, and include it in your path with

export PATH=$PATH:$PWD/bin.

You might need to do chmod 777 binvox in order to execute the file.

Setup your own data path

open paths.yaml and write your own path there. You can set up different paths for different machines under different user name.

Preprocessing the Physion dataset

1) We need to convert the mesh scenes into particle scenes. This line will generate a separate folder (dpi_data_dir specified in paths.yaml) that holds data for the particle-based models

bash run_preprocessing_tdw_cheap.sh [SCENARIO_NAME] [MODE]

e.g., bash run_preprocessing_tdw_cheap.sh Dominoes train SCENARIO_NAME can be one of the following: Dominoes, Collide, Support, Link, Contain, Roll, Drop, or Drape. MODE can be either train or test

You can visualize the original videos and the generated particle scenes with

python preprocessing_tdw_cheap.py --scenario Dominones --mode "train" --visualization 1

There will be videos generated under the folder vispy.

2) Then, try generate a train.txt and valid.txt files that indicates the trials you want to use for training and validaiton.

python create_train_valid.py

You can also design your specific split. Just put the trial names into one txt file.

3) For evalution on the red-hits-yellow prediciton, we can get the binary red-hits-yellow label txt file from the test dataset with

bash run_get_label_txt.sh [SCENARIO_NAME] test

This will generate a folder called labels under your output_folder dpi_data_dir. In the folder, each scenario will have a corresponding label file called [SCENARIO_NAME].txt

Training

Ok, now we are ready to start training the models.You can use the following command to train from scratch.

  • Train GNS
    bash scripts/train_gns.sh [SCENARIO_NAME] [GPU_ID]

SCENARIO_NAME can be one of the following: Dominoes, Collide, Support, Link, Contain, Roll, Drop and Drape.

  • Train DPI
    bash scripts/train_dpi.sh [SCENARIO_NAME] [GPU_ID]

Our implementation is different from the original DPI paper in 2 ways: (1) our model takes as inputs relative positions as opposed to absolute positions, (2) our model is trained with injected noise. These two features are suggested in the GNS paper, and we found them to be critcial for the models to generalize well to unseen scenes.

  • Train with multiple scenarios

You can also train with more than one scenarios by adding different scenario to the argument dataf

 python train.py  --env TDWdominoes --model_name GNS --log_per_iter 1000 --training_fpt 3 --ckp_per_iter 5000 --floor_cheat 1  --dataf "Dominoes, Collide, Support, Link, Roll, Drop, Contain, Drape" --outf "all_gns"
  • Visualize your training progress

Models and model logs are saved under [out_dir]/dump/dump_TDWdominoes. You can visualize the training progress using tensorboard

tensorboard --logdir MODEL_NAME/log

Evaluation

  • Evaluate GNS
bash scripts/eval_gns.sh [TRAIN_SCENARIO_NAME] [EPOCH] [ITER] [Test SCENARIO_NAME] [GPU_ID]

You can get the prediction txt file under eval/eval_TDWdominoes/[MODEL_NAME], e.g., test-Drape.txt, which contains results of testing the model on the Drape scenario. You can visualize the results with additional argument --vis 1.

  • Evaluate GNS-Ransac
bash scripts/eval_gns_ransac.sh [TRAIN_SCENARIO_NAME] [EPOCH] [ITER] [Test SCENARIO_NAME] [GPU_ID]
  • Evaluate DPI
bash scripts/eval_dpi.sh [TRAIN_SCENARIO_NAME] [EPOCH] [ITER] [Test SCENARIO_NAME] [GPU_ID]
  • Evaluate Models trained on multiple scenario Here we provide some example of evaluating on arbitray models trained on all scenarios.
bash eval_all_gns.sh [EPOCH] [ITER] [Test SCENARIO_NAME] [GPU_ID]
bash eval_all_dpi.sh [EPOCH] [ITER] [Test SCENARIO_NAME] [GPU_ID]
bash eval_all_gns_ransac.sh [EPOCH] [ITER] [Test SCENARIO_NAME] [GPU_ID]
  • Visualize trained Models Here we provide an example of visualizing the rollout results from trained arbitray models.
bash vis_gns.sh [EPOCH] [ITER] [Test SCENARIO_NAME] [GPU_ID]

You can find the visualization under eval/eval_TDWdominoes/[MODEL_NAME]/test-[Scenario]. We should see a gif for the original RGB videos, and another gif for the side-by-side comparison of gt particle scenes and the predicted particle scenes.

Citing Physion

If you find this codebase useful in your research, please consider citing:

@inproceedings{bear2021physion,
    Title={Physion: Evaluating Physical Prediction from Vision in Humans and Machines},
    author= {Daniel M. Bear and
           Elias Wang and
           Damian Mrowca and
           Felix J. Binder and
           Hsiao{-}Yu Fish Tung and
           R. T. Pramod and
           Cameron Holdaway and
           Sirui Tao and
           Kevin A. Smith and
           Fan{-}Yun Sun and
           Li Fei{-}Fei and
           Nancy Kanwisher and
           Joshua B. Tenenbaum and
           Daniel L. K. Yamins and
           Judith E. Fan},
    url = {https://arxiv.org/abs/2106.08261},
    archivePrefix = {arXiv},
    eprint = {2106.08261},
    Year = {2021}
}
Owner
Hsiao-Yu Fish Tung
Postdoc at MIT CoCosci Lab and Stanford NeuroAILab. PhD at CMU MLD
Hsiao-Yu Fish Tung
Code for DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learning

DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learning Pytorch Implementation for DisCo: Remedy Self-supervi

79 Jan 06, 2023
Semi-supervised Transfer Learning for Image Rain Removal. In CVPR 2019.

Semi-supervised Transfer Learning for Image Rain Removal This package contains the Python implementation of "Semi-supervised Transfer Learning for Ima

Wei Wei 59 Dec 26, 2022
Official implementation for "Symbolic Learning to Optimize: Towards Interpretability and Scalability"

Symbolic Learning to Optimize This is the official implementation for ICLR-2022 paper "Symbolic Learning to Optimize: Towards Interpretability and Sca

VITA 8 Dec 19, 2022
Official Pytorch implementation of the paper "MotionCLIP: Exposing Human Motion Generation to CLIP Space"

MotionCLIP Official Pytorch implementation of the paper "MotionCLIP: Exposing Human Motion Generation to CLIP Space". Please visit our webpage for mor

Guy Tevet 173 Dec 26, 2022
[CVPR 2021] VirTex: Learning Visual Representations from Textual Annotations

VirTex: Learning Visual Representations from Textual Annotations Karan Desai and Justin Johnson University of Michigan CVPR 2021 arxiv.org/abs/2006.06

Karan Desai 533 Dec 24, 2022
Python3 Implementation of (Subspace Constrained) Mean Shift Algorithm in Euclidean and Directional Product Spaces

(Subspace Constrained) Mean Shift Algorithms in Euclidean and/or Directional Product Spaces This repository contains Python3 code for the mean shift a

Yikun Zhang 0 Oct 19, 2021
A Neural Net Training Interface on TensorFlow, with focus on speed + flexibility

Tensorpack is a neural network training interface based on TensorFlow. Features: It's Yet Another TF high-level API, with speed, and flexibility built

Tensorpack 6.2k Jan 09, 2023
Code for the paper: On Pathologies in KL-Regularized Reinforcement Learning from Expert Demonstrations

Non-Parametric Prior Actor-Critic (N-PPAC) This repository contains the code for On Pathologies in KL-Regularized Reinforcement Learning from Expert D

Cong Lu 5 May 13, 2022
Unoffical implementation about Image Super-Resolution via Iterative Refinement by Pytorch

Image Super-Resolution via Iterative Refinement Paper | Project Brief This is a unoffical implementation about Image Super-Resolution via Iterative Re

LiangWei Jiang 2.5k Jan 02, 2023
Personal thermal comfort models using digital twins: Preference prediction with BIM-extracted spatial-temporal proximity data from Build2Vec

Personal thermal comfort models using digital twins: Preference prediction with BIM-extracted spatial-temporal proximity data from Build2Vec This repo

Building and Urban Data Science (BUDS) Group 5 Dec 02, 2022
A framework for joint super-resolution and image synthesis, without requiring real training data

SynthSR This repository contains code to train a Convolutional Neural Network (CNN) for Super-resolution (SR), or joint SR and data synthesis. The met

83 Jan 01, 2023
A whale detector design for the Kaggle whale-detector challenge!

CNN (InceptionV1) + STFT based Whale Detection Algorithm So, this repository is my PyTorch solution for the Kaggle whale-detection challenge. The obje

Tarin Ziyaee 92 Sep 28, 2021
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding

Vision Longformer This project provides the source code for the vision longformer paper. Multi-Scale Vision Longformer: A New Vision Transformer for H

Microsoft 209 Dec 30, 2022
A PyTorch Implementation of "Watch Your Step: Learning Node Embeddings via Graph Attention" (NeurIPS 2018).

Attention Walk ⠀⠀ A PyTorch Implementation of Watch Your Step: Learning Node Embeddings via Graph Attention (NIPS 2018). Abstract Graph embedding meth

Benedek Rozemberczki 303 Dec 09, 2022
StyleGAN-Human: A Data-Centric Odyssey of Human Generation

StyleGAN-Human: A Data-Centric Odyssey of Human Generation Abstract: Unconditional human image generation is an important task in vision and graphics,

stylegan-human 762 Jan 08, 2023
[ICCV 2021] Self-supervised Monocular Depth Estimation for All Day Images using Domain Separation

ADDS-DepthNet This is the official implementation of the paper Self-supervised Monocular Depth Estimation for All Day Images using Domain Separation I

LIU_LINA 52 Nov 24, 2022
Management Dashboard for Torchserve

Torchserve Dashboard Torchserve Dashboard using Streamlit Related blog post Usage Additional Requirement: torchserve (recommended:v0.5.2) Simply run:

Ceyda Cinarel 103 Dec 10, 2022
4th place solution to datafactory challenge by Intermarché.

Solution to Datafactory challenge by Intermarché. 4th place solution to datafactory challenge by Intermarché. The objective of the challenge is to pre

Raphael Sourty 11 Mar 19, 2022
For storing the complete exploration of Visual Question Answering for our B.Tech Project

Multi-Image vqa @authors: Akhilesh, Janhavi, Harsh Paper summary, Ideas tried and their corresponding results: on wiki Other discussions: on discussio

Harsh Raj 3 Jun 16, 2022
The Curious Layperson: Fine-Grained Image Recognition without Expert Labels (BMVC 2021)

The Curious Layperson: Fine-Grained Image Recognition without Expert Labels Subhabrata Choudhury, Iro Laina, Christian Rupprecht, Andrea Vedaldi Code

Subhabrata Choudhury 18 Dec 27, 2022