Pytorch implementation for the Dynamic Concept Learner (DCL). More details can be found at the project page.

Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning
Zhenfang Chen, Jiayuan Mao, Jiajun Wu, Kwan-Yee K. Wong, Joshua B. Tenenbaum, and Chuang Gan

• Python 3
• PyTorch 1.0 or higher, with NVIDIA CUDA Support
• Other required python packages specified by requirements.txt. See the Installation.

Install Jacinle: Clone the package, and add the bin path to your global PATH environment variable:

git clone https://github.com/vacancy/Jacinle --recursive
export PATH=<path_to_jacinle>/bin:$PATH

Clone this repository:

git clone https://github.com/zfchenUnique/DCL-Release.git --recursive

Create a conda environment for NS-CL, and install the requirements. This includes the required python packages from both Jacinle NS-CL. Most of the required packages have been included in the built-in anaconda package:

• Download videos, video annotation, questions and answers, and object proposals accordingly from the official website
• Transform videos into ".png" frames with ffmpeg.
• Organize the data as shown below.

  clevrer
  ├── annotation_00000-01000
  │   ├── annotation_00000.json
  │   ├── annotation_00001.json
  │   └── ...
  ├── ...
  ├── image_00000-01000
  │   │   ├── 1.png
  │   │   ├── 2.png
  │   │   └── ...
  │   └── ...
  ├── ...
  ├── questions
  │   ├── train.json
  │   ├── validation.json
  │   └── test.json
  ├── proposals
  │   ├── proposal_00000.json
  │   ├── proposal_00001.json
  │   └── ...
  

• Download the extracted object trajectories from google drive.
• Git clone the dynamic model, download image proposals and the pretrained propNet models and make dynamic prediction by

    git clone https://github.com/zfchenUnique/clevrer_dynamic_propnet.git
    cd clevrer_dynamic_propnet
    sh ./scripts/eval_fast_release_v2.sh 0

• Download the pretrained DCL model and parsed programs.

   sh scripts/script_test_prp_clevrer_qa.sh 0

• Get the accuracy on evalAI.

• Step 1: download the proposals from the region proposal network and extract object trajectories for train and val set by

   sh scripts/script_gen_tubes.sh

• Step 2: train a concept learner with descriptive and explanatory questions for static concepts (i.e. color, shape and material)

   sh scripts/script_train_dcl_stage1.sh 0

• Step 3: extract static attributes & refine object trajectories extract static attributes

   sh scripts/script_extract_attribute.sh

refine object trajectories

   sh scripts/script_gen_tubes_refine.sh

• Step 4: extract predictive and counterfactual scenes by

    cd clevrer_dynamic_propnet
    sh ./scripts/train_tube_box_only.sh # train
    sh ./scripts/train_tube.sh # train
    sh ./scripts/eval_fast_release_v2.sh 0 # val

• Step 5: train DCL with all questions and the refined trajectories

   sh scripts/script_train_dcl_stage2.sh 0

• Step 1: download expression annotation and parsed programs from google drive
• Step 2: evaluate the performance on CLEVRER-Grounding

    sh ./scripts/script_grounding.sh  0
    jac-crun 0 scripts/script_evaluate_grounding.py

• Step 1: download expression annotation and parsed programs from google drive
• Step 2: evaluate the performance on CLEVRER-Retrieval

    sh ./scripts/script_retrieval.sh  0
    jac-crun 0 scripts/script_evaluate_retrieval.py

• Step 1: download question annotation from google drive and videos from dropbox under the UETorch repo.
• Step 2: train on Tower block QA

    sh ./scripts/script_train_blocks.sh 0

• Step 3: download the pretrain model from google drive and evaluate on Tower block QA

    sh ./scripts/script_eval_blocks.sh 0

• Qualitative Results
• CLEVRER-Grounding training set Annotation
• CLEVRER-Retrieval training set Annotation
• Project Page

If you find this repo useful in your research, please consider citing:

@inproceedings{zfchen2021iclr,
    title={Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning},
    author={Chen, Zhenfang and Mao, Jiayuan and Wu, Jiajun and Wong, Kwan-Yee K and Tenenbaum, Joshua B. and Gan, Chuang},
    booktitle={International Conference on Learning Representations},
    year={2021}
    }