Code for "Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks", CVPR 2021

Overview

Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks

Example 1 Example 2 Example 3

This repository contains the code that accompanies our CVPR 2021 paper Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks

You can find detailed usage instructions for training your own models and using our pretrained models below.

If you found this work influential or helpful for your research, please consider citing

@Inproceedings{Paschalidou2021CVPR,
     title = {Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks},
     author = {Paschalidou, Despoina and Katharopoulos, Angelos and Geiger, Andreas and Fidler, Sanja},
     booktitle = {Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)},
     year = {2021}
}

Installation & Dependencies

Our codebase has the following dependencies:

For the visualizations, we use simple-3dviz, which is our easy-to-use library for visualizing 3D data using Python and ModernGL and matplotlib for the colormaps. Note that simple-3dviz provides a lightweight and easy-to-use scene viewer using wxpython. If you wish you use our scripts for visualizing the reconstructed primitives, you will need to also install wxpython.

The simplest way to make sure that you have all dependencies in place is to use conda. You can create a conda environment called neural_parts using

conda env create -f environment.yaml
conda activate neural_parts

Next compile the extenstion modules. You can do this via

python setup.py build_ext --inplace
pip install -e .

Demo

Example Output Example Output

You can now test our code on various inputs. To this end, simply download some input samples together with our pretrained models on D-FAUAST humans, ShapeNet chairs and ShapeNet planes from here. Now extract the nerual_parts_demo.zip that you just downloaded in the demo folder. To run our demo on the D-FAUST humans simply run

python demo.py ../config/dfaust_6.yaml --we ../demo/model_dfaust_6 --model_tag 50027_jumping_jacks:00135 --camera_target='-0.030173788,-0.10342446,-0.0021887198' --camera_position='0.076685235,-0.14528269,1.2060229' --up='0,1,0' --with_rotating_camera

This script should create a folder demo/output, where the per-primitive meshes are stored as .obj files. Similarly, you can now also run the demo for the input airplane

python demo.py ../config/shapenet_5.yaml --we ../demo/model_planes_5 --model_tag 02691156:7b134f6573e7270fb0a79e28606cb167 --camera_target='-0.030173788,-0.10342446,-0.0021887198' --camera_position='0.076685235,-0.14528269,1.2060229' --up='0,1,0' --with_rotating_camera

Usage

As soon as you have installed all dependencies and have obtained the preprocessed data, you can now start training new models from scratch, evaluate our pre-trained models and visualize the recovered primitives using one of our pre-trained models.

Reconstruction

To generate meshes using a trained model, we provide the forward_pass.py and the visualize_predictions.py scripts. Their difference is that the first performs the forward pass and generates a per-primitive mesh that is saved as an .obj file. Similarly, the visualize_predictions.py script performs the forward pass and visualizes the predicted primitives using simple-3dviz. The forward_pass.py script is ideal for reconstructing inputs on a heeadless server and you can run it by executing

python forward_pass.py path_to_config_yaml path_to_output_dir --weight_file path_to_weight_file --model_tag MODEL_TAG

where the argument --weight_file specifies the path to a trained model and the argument --model_tag defines the model_tag of the input to be reconstructed.

To run the visualize_predictions.py script you need to run

python visualize_predictions.py path_to_config_yaml path_to_output_dir --weight_file path_to_weight_file --model_tag MODEL_TAG

Using this script, you can easily render the prediction into .png images or a .gif, as well as perform various animations by rotating the camera. Furthermore, you can also specify the camera position, the up vector and the camera target as well as visualize the target mesh together with the predicted primitives simply by adding the --mesh argument.

Evaluation

For evaluation of the models we provide the script evaluate.py. You can run it using:

python evaluate.py path_to_config_yaml path_to_output_dir

The script reconstructs the input and evaluates the generated meshes using a standardized protocol. For each input, the script generates a .npz file that contains the various metrics for that particular input. Note that this script can also be executed multiple times in order to speed up the evaluation process. For example, if you wish to run the evaluation on 6 nodes, you can simply run

for i in {1..6}; do python evaluate.py path_to_config_yaml path_to_output_dir & done
[1] 9489
[2] 9490
[3] 9491
[4] 9492
[5] 9493
[6] 9494

wait
Running code on cpu
Running code on cpu
Running code on cpu
Running code on cpu
Running code on cpu
Running code on cpu

Again the script generates a per-input file in the output directory with the computed metrics.

Training

Finally, to train a new network from scratch, we provide the train_network.py script. To execute this script, you need to specify the path to the configuration file you wish to use and the path to the output directory, where the trained models and the training statistics will be saved. Namely, to train a new model from scratch, you simply need to run

python train_network.py path_to_config_yaml path_to_output_dir

Note tha it is also possible to start from a previously trained model by specifying the --weight_file argument, which should contain the path to a previously trained model. Furthermore, by using the arguments --model_tag and --category_tag, you can also train your network on a particular model (e.g. a specific plane, car, human etc.) or a specific object category (e.g. planes, chairs etc.)

Note that, if you want to use the RAdam optimizer during training, you will have to also install to download and install the corresponding code from this repository.

License

Our code is released under the MIT license which practically allows anyone to do anything with it. MIT license found in the LICENSE file.

Relevant Research

Below we list some papers that are relevant to our work.

Ours:

  • Learning Unsupervised Hierarchical Part Decomposition of 3D Objects from a Single RGB Image pdf,project-page
  • Superquadrics Revisited: Learning 3D Shape Parsing beyond Cuboids pdf,project-page

By Others:

  • Learning Shape Abstractions by Assembling Volumetric Primitives pdf
  • 3D-PRNN: Generating Shape Primitives with Recurrent Neural Networks pdf
  • Im2Struct: Recovering 3D Shape Structure From a Single RGB Image pdf
  • Learning shape templates with structured implicit functions pdf
  • CvxNet: Learnable Convex Decomposition pdf
Image-to-Image Translation with Conditional Adversarial Networks (Pix2pix) implementation in keras

pix2pix-keras Pix2pix implementation in keras. Original paper: Image-to-Image Translation with Conditional Adversarial Networks (pix2pix) Paper Author

William Falcon 141 Dec 30, 2022
Use unsupervised and supervised learning to predict stocks

AIAlpha: Multilayer neural network architecture for stock return prediction This project is meant to be an advanced implementation of stacked neural n

Vivek Palaniappan 1.5k Dec 26, 2022
duralava is a neural network which can simulate a lava lamp in an infinite loop.

duralava duralava is a neural network which can simulate a lava lamp in an infinite loop. Example This is not a real lava lamp but a "fake" one genera

Maximilian Bachl 87 Dec 20, 2022
Deep Sea Treasure Environment for Multi-Objective Optimization Research

DeepSeaTreasure Environment Installation In order to get started with this environment, you can install it using the following command: python3 -m pip

imec IDLab 6 Nov 14, 2022
This is an implementation for the CVPR2020 paper "Learning Invariant Representation for Unsupervised Image Restoration"

Learning Invariant Representation for Unsupervised Image Restoration (CVPR 2020) Introduction This is an implementation for the paper "Learning Invari

GarField 88 Nov 07, 2022
Evaluation suite for large-scale language models.

This repo contains code for running the evaluations and reproducing the results from the Jurassic-1 Technical Paper (see blog post), with current support for running the tasks through both the AI21 S

71 Dec 17, 2022
LTR_CrossEncoder: Legal Text Retrieval Zalo AI Challenge 2021

LTR_CrossEncoder: Legal Text Retrieval Zalo AI Challenge 2021 We propose a cross encoder model (LTR_CrossEncoder) for information retrieval, re-retrie

Xuan Hieu Duong 7 Jan 12, 2022
Trash Sorter Extraordinaire is a software which efficiently detects the different types of waste in a pile of random trash through feeding it pictures or videos.

Trash-Sorter-Extraordinaire Trash Sorter Extraordinaire is a software which efficiently detects the different types of waste in a pile of random trash

Rameen Mahmood 1 Nov 07, 2021
The official repo for OC-SORT: Observation-Centric SORT on video Multi-Object Tracking. OC-SORT is simple, online and robust to occlusion/non-linear motion.

OC-SORT Observation-Centric SORT (OC-SORT) is a pure motion-model-based multi-object tracker. It aims to improve tracking robustness in crowded scenes

Jinkun Cao 325 Jan 05, 2023
Action Segmentation Evaluation

Reference Action Segmentation Evaluation Code This repository contains the reference code for action segmentation evaluation. If you have a bug-fix/im

5 May 22, 2022
A stable algorithm for GAN training

DRAGAN (Deep Regret Analytic Generative Adversarial Networks) Link to our paper - https://arxiv.org/abs/1705.07215 Pytorch implementation (thanks!) -

195 Oct 10, 2022
Official code of "R2RNet: Low-light Image Enhancement via Real-low to Real-normal Network."

R2RNet Official code of "R2RNet: Low-light Image Enhancement via Real-low to Real-normal Network." Jiang Hai, Zhu Xuan, Ren Yang, Yutong Hao, Fengzhu

77 Dec 24, 2022
PyExplainer: A Local Rule-Based Model-Agnostic Technique (Explainable AI)

PyExplainer PyExplainer is a local rule-based model-agnostic technique for generating explanations (i.e., why a commit is predicted as defective) of J

AI Wizards for Software Management (AWSM) Research Group 14 Nov 13, 2022
ChatBot-Pytorch - A GPT-2 ChatBot implemented using Pytorch and Huggingface-transformers

ChatBot-Pytorch A GPT-2 ChatBot implemented using Pytorch and Huggingface-transf

ParZival 42 Dec 09, 2022
Addon and nodes for working with structural biology and molecular data in Blender.

Molecular Nodes 🧬 🔬 💻 Buy Me a Coffee to Keep Development Going! Join a Community of Blender SciVis People! What is Molecular Nodes? Molecular Node

Brady Johnston 456 Jan 08, 2023
Strongly local p-norm-cut algorithms for semi-supervised learning and local graph clustering

Strongly local p-norm-cut algorithms for semi-supervised learning and local graph clustering

Meng Liu 2 Jul 19, 2022
Code for the TASLP paper "PSLA: Improving Audio Tagging With Pretraining, Sampling, Labeling, and Aggregation".

PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregation Introduction Getting Started FSD50K Recipe AudioSet Recipe Label E

Yuan Gong 84 Dec 27, 2022
Awesome Weak-Shot Learning

Awesome Weak-Shot Learning In weak-shot learning, all categories are split into non-overlapped base categories and novel categories, in which base cat

BCMI 162 Dec 30, 2022
(ICCV 2021) PyTorch implementation of Paper "Progressive Correspondence Pruning by Consensus Learning"

CLNet (ICCV 2021) PyTorch implementation of Paper "Progressive Correspondence Pruning by Consensus Learning" [project page] [paper] Citing CLNet If yo

Chen Zhao 22 Aug 26, 2022
The official implementation of ICCV paper "Box-Aware Feature Enhancement for Single Object Tracking on Point Clouds".

Box-Aware Tracker (BAT) Pytorch-Lightning implementation of the Box-Aware Tracker. Box-Aware Feature Enhancement for Single Object Tracking on Point C

Kangel Zenn 5 Mar 26, 2022