Open source repository for the code accompanying the paper 'PatchNets: Patch-Based Generalizable Deep Implicit 3D Shape Representations'.

Overview

PatchNets

This is the official repository for the project "PatchNets: Patch-Based Generalizable Deep Implicit 3D Shape Representations". For details, we refer to our project page, which also includes supplemental videos.

This code requires a functioning installation of DeepSDF, which can then be modified using the provided files.

(Optional) Making ShapeNet V1 Watertight

If you want to use ShapeNet, please follow these steps:

  1. Download Occupancy Networks
  2. On Linux, follow the installation steps from there:
conda env create -f environment.yaml
conda activate mesh_funcspace
python setup.py build_ext --inplace
  1. Install the four external dependencies from external/mesh-fusion:
    • for libfusioncpu and libfusiongpu, run cmake and then setup.py
    • for libmcubes and librender, run setup.py
  2. Replace the original OccNet files with the included slightly modified versions. This mostly switches to using .obj instead of .off
  3. Prepare the original Shapenet meshes by copying all objs as follows: from 02858304/1b2e790b7c57fc5d2a08194fd3f4120d/model.obj to 02858304/1b2e790b7c57fc5d2a08194fd3f4120d.obj
  4. Use generate_watertight_meshes_and_sample_points() from useful_scripts.py. Needs to be run twice, see comment at generate_command.
  5. On a Linux machine with display, activate mesh_funcspace
  6. Run the generated command.sh. Note: this preprocessing crashes frequently because some meshes cause issues. They need to be deleted.

Preprocessing

During preprocessing, we generate SDF samples from obj files.

The C++ files in src/ are modified versions of the corresponding DeepSDF files. Please follow the instruction on the DeepSDF github repo to compile these. Then run preprocess_data.py. There is a special flag in preprocess_data.py for easier handling of ShapeNet. There is also an example command for preprocessing ShapeNet in the code comments. If you want to use depth completion, add the --randomdepth and --depth flags to the call to preprocess_data.py.

Training

The files in code/ largely follow DeepSDF and replace the corresponding files in your DeepSDF installation. Note that some legacy functions from these files might not be compatible with PatchNets.

  • Some settings files are available in code/specs/. The training/test splits can be found in code/examples/splits/. The DataSource and, if used, the patch_network_pretrained_path and pretrained_depth_encoder_weights need to be adapted.
  • Set a folder that collects all experiments in code/localization/SystemSpecific.py.
  • The code uses code/specs.json as the settings file. Replace this file with the desired settings file.
  • The code creates a results folder, which also includes a backup. This is necessary to later run the evaluation script.
  • Throughout the code, metadata refers to patch extrinsics.
  • mixture_latent_mode can be set to all_explicit for normal PatchNets mode or to all_implicit for use with object latents.
    • Some weights automatically change in deep_sdf_decoder.py depending on whether all_explicit or all_implicit is used.
  • For all_implicit/object latents, please set sdf_filename under use_precomputed_bias_init in deep_sdf_decoder.py to an .npz file that was obtained via Preprocessing and for which initialize_mixture_latent_vector() from train_deep_sdf.py has been run (e.g. by including it in the training set and training a normal PatchNet). MixtureCodeLength is the object latent size and PatchCodeLength is the size of each of the regressed patch codes.
  • For all_explicit/normal PatchNets, MixtureCodeLength needs to be compatible with PatchCodeLength. Set MixtureCodeLength = (PatchCodeLength + 7) x num_patches. The 7 comes from position (3) + rotation (3) + scale (1). Always use 7, regardless of whether scale and/or rotation are used. Consider keeping the patch extrinsics fixed at their initial values instead of optimizing them with the extrinsics loss, see the second stage of StagedTraining.
  • When using staged training, NumEpochs and the total Lengths of each Staged schedule should be equal. Also note that both Staged schedules should have the exact same Lengths list.

Evaluation

  1. Fit PatchNets to test data: Use train_deep_sdf.py to run the trained network on the test data. Getting the patch parameters for a test set is almost the same workflow as training a network, except that the network weights are initialized and then kept fixed and a few other settings are changed. Please see included test specs.json for examples. In all cases, set test_time = True, train_patch_network = False, train_object_to_patch = False. Set patch_network_pretrained_path in the test specs.json to the results folder of the trained network. Make sure that ScenesPerBatch is a multiple of the test set size. Adjust the learning rate schedules according to the test specs.json examples included.
  2. Get quantitative evaluation: Use evaluate_patch_network_metrics() from useful_scripts.py with the test results folder. Needs to be run twice, see comment at generate_meshes. Running this script requires an installation of Occupancy Networks, see comments in evaluate_patch_network_metrics(). It also requires the obj files of the dataset that were used for Preprocessing.

Applications, Experiments, and Mesh Extraction

useful_scripts.py contains code for the object latent applications from Sec. 4.3: latent interpolation, the generative model and depth completion. The depth completion code contains a mode for quantitative evaluation. useful_scripts.py also contains code to extract meshes.

code/deep_sdf/data.py contains the code snippet used for the synthetic noise experiments in Sec. 7 of the supplementary material.

Additional Functionality

The code contains additional functionalities that are not part of the publication. They might work but have not been thoroughly tested and can be removed.

  • wrappers to allow for easy interaction with a trained network (do not remove, required to run evaluation)
    • _setup_torch_network() in useful_scripts.py
  • a patch encoder
    • Instead of autodecoding a patch latent code, it is regressed from SDF point samples that lie inside the patch.
    • Encoder in specs.json. Check that this works as intended, later changes to the code might have broken something.
  • a depth encoder
    • A depth encoder maps from one depth map to all patch parameters.
    • use_depth_encoder in specs.json. Check that this works as intended, later changes to the code might have broken something.
  • a tiny PatchNet version
    • The latent code is reshaped and used as network weights, i.e. there are no shared weights between different patches.
    • dims in specs.json should be set to something small like [ 8, 8, 8, 8, 8, 8, 8 ]
    • use_tiny_patchnet in specs.json
    • Requires to set PatchLatentCode correctly, the desired value is printed by _initialize_tiny_patchnet() in deep_sdf_decoder.py.
  • a hierarchical representation
    • Represents/encodes a shape using large patches for simple regions and smaller patches for complex regions of the geometry.
    • hierarchical_representation() in useful_scripts.py. Never tested. Later changes to the network code might also have broken something.
  • simplified curriculum weighting from Curriculum DeepSDF
    • use_curriculum_weighting in specs.json. Additional parameters are in train_deep_sdf.py. This is our own implementation, not based on their repo, so mistakes are ours.
  • positional encoding from NeRF
    • positional_encoding in specs.json. Additional parameters are in train_deep_sdf.py. This is our own implementation, not based on their repo, so mistakes are ours.
  • a Neural ODE deformation model for patches
    • Instead of a simple MLP regressing the SDF value, a velocity field first deforms the patch region and then the z-value of the final xyz position is returned as the SDF value. Thus the field flattens the surface to lie in the z=0 plane. Very slow due to Neural ODE. Might be useful to get UV maps/a direct surface parametrization.
    • use_ode and time_dependent_ode in specs.json. Additional parameters are in train_deep_sdf.py.
  • a mixed representation that has explicit patch latent codes and only regresses patch extrinsics from an object latent code
    • Set mixture_latent_mode in specs.json to patch_explicit_meta_implicit. posrot_latent_size is the size of the object latent code in this case. mixture_to_patch_parameters is the network that regresses the patch extrinsics. Check that this works as intended, later changes to the code might have broken something.

Citation

This code builds on DeepSDF. Please consider citing DeepSDF and PatchNets if you use this code.

@article{Tretschk2020PatchNets, 
    author = {Tretschk, Edgar and Tewari, Ayush and Golyanik, Vladislav and Zollh\"{o}fer, Michael and Stoll, Carsten and Theobalt, Christian}, 
    title = "{PatchNets: Patch-Based Generalizable Deep Implicit 3D Shape Representations}", 
    journal = {European Conference on Computer Vision (ECCV)}, 
    year = "2020" 
} 
@InProceedings{Park_2019_CVPR,
    author = {Park, Jeong Joon and Florence, Peter and Straub, Julian and Newcombe, Richard and Lovegrove, Steven},
    title = {DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation},
    booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    month = {June},
    year = {2019}
}

License

Please note that this code is released under an MIT licence, see LICENCE. We have included and modified third-party components, which have their own licenses. We thank all of the respective authors for releasing their code, especially the team behind DeepSDF!

Gas detection for Raspberry Pi using ADS1x15 and MQ-2 sensors

Gas detection Gas detection for Raspberry Pi using ADS1x15 and MQ-2 sensors. Description The MQ-2 sensor can detect multiple gases (CO, H2, CH4, LPG,

Filip Š 15 Sep 30, 2022
BARTScore: Evaluating Generated Text as Text Generation

This is the Repo for the paper: BARTScore: Evaluating Generated Text as Text Generation Updates 2021.06.28 Release online evaluation Demo 2021.06.25 R

NeuLab 196 Dec 17, 2022
Official implementation of Deep Convolutional Dictionary Learning for Image Denoising.

DCDicL for Image Denoising Hongyi Zheng*, Hongwei Yong*, Lei Zhang, "Deep Convolutional Dictionary Learning for Image Denoising," in CVPR 2021. (* Equ

Z80 91 Dec 21, 2022
Framework web SnakeServer.

SnakeServer - Framework Web 🐍 Documentação oficial do framework SnakeServer. Conteúdo Sobre Como contribuir Enviar relatórios de segurança Pull reque

Jaedson Silva 0 Jul 21, 2022
Differentiable scientific computing library

xitorch: differentiable scientific computing library xitorch is a PyTorch-based library of differentiable functions and functionals that can be widely

98 Dec 26, 2022
Official implementation of the RAVE model: a Realtime Audio Variational autoEncoder

Official implementation of the RAVE model: a Realtime Audio Variational autoEncoder

Antoine Caillon 589 Jan 02, 2023
Artifacts for paper "MMO: Meta Multi-Objectivization for Software Configuration Tuning"

MMO: Meta Multi-Objectivization for Software Configuration Tuning This repository contains the data and code for the following paper that is currently

0 Nov 17, 2021
GraPE is a Rust/Python library for high-performance Graph Processing and Embedding.

GraPE GraPE (Graph Processing and Embedding) is a fast graph processing and embedding library, designed to scale with big graphs and to run on both of

AnacletoLab 194 Dec 29, 2022
A Survey on Deep Learning Technique for Video Segmentation

A Survey on Deep Learning Technique for Video Segmentation A Survey on Deep Learning Technique for Video Segmentation Wenguan Wang, Tianfei Zhou, Fati

Tianfei Zhou 112 Dec 12, 2022
Efficient neural networks for analog audio effect modeling

micro-TCN Efficient neural networks for audio effect modeling

Christian Steinmetz 94 Dec 29, 2022
基于tensorflow 2.x的图片识别工具集

Classification.tf2 基于tensorflow 2.x的图片识别工具集 功能 粗粒度场景图片分类 细粒度场景图片分类 其他场景图片分类 模型部署 tensorflow serving本地推理和docker部署 tensorRT onnx ... 数据集 https://hyper.a

Wei Qi 1 Nov 03, 2021
This repository contains tutorials for the py4DSTEM Python package

py4DSTEM Tutorials This repository contains tutorials for the py4DSTEM Python package. For more information about py4DSTEM, including installation ins

11 Dec 23, 2022
Code for ICDM2020 full paper: "Sub-graph Contrast for Scalable Self-Supervised Graph Representation Learning"

Subg-Con Sub-graph Contrast for Scalable Self-Supervised Graph Representation Learning (Jiao et al., ICDM 2020): https://arxiv.org/abs/2009.10273 Over

34 Jul 06, 2022
Collision risk estimation using stochastic motion models

collision_risk_estimation Collision risk estimation using stochastic motion models. This is a new approach, based on stochastic models, to predict the

Unmesh 7 Jun 26, 2022
PCACE: A Statistical Approach to Ranking Neurons for CNN Interpretability

PCACE: A Statistical Approach to Ranking Neurons for CNN Interpretability PCACE is a new algorithm for ranking neurons in a CNN architecture in order

4 Jan 04, 2022
Pytorch tutorials for Neural Style transfert

PyTorch Tutorials This tutorial is no longer maintained. Please use the official version: https://pytorch.org/tutorials/advanced/neural_style_tutorial

Alexis David Jacq 135 Jun 26, 2022
R-Drop: Regularized Dropout for Neural Networks

R-Drop: Regularized Dropout for Neural Networks R-drop is a simple yet very effective regularization method built upon dropout, by minimizing the bidi

756 Dec 27, 2022
Video Matting via Consistency-Regularized Graph Neural Networks

Video Matting via Consistency-Regularized Graph Neural Networks Project Page | Real Data | Paper Installation Our code has been tested on Python 3.7,

41 Dec 26, 2022
PoolFormer: MetaFormer is Actually What You Need for Vision

PoolFormer: MetaFormer is Actually What You Need for Vision (arXiv) This is a PyTorch implementation of PoolFormer proposed by our paper "MetaFormer i

Sea AI Lab 1k Dec 30, 2022
Fake-user-agent-traffic-geneator - Python CLI Tool to generate fake traffic against URLs with configurable user-agents

Fake traffic generator for Gartner Demo Generate fake traffic to URLs with custo

New Relic Experimental 3 Oct 31, 2022