POCO: Point Convolution for Surface Reconstruction

Related tags

Deep LearningPOCO
Overview

POCO: Point Convolution for Surface Reconstruction

by: Alexandre Boulch and Renaud Marlet

Abstract

Implicit neural networks have been successfully used for surface reconstruction from point clouds. However, many of them face scalability issues as they encode the isosurface function of a whole object or scene into a single latent vector. To overcome this limitation, a few approaches infer latent vectors on a coarse regular 3D grid or on 3D patches, and interpolate them to answer occupancy queries. In doing so, they loose the direct connection with the input points sampled on the surface of objects, and they attach information uniformly in space rather than where it matters the most, i.e., near the surface. Besides, relying on fixed patch sizes may require discretization tuning. To address these issues, we propose to use point cloud convolutions and compute latent vectors at each input point. We then perform a learning-based interpolation on nearest neighbors using inferred weights. Experiments on both object and scene datasets show that our approach significantly outperforms other methods on most classical metrics, producing finer details and better reconstructing thinner volumes.

Citation

TODO

Dependencies

Our code rely on Pytorch Geometric, which should be installed. Then, run:

python setup.py build_ext --inplace

Data

ShapeNet (Occupancy Network pre-processing)

We use the ShapeNet dataset as pre-processed by Occupancy Networks. Please refer to original repository for downloading the data.

It should be placed in the folder data/ShapeNet.

SyntheticRooms (Occupancy Network pre-processing)

We use the SyntheticRooms dataset as created by Occupancy Networks. Please refer to original repository for downloading the data.

It should be placed in the folder data/synthetic_room_dataset.

ABC / RealWorld / Famous and Thingi10k

They should be placed in the folders: data/3d_shapes_abc, data/3d_shapes_abc_training, data/3d_shapes_famous, data/3d_shapes_real_world and data/3d_shapes_thingi10k.

SceneNet

Download the SceneNet dataset.

Watertight models

We use the code from https://github.com/hjwdzh/Manifold for watertight model creation. Please download and compile it.

To create the watertight models, create a destination directory data/SceneNet, copy-paste the python script datasets/scenenet_watertight.py into the directory and run it with python scenenet_watertight.py. You will need to modify the raw_data_dir and the manifold_code_dir to match your data and code locations.

Creation of the point clouds

You can use the script datasets/scenenet_sample.py. You will also need to modify the paths.

Precomputed meshes and point clouds

For easy use of the code, we provide precomputed meshes, and point clouds SceneNet20, SceneNet100, SceneNet500 and SceneNet1000.

Location

The watertight models should be in data/SceneNet.

The points are located in data/SceneNet20, data/SceneNet100, data/SceneNet1000 and data/SceneNet500.

Training

# train on ShapeNet with 3k points, noise and without normals 
python train.py --config configs/config_shapenet.yaml 

# train on ShapeNet with 10k points, no noise and normals
python train.py --config configs/config_shapenet.yaml --normals True --random_noise 0 --experiment_name Normals

# train on ABC with 3k points
python train.py --config configs/config_abc.yaml --experiment_name 3k

# train on ABC with 10k points
python train.py --config configs/config_abc.yaml --experiment_name 10k --manifold_points 10000 --training_batch_size 8

Generation

ShapeNet

python generate.py --config results/ShapeNet_None_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --gen_resolution_global 128

SyntheticRooms

python generate.py --config results/SyntheticRooms_None_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --gen_resolution_global 256 --num_mesh 1

ABC, RealWorld, Famous and Thingi10k

# Model trained with 3k points
python generate.py --config results/ABC_3k_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --dataset_name DATASET_NAME --dataset_root data/DATASET_DIR --gen_resolution_global 256

python generate.py --config results/ABC_3k_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --dataset_name DATASET_NAME --dataset_root data/DATASET_DIR --gen_resolution_global 256 --manifold_points -1 --gen_subsample_manifold 3000 --gen_subsample_manifold_iter 10 --gen_descriptor gen_sub3k_iter10

# Model trained with 10k points
python generate.py --config results/ABC_10k_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --dataset_name DATASET_NAME --dataset_root data/DATASET_DIR --gen_resolution_global 256

python generate.py --config results/ABC_10k_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --dataset_name DATASET_NAME --dataset_root data/DATASET_DIR --gen_resolution_global 256 --manifold_points -1 --noise 0.0 --gen_subsample_manifold 10000 --gen_subsample_manifold_iter 10 --gen_descriptor gen_sub3k_iter10

For faster generation, one would want to use a lower marching cubes parameter --gen_resolution_global, e.g. 128.

DATASET_NAME should be replaced by ABCTest, ABCTestNoiseFree, ABCTestExtraNoise, RealWorld, FamousTest, FamousTestNoiseFree, FamousTestExtraNoisy, FamousTestSparse, FamousTestDense, Thingi10kTest, Thingi10kTestNoiseFree, Thingi10kTestExtraNoisy, Thingi10kTestSparse, Thingi10kTestDense

DATASET_DIR should be replaced by 3d_shapes_abc, 3d_shapes_famous, 3d_shapes_real_world, 3d_shapes_thingi10k

SceneNet

python generate.py --config results/ShapeNet_Normals_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --gen_autoscale True --gen_autoscale_target 0.01 --gen_resolution_metric 0.01 --dataset_name DATASET_NAME --dataset_root data/DATASET_NAME --manifold_points -1

DATASET_NAME should be replaced by SceneNet20, SceneNet100, SceneNet500, SceneNet1000.

The --gen_autoscale_target parameter is the average distance between a point and its nearest neighbor in the training set, 0.01 for ShapeNet train set with 3000 points.

Evaluation

ShapeNet

python eval_meshes.py --gendir results/ShapeNet_None_FKAConv_InterpAttentionKHeadsNet_None/gen_ShapeNet_test_3000/ --meshdir meshes --dataset ShapeNet --split test --gtdir data/ShapeNet

SyntheticRooms

python eval_meshes.py --gendir results/SyntheticRooms_None_FKAConv_InterpAttentionKHeadsNet_None/gen_SyntheticRooms_test_10000/ --meshdir meshes --dataset SyntheticRooms --split test --gtdir data/synthetic_room_dataset

SceneNet

python eval_scenenet.py --gendir results/ShapeNet_Normals_FKAConv_InterpAttentionKHeadsNet_None/gen_SceneNet20_test_allPts/ --meshdir meshes --gtdir data/SceneNet

ABC, Famous and Thingi10k

For evaluation on the ABCTest please run:

python eval_point2surf_datasets.py --gendir results/ABC_3k_FKAConv_InterpAttentionKHeadsNet_None/gen_ABCTest_test_3000 --meshdir meshes/04_pts/ --gtdir data/3d_shapes_abc/abc/

You can adapt the paths to evaluate on the different datasets and noise variants.

Pretrained models

We provide pre-trained models for FKAConv backbone.

ShapeNet 3k, noise, no normals

ShapeNet 3k, no noise, normals

SyntheticRooms 10k

ABC 3k

ABC 10k

Configuration

The code was used with Ubuntu, Python 3.7.10, Cuda 11.1 and Pytorch 1.8.1

Comments
  • Package Versions

    Package Versions

    Hi,

    I tried to reproduce your results, but I ran into a possible version mismatch between Pytorch and Pytorch_geometric.

    I created my environment with the following commands:

    conda create --name poco python=3.7.10
conda install pytorch==1.8.1 torchvision==0.9.1 torchaudio==0.8.1 cudatoolkit=11.1 -c pytorch -c conda-forge
conda install -c conda-forge cython
conda install -c conda-forge tqdm 
conda install -c conda-forge scikit-image 
conda install -c open3d-admin open3d 
conda install -c conda-forge scikit-learn 
conda install -c conda-forge pyyaml 
conda install -c conda-forge addict 
conda install -c conda-forge pandas 
conda install -c conda-forge plyfile 
conda install -c conda-forge pytorch_geometric

    Compilation with python setup.py build_ext --inplace seems to work but python generate.py --config results/ABC_10k_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --dataset_name DATASET_NAME --dataset_root data/3d_shapes_abc/abc/ --gen_resolution_global 256 results in OSError: /home/perler/miniconda3/envs/poco/lib/python3.7/site-packages/torch_sparse/_version.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

    Installed versions are:

    (poco) [email protected]:~/repos/poco$ conda list pytorch
# packages in environment at /home/perler/miniconda3/envs/poco:
#
# Name                    Version                   Build  Channel
pytorch                   1.8.1           py3.7_cuda11.1_cudnn8.0.5_0    pytorch
pytorch-cpu               1.1.0               py3.7_cpu_0    pytorch
pytorch_geometric         2.0.3              pyh6c4a22f_0    conda-forge
pytorch_sparse            0.6.4            py37hcae2be3_0    conda-forge

    Again, the CPU-version... but that's a different issue.

    AFAIK, they added sparse tensors only recently to Pytorch, so the installed Pytorch-geometric version might be too new. Which version of Pytorch-geometric do I need?

    Can you please create a requiremtents.txt and/or environment.yaml?

    opened by ErlerPhilipp 13
  • Cython related error when building extensions

    Cython related error when building extensions

    Hi, I completely followed the instructions, using cuda11.1 and Ubuntu, but still having this issue gcc: error: /usr/POCO/build/temp.linux-x86_64-3.7/eval/src/utils/libkdtree/pykdtree/kdtree.o: No such file or directory gcc: error: /usr/POCO/build/temp.linux-x86_64-3.7/eval/src/utils/libkdtree/pykdtree/_kdtree_core.o: No such file or directory It's asking to link .o files before it is generated. But looks like required Cython 0.27.3 is not compatible with Python 3.7.10, which is required by this project. I'm wondering does anyone have a working combination of the version of the packages? Thank you!

    opened by zhaoyuanyuan2011 5
  • Download ABC / RealWorld / Famous and Thingi10k data

    Download ABC / RealWorld / Famous and Thingi10k data

    Hello,

    Thanks for the excellent paper and for releasing the code.

    I'm wondering where I can download the ABC, RealWorld, Famous, and Thingi10k data?

    Best and have a nice day, Mulin

    opened by MulinYu 2
  • Training on my own dataset

    Training on my own dataset

    I met some problem when training on my own dataset, which is processed as ShapeNet, containing pointclouds with normals, SDF points and occ values. However, there is an error when I try to train on it.

    File "/media/yangxilab/DiskB/sanren/3Drecon/POCOnasa/networks/backbone/fkaconv_network.py", line 193, in forward x0 = self.activation(self.bn0(self.cv0(x, pos, pos, data["ids00"]))) KeyError: 'ids00'

    I'm confusing that what data["ids00"] in x0 = self.activation(self.bn0(self.cv0(x, pos, pos, data["ids00"]))) refers to? And similarily, there are many unkown data like data["ids10"]. How could I get this data? Thanks in advance!

    opened by Green89757 2
  • Failed to run

    Failed to run "python setup.py build_ext --inplace"

    Hello,

    After installing Pytorch and Pytorch Geometric, running python setup.py build_ext --inplace gives the following error:

    gcc: error: /my/computer/path/to/POCO/build/temp.linux-x86_64-3.7/eval/src/utils/libkdtree/pykdtree/_kdtree_core.o: No such file or directory
error: command 'gcc' failed with exit status 1

    Any suggestions would be appreciated.

    opened by TixiaoShan 2
  • Bug during generating

    Bug during generating

    Hello again,

    When I try to generate the shapenet results with the command:

    python generate.py --config results/ShapeNet_Normals_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --gen_resolution_global 128

    I got this error:

    **loading CUDA OK INFO:root:Creating the network INFO:root:InterpNet - Simple - K=64 INFO:root:Network -- backbone -- 12783956 parameters INFO:root:Network -- projection -- 6498 parameters INFO:root:Network -- Number of parameters 12790454 INFO:root:Getting the dataset INFO:root:Normals as features INFO:root:Dataset - ShapeNet- None INFO:root:Dataset - len 8751 INFO:root:InterpNet - Simple - K=64 INFO:root:Network -- backbone -- 12783956 parameters INFO:root:Network -- projection -- 6498 parameters

    0%| | 0/8751 [00:00<?, ?it/s] 0%| | 0/8751 [00:04<?, ?it/s] Traceback (most recent call last): File "generate.py", line 594, in main(config) File "generate.py", line 539, in main step=step File "generate.py", line 170, in export_mesh_and_refine_vertices_region_growing_v2 mesh = o3d.geometry.TriangleMesh(o3d_verts, o3d_faces) TypeError: init(): incompatible constructor arguments. The following argument types are supported: 1. open3d.open3d.geometry.TriangleMesh() 2. open3d.open3d.geometry.TriangleMesh(arg0: open3d.open3d.geometry.TriangleMesh)

    Invoked with: std::vectorEigen::Vector3d with 133788 elements. Use numpy.asarray() to access data., std::vectorEigen::Vector3i with 267574 elements. Use numpy.asarray() to access data.

    Did you forget to #include <pybind11/stl.h>? Or <pybind11/complex.h>, <pybind11/functional.h>, <pybind11/chrono.h>, etc. Some automatic conversions are optional and require extra headers to be included when compiling your pybind11 module. **

    Thanks in advance and have a nice day. Best. Mulin

    opened by MulinYu 1
  • It takes one hour to generate the mesh from 16k point clouds in ScanNet dataset

    It takes one hour to generate the mesh from 16k point clouds in ScanNet dataset

    Dear author,

    Thanks for sharing your interesting work!

    I use the provided script to generate the mesh from the ScanNet dataset, and I found that it takes a long time (e.g, nearly 1 hour from 16K point clouds input). I am wondering if it is a normal situation. BTW, I just use the following script to generate mesh; python generate.py --config results/ShapeNet_Normals_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --gen_autoscale True --gen_autoscale_target 0.01 --gen_resolution_metric 0.01 --dataset_name SceneNet20 --dataset_root data/SceneNet20 --manifold_points -1

    Looking forward to your reply. Many thanks in advance.

    Best, Runsong

    opened by Runsong123 0
  • RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when running the code

    RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when running the code

    Hi! Thanks for your interesting and enlightening work on point cloud reconstruction tasks, and we are trying to reproduce your work. However, we encounter an error when running your code: Capture I am wondering how to fix this bug? Thanks for your timely response.

    opened by CUHKWilliam 3
Owner
valeo.ai
We are an international team based in Paris, conducting AI research for Valeo automotive applications, in collaboration with world-class academics.
valeo.ai
GitHub Repository
Approaches to modeling terrain and maps in python

topography 🌎 Contains different approaches to modeling terrain and topographic-style maps in python Features Inverse Distance Weighting (IDW) A given

John Gutierrez 1 Aug 10, 2022
The implemetation of Dynamic Nerual Garments proposed in Siggraph Asia 2021

DynamicNeuralGarments Introduction This repository contains the implemetation of Dynamic Nerual Garments proposed in Siggraph Asia 2021. ./GarmentMoti

42 Dec 27, 2022
Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval (NeurIPS'21)

Baleen Baleen is a state-of-the-art model for multi-hop reasoning, enabling scalable multi-hop search over massive collections for knowledge-intensive

Stanford Future Data Systems 22 Dec 05, 2022
High performance, easy-to-use, and scalable machine learning (ML) package, including linear model (LR), factorization machines (FM), and field-aware factorization machines (FFM) for Python and CLI interface.

What is xLearn? xLearn is a high performance, easy-to-use, and scalable machine learning package that contains linear model (LR), factorization machin

Chao Ma 3k Jan 03, 2023
Encoding Causal Macrovariables

Encoding Causal Macrovariables Data Natural climate data ('El Nino') Self-generated data ('Simulated') Experiments Detecting macrovariables through th

Benedikt Höltgen 3 Jul 31, 2022
OMLT: Optimization and Machine Learning Toolkit

OMLT is a Python package for representing machine learning models (neural networks and gradient-boosted trees) within the Pyomo optimization environment.

C⚙G - Imperial College London 179 Jan 02, 2023
Attack classification models with transferability, black-box attack; unrestricted adversarial attacks on imagenet

Attack classification models with transferability, black-box attack; unrestricted adversarial attacks on imagenet, CVPR2021 安全AI挑战者计划第六期：ImageNet无限制对抗攻击 决赛第四名（team name: Advers）

51 Dec 01, 2022
source code the paper Fast and Robust Iterative Closet Point.

Fast-Robust-ICP This repository includes the source code the paper Fast and Robust Iterative Closet Point. Authors: Juyong Zhang, Yuxin Yao, Bailin De

yaoyuxin 320 Dec 28, 2022
MonoRCNN is a monocular 3D object detection method for automonous driving

MonoRCNN MonoRCNN is a monocular 3D object detection method for automonous driving, published at ICCV 2021. This project is an implementation of MonoR

87 Dec 27, 2022
UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss

UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss This repository contains the TensorFlow implementation of the paper UnF

Simon Meister 270 Nov 06, 2022
Code for DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learning

DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learning Pytorch Implementation for DisCo: Remedy Self-supervi

79 Jan 06, 2023
MAGMA - a GPT-style multimodal model that can understand any combination of images and language

MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuning Authors repo (alphabetical) Constantin (CoEich), Mayukh (Mayukh

Aleph Alpha GmbH 331 Jan 03, 2023
As-ViT: Auto-scaling Vision Transformers without Training

As-ViT: Auto-scaling Vision Transformers without Training [PDF] Wuyang Chen, Wei Huang, Xianzhi Du, Xiaodan Song, Zhangyang Wang, Denny Zhou In ICLR 2

VITA 68 Sep 05, 2022
Official repository for the paper "Self-Supervised Models are Continual Learners" (CVPR 2022)

Self-Supervised Models are Continual Learners This is the official repository for the paper: Self-Supervised Models are Continual Learners Enrico Fini

Enrico Fini 73 Dec 18, 2022
Multiple Object Tracking with Yolov5!

Tracking with yolov5 This implementation is for who need to tracking multi-object only with detector. You can easily track mult-object with your well

9 Nov 08, 2022
Class-Balanced Loss Based on Effective Number of Samples. CVPR 2019

Class-Balanced Loss Based on Effective Number of Samples Tensorflow code for the paper: Class-Balanced Loss Based on Effective Number of Samples Yin C

Yin Cui 546 Jan 08, 2023
ManipulaTHOR, a framework that facilitates visual manipulation of objects using a robotic arm

ManipulaTHOR: A Framework for Visual Object Manipulation Kiana Ehsani, Winson Han, Alvaro Herrasti, Eli VanderBilt, Luca Weihs, Eric Kolve, Aniruddha

AI2 65 Dec 30, 2022
A simple Python library for stochastic graphical ecological models

What is Viridicle? Viridicle is a library for simulating stochastic graphical ecological models. It implements the continuous time models described in

Theorem Engine 0 Dec 04, 2021
Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement (NeurIPS 2020)

MTTS-CAN: Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement Paper Xin Liu, Josh Fromm, Shwetak Patel, Daniel M

Xin Liu 106 Dec 30, 2022
A flexible tool for creating, organizing, and sharing visualizations of live, rich data. Supports Torch and Numpy.

Visdom A flexible tool for creating, organizing, and sharing visualizations of live, rich data. Supports Python. Overview Concepts Setup Usage API To

FOSSASIA 9.4k Jan 07, 2023