GeoTransformer - Geometric Transformer for Fast and Robust Point Cloud Registration

Overview

Geometric Transformer for Fast and Robust Point Cloud Registration

PyTorch implementation of the paper:

Geometric Transformer for Fast and Robust Point Cloud Registration.

Zheng Qin, Hao Yu, Changjian Wang, Yulan Guo, Yuxing Peng, and Kai Xu

Introduction

We study the problem of extracting accurate correspondences for point cloud registration. Recent keypoint-free methods bypass the detection of repeatable keypoints which is difficult in low-overlap scenarios, showing great potential in registration. They seek correspondences over downsampled superpoints, which are then propagated to dense points. Superpoints are matched based on whether their neighboring patches overlap. Such sparse and loose matching requires contextual features capturing the geometric structure of the point clouds. We propose Geometric Transformer to learn geometric feature for robust superpoint matching. It encodes pair-wise distances and triplet-wise angles, making it robust in low-overlap cases and invariant to rigid transformation. The simplistic design attains surprisingly high matching accuracy such that no RANSAC is required in the estimation of alignment transformation, leading to $100$ times acceleration. Our method improves the inlier ratio by $17% \sim 30%$ and the registration recall by over $7%$ on the challenging 3DLoMatch benchmark. Code will be released for paper reproduction.

News

Installation

Please use the following command for installation.

# It is recommended to create a new environment
conda create -n geotransformer python==3.8
conda activate geotransformer

# [Optional] If you are using CUDA 11.0 or newer, please install `torch==1.7.1+cu110`
pip install torch==1.7.1+cu110 -f https://download.pytorch.org/whl/torch_stable.html

# Install packages and other dependencies
python setup.py build develop

# Compile c++ wrappers
cd geotransformer/cpp_wrappers
sh ./compile_wrappers.sh

Code has been tested with Ubuntu 20.04, GCC 9.3.0, Python 3.8, PyTorch 1.7.1, CUDA 11.1 and cuDNN 8.1.0.

Data preparation

We provide code for training and testing on 3DMatch.

The dataset can be download from PREDATOR. The data should be organized as follows:

--data--3DMatch--metadata
              |--data--train--7-scenes-chess--cloud_bin_0.pth
                    |      |               |--...
                    |      |--...
                    |--test--7-scenes-redkitchen--cloud_bin_0.pth
                          |                    |--...
                          |--...

Training

The code for GeoTransformer is in experiments/geotransformer.3dmatch. Use the following command for training.

CUDA_VISIBLE_DEVICES=0 python trainval.py
# use "--snapshot=path/to/snapshot" to resume training.

Testing

Use the following command for testing.

# 3DMatch
CUDA_VISIBLE_DEVICES=0 ./eval.sh EPOCH 3DMatch
# 3DLoMatch
CUDA_VISIBLE_DEVICES=0 ./eval.sh EPOCH 3DLoMatch

EPOCH is the epoch id.

We also provide pretrained weights in weights, use the following command to test the pretrained weights.

CUDA_VISIBLE_DEVICES=0 python test.py --snapshot=../../weights/geotransformer-3dmatch.pth.tar --benchmark=3DMatch
CUDA_VISIBLE_DEVICES=0 python eval.py --run_matching --run_registration --benchmark=3DMatch

Replace 3DMatch with 3DLoMatch to evaluate on 3DLoMatch.

Results

Benchmark FMR IR RR
3DMatch 97.7 70.3 91.5
3DLoMatch 88.1 43.3 74.0
Comments
  • Test and visualize on two given point clouds

    Test and visualize on two given point clouds

    Hi, thanks for sharing your work. Is it easy to give some guidance or do you have any tutorial to test/evaluate the pre-trained models on two individual given point clouds?

    opened by ttsesm 10
  • Question about kitti result reproduce.

    Question about kitti result reproduce.

    Hi ,thanks for your amazing work.

    However , I have met a problem of kitti dataset result reproduce. I followed your code to train and test on kitti dataset , but my result seems always lower than the result in your paper. I can achieve Registration recall of 99.8 and RTE 6.8 same as yours ,but my result of RRE is about 0.33 which is much higher than 0.24 in your paper.

    Can you give me some idea of how to reproduce the same result ?

    opened by vision507 9
  • How to draw the point cloud like this ?

    How to draw the point cloud like this ?

    Hi , thanks for your great work. And I would like to know how can I get this kind of visualization result in your paper. 屏幕截图 2022-08-04 081115 When I use open3d to draw the point clouds , it seems very ugly, so I'm wondering how can I get your visualization result. It would be very grateful.

    opened by vision507 9
  • How to group data sets after downloading

    How to group data sets after downloading

    Hello, author: I ran the python files in the Kitti dataset and ModelNet dataset folders according to your requirements, but they did not work and were not classified into the ones you described. Is there a specific method? thank you!

    opened by yangsihai97 9
  • Some questions about downsampling of point clouds

    Some questions about downsampling of point clouds

    Hi there,

    Really appreciate releasing your amazing work! I'm quite new to point clouds and I have some questions regarding downsampling of the point clouds.

    Take 3DMatch as an example. The original raw dataset downloaded from PREDATOR is quite dense, and is preprocessed so that each grid size is 2.5cm. In your work, you further subsample the point clouds using KPConv (the first level downsampled points). My first question is what's the purpose of subsampling? Would a denser point cloud be more informative for registration?

    Also, is there a specific reason you downsample the points further for dense point correspondences as I believe the preprocessed input data is already of density 2.5cm?

    Last, is there a quick way to visualize the registration result for the 3DMatch benchmark? By running eval.py and test.py it seems to only produce the quantitative metrics.

    Thanks for your patience and reply in advance!

    opened by jinhong-ni 5
  • 3DMatch dataset

    3DMatch dataset

    Hi,

    I have a question regarding the D.2. section from the Supplementary material.

    Could you please share which 3DMatch files you use for the evaluation?

    • Do you use those with overlap > 30%?
    • Do you use consecutive pairs of point cloud indices (the original 3DMatch filters the pairs (i,j) with i+1 < j)?

    You mention that you use the files from PREDATOR, but PREDATOR has more pairs than the original 3DMatch dataset. Can you please elaborate?

    Thank you in advance for you response. Hopefully we can clear all this confusion of the registration pairs used in the datasets. Every paper seems to do it a bit differently.

    opened by DavidBoja 4
  • why need gt transoform in demo.py

    why need gt transoform in demo.py

    in line 74 in experiments/geotransformer.3dmatch.stage4.gse.k3.max.oacl.stage2.sinkhorn/model.py , transform = data_dict['transform'].detach(), and in line 110 you use gt transform in get_node_correspondences. however If i don't know how the transform matrix of src to ref, how can I use demo.py

    opened by MaybeOjbk 4
  • The different RR in test.py and eval.py

    The different RR in test.py and eval.py

    Hello, thank you for you amazing work again. When I evaluated on 3DMatch, I found the RR reported by test.py and eval.py are very different, what makes this difference?

    opened by Hui-design 4
  • Question about radius search

    Question about radius search

    First of all, thanks for your excellent work! Radius search in KPConv is quiet slow for large scale point clouds and it usually takes seconds in my own experiments. Is there a more efficient way for radius search, such as a version that can run on GPU. Hope to get your answer!

    opened by Fleurrr 4
  • Questions about RANSAC

    Questions about RANSAC

    Dear authors,

    Thanks for open-sourcing your amazing work. I'm experimenting with RANSAC and I noticed some unexpected results compared to the paper.

    What I've done is first change the number of iterations in RANSAC to 50k. The evaluation runs extremely slowly (approx. 16 hrs in contrast to about 1 hr for local to global), and I only got performance similar to local to global. I also tried to change the number of points in RANSAC to a larger number (by default it is set to 3), such as 250, 500, 1000, etc. I did not achieve a significant gain in performance, even the recall collapses to zero if the number of points is set too big (like 5k). This is contradicting to what's reported in the paper, as the performance is reported to keep improving as the number of samples increases.

    I would be super grateful if you could guide me through what I've done wrong here. Thanks in advance for your patience and help!

    opened by jinhong-ni 4
  • RuntimeError: Error compiling objects for extension

    RuntimeError: Error compiling objects for extension

    Hello, sir. The question is about creating the code runtime environment. After reinstalling CUDA and downloading torch, I executed 'python setup.py build develop' and it still report the error, 'RuntimeError: Error compiling objects for extension'.

    Traceback (most recent call last): File "setup.py", line 5, in setup( File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_init_.py", line 87, in setup return distutils.core.setup(**attrs) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\core.py", line 148, in setup return run_commands(dist) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\core.py", line 163, in run_commands dist.run_commands() File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\dist.py", line 967, in run_commands self.run_command(cmd) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools\dist.py", line 1214, in run_command super().run_command(command) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\dist.py", line 986, in run_command cmd_obj.run() File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\command\build.py", line 135, in run self.run_command(cmd_name) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\cmd.py", line 313, in run_command self.distribution.run_command(command) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools\dist.py", line 1214, in run_command super().run_command(command) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\dist.py", line 986, in run_command cmd_obj.run() File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools\command\build_ext.py", line 79, in run _build_ext.run(self) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\command\build_ext.py", line 339, in run self.build_extensions() File "F:\Anaconda3\envs\geotransformer\lib\site-packages\torch\utils\cpp_extension.py", line 670, in build_extensions build_ext.build_extensions(self) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\command\build_ext.py", line 448, in build_extensions self._build_extensions_serial() File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\command\build_ext.py", line 473, in _build_extensions_serial self.build_extension(ext) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools\command\build_ext.py", line 202, in build_extension _build_ext.build_extension(self, ext) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\command\build_ext.py", line 528, in build_extension objects = self.compiler.compile(sources, File "F:\Anaconda3\envs\geotransformer\lib\site-packages\torch\utils\cpp_extension.py", line 643, in win_wrap_ninja_compile _write_ninja_file_and_compile_objects( File "F:\Anaconda3\envs\geotransformer\lib\site-packages\torch\utils\cpp_extension.py", line 1250, in _write_ninja_file_and_compile_objects _run_ninja_build( File "F:\Anaconda3\envs\geotransformer\lib\site-packages\torch\utils\cpp_extension.py", line 1555, in _run_ninja_build raise RuntimeError(message) from e RuntimeError: Error compiling objects for extension

    Is this because I use win10? Or other issues caused the error?

    opened by ChaunceyQ 4
  • The training model is not effective

    The training model is not effective

    Thank you for your excellent work. When I use the LGR training model without changing the code, the result on 3DMatch (about 91.0%) was still quite different from the model you provided (92.5%)

    opened by awdx551 3
Owner
Zheng Qin
computer vision, deep learning
Zheng Qin
SpeechNAS Better Trade off between Latency and Accuracy for Large Scale Speaker Verification

SpeechNAS Better Trade off between Latency and Accuracy for Large Scale Speaker Verification

Wentao Zhu 24 May 20, 2022
This is the official code for the paper "Tracker Meets Night: A Transformer Enhancer for UAV Tracking".

SCT This is the official code for the paper "Tracker Meets Night: A Transformer Enhancer for UAV Tracking" The spatial-channel Transformer (SCT) enhan

Intelligent Vision for Robotics in Complex Environment 27 Nov 23, 2022
Generative Adversarial Text-to-Image Synthesis

###Generative Adversarial Text-to-Image Synthesis Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, Honglak Lee This is the

Scott Ellison Reed 883 Dec 31, 2022
PyDeepFakeDet is an integrated and scalable tool for Deepfake detection.

PyDeepFakeDet An integrated and scalable library for Deepfake detection research. Introduction PyDeepFakeDet is an integrated and scalable Deepfake de

Junke, Wang 49 Dec 11, 2022
Python implementation of Bayesian optimization over permutation spaces.

Bayesian Optimization over Permutation Spaces This repository contains the source code and the resources related to the paper "Bayesian Optimization o

Aryan Deshwal 9 Dec 23, 2022
VGG16 model-based classification project about brain tumor detection.

Brain-Tumor-Classification-with-MRI VGG16 model-based classification project about brain tumor detection. First, you can check what people are doing o

Atakan Erdoğan 2 Mar 21, 2022
Official implementation of the ICML2021 paper "Elastic Graph Neural Networks"

ElasticGNN This repository includes the official implementation of ElasticGNN in the paper "Elastic Graph Neural Networks" [ICML 2021]. Xiaorui Liu, W

liuxiaorui 34 Dec 04, 2022
Pytorch Implementation of paper "Noisy Natural Gradient as Variational Inference"

Noisy Natural Gradient as Variational Inference PyTorch implementation of Noisy Natural Gradient as Variational Inference. Requirements Python 3 Pytor

Tony JiHyun Kim 119 Dec 02, 2022
LegoDNN: a block-grained scaling tool for mobile vision systems

Table of contents 1 Introduction 1.1 Major features 1.2 Architecture 2 Code and Installation 2.1 Code 2.2 Installation 3 Repository of DNNs in vision

41 Dec 24, 2022
Effect of Deep Transfer and Multi task Learning on Sperm Abnormality Detection

Effect of Deep Transfer and Multi task Learning on Sperm Abnormality Detection Introduction This repository includes codes and models of "Effect of De

Amir Abbasi 5 Sep 05, 2022
Discover hidden deepweb pages

DeepWeb Scapper Att: Demo version An simple script to scrappe deepweb to find pages. Will return if any of those exists and will save on a file. You s

Héber Júlio 77 Oct 02, 2022
LiDAR Distillation: Bridging the Beam-Induced Domain Gap for 3D Object Detection

LiDAR Distillation Paper | Model LiDAR Distillation: Bridging the Beam-Induced Domain Gap for 3D Object Detection Yi Wei, Zibu Wei, Yongming Rao, Jiax

Yi Wei 75 Dec 22, 2022
A working implementation of the Categorical DQN (Distributional RL).

Categorical DQN. Implementation of the Categorical DQN as described in A distributional Perspective on Reinforcement Learning. Thanks to @tudor-berari

Florin Gogianu 98 Sep 20, 2022
Discord-Protect is a simple discord bot allowing you to have some security on your discord server by ordering a captcha to the user who joins your server.

Discord-Protect Discord-Protect is a simple discord bot allowing you to have some security on your discord server by ordering a captcha to the user wh

Tir Omar 2 Oct 28, 2021
Coded illumination for improved lensless imaging

CodedCam Coded Illumination for Improved Lensless Imaging Paper | Supplementary results | Data and Code are available. Coded illumination for improved

Computational Sensing and Information Processing Lab 1 Nov 29, 2021
DiffQ performs differentiable quantization using pseudo quantization noise. It can automatically tune the number of bits used per weight or group of weights, in order to achieve a given trade-off between model size and accuracy.

Differentiable Model Compression via Pseudo Quantization Noise DiffQ performs differentiable quantization using pseudo quantization noise. It can auto

Facebook Research 145 Dec 30, 2022
A system used to detect whether a person is wearing a medical mask or not.

Mask_Detection_System A system used to detect whether a person is wearing a medical mask or not. To open the program, please follow these steps: Make

Mohamed Emad 0 Nov 17, 2022
simple artificial intelligence utilities

Simple AI Project home: http://github.com/simpleai-team/simpleai This lib implements many of the artificial intelligence algorithms described on the b

921 Dec 08, 2022
A new test set for ImageNet

ImageNetV2 The ImageNetV2 dataset contains new test data for the ImageNet benchmark. This repository provides associated code for assembling and worki

186 Dec 18, 2022
Library for machine learning stacking generalization.

stacked_generalization Implemented machine learning *stacking technic[1]* as handy library in Python. Feature weighted linear stacking is also availab

114 Jul 19, 2022