GeoTransformer - Geometric Transformer for Fast and Robust Point Cloud Registration

Overview

Geometric Transformer for Fast and Robust Point Cloud Registration

PyTorch implementation of the paper:

Geometric Transformer for Fast and Robust Point Cloud Registration.

Zheng Qin, Hao Yu, Changjian Wang, Yulan Guo, Yuxing Peng, and Kai Xu

Introduction

We study the problem of extracting accurate correspondences for point cloud registration. Recent keypoint-free methods bypass the detection of repeatable keypoints which is difficult in low-overlap scenarios, showing great potential in registration. They seek correspondences over downsampled superpoints, which are then propagated to dense points. Superpoints are matched based on whether their neighboring patches overlap. Such sparse and loose matching requires contextual features capturing the geometric structure of the point clouds. We propose Geometric Transformer to learn geometric feature for robust superpoint matching. It encodes pair-wise distances and triplet-wise angles, making it robust in low-overlap cases and invariant to rigid transformation. The simplistic design attains surprisingly high matching accuracy such that no RANSAC is required in the estimation of alignment transformation, leading to $100$ times acceleration. Our method improves the inlier ratio by $17% \sim 30%$ and the registration recall by over $7%$ on the challenging 3DLoMatch benchmark. Code will be released for paper reproduction.

News

Installation

Please use the following command for installation.

# It is recommended to create a new environment
conda create -n geotransformer python==3.8
conda activate geotransformer

# [Optional] If you are using CUDA 11.0 or newer, please install `torch==1.7.1+cu110`
pip install torch==1.7.1+cu110 -f https://download.pytorch.org/whl/torch_stable.html

# Install packages and other dependencies
python setup.py build develop

# Compile c++ wrappers
cd geotransformer/cpp_wrappers
sh ./compile_wrappers.sh

Code has been tested with Ubuntu 20.04, GCC 9.3.0, Python 3.8, PyTorch 1.7.1, CUDA 11.1 and cuDNN 8.1.0.

Data preparation

We provide code for training and testing on 3DMatch.

The dataset can be download from PREDATOR. The data should be organized as follows:

--data--3DMatch--metadata
              |--data--train--7-scenes-chess--cloud_bin_0.pth
                    |      |               |--...
                    |      |--...
                    |--test--7-scenes-redkitchen--cloud_bin_0.pth
                          |                    |--...
                          |--...

Training

The code for GeoTransformer is in experiments/geotransformer.3dmatch. Use the following command for training.

CUDA_VISIBLE_DEVICES=0 python trainval.py
# use "--snapshot=path/to/snapshot" to resume training.

Testing

Use the following command for testing.

# 3DMatch
CUDA_VISIBLE_DEVICES=0 ./eval.sh EPOCH 3DMatch
# 3DLoMatch
CUDA_VISIBLE_DEVICES=0 ./eval.sh EPOCH 3DLoMatch

EPOCH is the epoch id.

We also provide pretrained weights in weights, use the following command to test the pretrained weights.

CUDA_VISIBLE_DEVICES=0 python test.py --snapshot=../../weights/geotransformer-3dmatch.pth.tar --benchmark=3DMatch
CUDA_VISIBLE_DEVICES=0 python eval.py --run_matching --run_registration --benchmark=3DMatch

Replace 3DMatch with 3DLoMatch to evaluate on 3DLoMatch.

Results

Benchmark FMR IR RR
3DMatch 97.7 70.3 91.5
3DLoMatch 88.1 43.3 74.0
Comments
  • Test and visualize on two given point clouds

    Test and visualize on two given point clouds

    Hi, thanks for sharing your work. Is it easy to give some guidance or do you have any tutorial to test/evaluate the pre-trained models on two individual given point clouds?

    opened by ttsesm 10
  • Question about kitti result reproduce.

    Question about kitti result reproduce.

    Hi ,thanks for your amazing work.

    However , I have met a problem of kitti dataset result reproduce. I followed your code to train and test on kitti dataset , but my result seems always lower than the result in your paper. I can achieve Registration recall of 99.8 and RTE 6.8 same as yours ,but my result of RRE is about 0.33 which is much higher than 0.24 in your paper.

    Can you give me some idea of how to reproduce the same result ?

    opened by vision507 9
  • How to draw the point cloud like this ?

    How to draw the point cloud like this ?

    Hi , thanks for your great work. And I would like to know how can I get this kind of visualization result in your paper. 屏幕截图 2022-08-04 081115 When I use open3d to draw the point clouds , it seems very ugly, so I'm wondering how can I get your visualization result. It would be very grateful.

    opened by vision507 9
  • How to group data sets after downloading

    How to group data sets after downloading

    Hello, author: I ran the python files in the Kitti dataset and ModelNet dataset folders according to your requirements, but they did not work and were not classified into the ones you described. Is there a specific method? thank you!

    opened by yangsihai97 9
  • Some questions about downsampling of point clouds

    Some questions about downsampling of point clouds

    Hi there,

    Really appreciate releasing your amazing work! I'm quite new to point clouds and I have some questions regarding downsampling of the point clouds.

    Take 3DMatch as an example. The original raw dataset downloaded from PREDATOR is quite dense, and is preprocessed so that each grid size is 2.5cm. In your work, you further subsample the point clouds using KPConv (the first level downsampled points). My first question is what's the purpose of subsampling? Would a denser point cloud be more informative for registration?

    Also, is there a specific reason you downsample the points further for dense point correspondences as I believe the preprocessed input data is already of density 2.5cm?

    Last, is there a quick way to visualize the registration result for the 3DMatch benchmark? By running eval.py and test.py it seems to only produce the quantitative metrics.

    Thanks for your patience and reply in advance!

    opened by jinhong-ni 5
  • 3DMatch dataset

    3DMatch dataset

    Hi,

    I have a question regarding the D.2. section from the Supplementary material.

    Could you please share which 3DMatch files you use for the evaluation?

    • Do you use those with overlap > 30%?
    • Do you use consecutive pairs of point cloud indices (the original 3DMatch filters the pairs (i,j) with i+1 < j)?

    You mention that you use the files from PREDATOR, but PREDATOR has more pairs than the original 3DMatch dataset. Can you please elaborate?

    Thank you in advance for you response. Hopefully we can clear all this confusion of the registration pairs used in the datasets. Every paper seems to do it a bit differently.

    opened by DavidBoja 4
  • why need gt transoform in demo.py

    why need gt transoform in demo.py

    in line 74 in experiments/geotransformer.3dmatch.stage4.gse.k3.max.oacl.stage2.sinkhorn/model.py , transform = data_dict['transform'].detach(), and in line 110 you use gt transform in get_node_correspondences. however If i don't know how the transform matrix of src to ref, how can I use demo.py

    opened by MaybeOjbk 4
  • The different RR in test.py and eval.py

    The different RR in test.py and eval.py

    Hello, thank you for you amazing work again. When I evaluated on 3DMatch, I found the RR reported by test.py and eval.py are very different, what makes this difference?

    opened by Hui-design 4
  • Question about radius search

    Question about radius search

    First of all, thanks for your excellent work! Radius search in KPConv is quiet slow for large scale point clouds and it usually takes seconds in my own experiments. Is there a more efficient way for radius search, such as a version that can run on GPU. Hope to get your answer!

    opened by Fleurrr 4
  • Questions about RANSAC

    Questions about RANSAC

    Dear authors,

    Thanks for open-sourcing your amazing work. I'm experimenting with RANSAC and I noticed some unexpected results compared to the paper.

    What I've done is first change the number of iterations in RANSAC to 50k. The evaluation runs extremely slowly (approx. 16 hrs in contrast to about 1 hr for local to global), and I only got performance similar to local to global. I also tried to change the number of points in RANSAC to a larger number (by default it is set to 3), such as 250, 500, 1000, etc. I did not achieve a significant gain in performance, even the recall collapses to zero if the number of points is set too big (like 5k). This is contradicting to what's reported in the paper, as the performance is reported to keep improving as the number of samples increases.

    I would be super grateful if you could guide me through what I've done wrong here. Thanks in advance for your patience and help!

    opened by jinhong-ni 4
  • RuntimeError: Error compiling objects for extension

    RuntimeError: Error compiling objects for extension

    Hello, sir. The question is about creating the code runtime environment. After reinstalling CUDA and downloading torch, I executed 'python setup.py build develop' and it still report the error, 'RuntimeError: Error compiling objects for extension'.

    Traceback (most recent call last): File "setup.py", line 5, in setup( File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_init_.py", line 87, in setup return distutils.core.setup(**attrs) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\core.py", line 148, in setup return run_commands(dist) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\core.py", line 163, in run_commands dist.run_commands() File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\dist.py", line 967, in run_commands self.run_command(cmd) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools\dist.py", line 1214, in run_command super().run_command(command) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\dist.py", line 986, in run_command cmd_obj.run() File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\command\build.py", line 135, in run self.run_command(cmd_name) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\cmd.py", line 313, in run_command self.distribution.run_command(command) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools\dist.py", line 1214, in run_command super().run_command(command) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\dist.py", line 986, in run_command cmd_obj.run() File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools\command\build_ext.py", line 79, in run _build_ext.run(self) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\command\build_ext.py", line 339, in run self.build_extensions() File "F:\Anaconda3\envs\geotransformer\lib\site-packages\torch\utils\cpp_extension.py", line 670, in build_extensions build_ext.build_extensions(self) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\command\build_ext.py", line 448, in build_extensions self._build_extensions_serial() File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\command\build_ext.py", line 473, in _build_extensions_serial self.build_extension(ext) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools\command\build_ext.py", line 202, in build_extension _build_ext.build_extension(self, ext) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\command\build_ext.py", line 528, in build_extension objects = self.compiler.compile(sources, File "F:\Anaconda3\envs\geotransformer\lib\site-packages\torch\utils\cpp_extension.py", line 643, in win_wrap_ninja_compile _write_ninja_file_and_compile_objects( File "F:\Anaconda3\envs\geotransformer\lib\site-packages\torch\utils\cpp_extension.py", line 1250, in _write_ninja_file_and_compile_objects _run_ninja_build( File "F:\Anaconda3\envs\geotransformer\lib\site-packages\torch\utils\cpp_extension.py", line 1555, in _run_ninja_build raise RuntimeError(message) from e RuntimeError: Error compiling objects for extension

    Is this because I use win10? Or other issues caused the error?

    opened by ChaunceyQ 4
  • The training model is not effective

    The training model is not effective

    Thank you for your excellent work. When I use the LGR training model without changing the code, the result on 3DMatch (about 91.0%) was still quite different from the model you provided (92.5%)

    opened by awdx551 3
Owner
Zheng Qin
computer vision, deep learning
Zheng Qin
A Python module for the generation and training of an entry-level feedforward neural network.

ff-neural-network A Python module for the generation and training of an entry-level feedforward neural network. This repository serves as a repurposin

Riadh 2 Jan 31, 2022
Algorithms for outlier, adversarial and drift detection

Alibi Detect is an open source Python library focused on outlier, adversarial and drift detection. The package aims to cover both online and offline d

Seldon 1.6k Dec 31, 2022
Certifiable Outlier-Robust Geometric Perception

Certifiable Outlier-Robust Geometric Perception About This repository holds the implementation for certifiably solving outlier-robust geometric percep

83 Dec 31, 2022
object detection; robust detection; ACM MM21 grand challenge; Security AI Challenger Phase VII

赛题背景 在商品知识产权领域,知识产权体现为在线商品的设计和品牌。不幸的是,在每一天,存在着非法商户通过一些对抗手段干扰商标识别来逃避侵权,这带来了很高的知识产权风险和财务损失。为了促进先进的多媒体人工智能技术的发展,以保护企业来之不易的创作和想法免受恶意使用和剽窃,因此提出了鲁棒性标识检测挑战赛

65 Dec 22, 2022
A Rao-Blackwellized Particle Filter for 6D Object Pose Tracking

PoseRBPF: A Rao-Blackwellized Particle Filter for 6D Object Pose Tracking PoseRBPF Paper Self-supervision Paper Pose Estimation Video Robot Manipulati

NVIDIA Research Projects 107 Dec 25, 2022
Segmentation vgg16 fcn - cityscapes

VGGSegmentation Segmentation vgg16 fcn - cityscapes Priprema skupa skripta prepare_dataset_downsampled.py Iz slika cityscapesa izrezuje haubu automobi

6 Oct 24, 2020
A Python module for parallel optimization of expensive black-box functions

blackbox: A Python module for parallel optimization of expensive black-box functions What is this? A minimalistic and easy-to-use Python module that e

Paul Knysh 426 Dec 08, 2022
Head2Toe: Utilizing Intermediate Representations for Better OOD Generalization

Head2Toe: Utilizing Intermediate Representations for Better OOD Generalization Code for reproducing our results in the Head2Toe paper. Paper: arxiv.or

Google Research 62 Dec 12, 2022
Official Implementation of CoSMo: Content-Style Modulation for Image Retrieval with Text Feedback

CoSMo.pytorch Official Implementation of CoSMo: Content-Style Modulation for Image Retrieval with Text Feedback, Seungmin Lee*, Dongwan Kim*, Bohyung

Seung Min Lee 54 Dec 08, 2022
Le dataset des images du projet d'IA de 2021

face-mask-dataset-ilc-2021 Le dataset des images du projet d'IA de 2021, Indiquez vos id git dans la issue pour les droits TL;DR: Choisir 200 images J

7 Nov 15, 2021
🌈 PyTorch Implementation for EMNLP'21 Findings "Reasoning Visual Dialog with Sparse Graph Learning and Knowledge Transfer"

SGLKT-VisDial Pytorch Implementation for the paper: Reasoning Visual Dialog with Sparse Graph Learning and Knowledge Transfer Gi-Cheon Kang, Junseok P

Gi-Cheon Kang 9 Jul 05, 2022
Simple Pixelbot for Diablo 2 Resurrected written in python and opencv.

Simple Pixelbot for Diablo 2 Resurrected written in python and opencv. Obviously only use it in offline mode as it is against the TOS of Blizzard to use it in online mode!

468 Jan 03, 2023
MusicYOLO framework uses the object detection model, YOLOx, to locate notes in the spectrogram.

MusicYOLO MusicYOLO framework uses the object detection model, YOLOX, to locate notes in the spectrogram. Its performance on the ISMIR2014 dataset, MI

Xianke Wang 2 Aug 02, 2022
CAR-API: Cityscapes Attributes Recognition API

CAR-API: Cityscapes Attributes Recognition API This is the official api to download and fetch attributes annotations for Cityscapes Dataset. Content I

Kareem Metwaly 5 Dec 22, 2022
An Unpaired Sketch-to-Photo Translation Model

Unpaired-Sketch-to-Photo-Translation We have released our code at https://github.com/rt219/Unsupervised-Sketch-to-Photo-Synthesis This project is the

38 Oct 28, 2022
TensorFlow GNN is a library to build Graph Neural Networks on the TensorFlow platform.

TensorFlow GNN This is an early (alpha) release to get community feedback. It's under active development and we may break API compatibility in the fut

889 Dec 30, 2022
RodoSol-ALPR Dataset

RodoSol-ALPR Dataset This dataset, called RodoSol-ALPR dataset, contains 20,000 images captured by static cameras located at pay tolls owned by the Ro

Rayson Laroca 45 Dec 15, 2022
Code and data for "TURL: Table Understanding through Representation Learning"

TURL This Repo contains code and data for "TURL: Table Understanding through Representation Learning". Environment and Setup Data Pretraining Finetuni

SunLab-OSU 63 Nov 23, 2022
Erpnext app for make employee salary on payroll entry based on one or more project with percentage for all project equal 100 %

Project Payroll this app for make payroll for employee based on projects like project on 30 % and project 2 70 % as account dimension it makes genral

Ibrahim Morghim 8 Jan 02, 2023
A real-time approach for mapping all human pixels of 2D RGB images to a 3D surface-based model of the body

DensePose: Dense Human Pose Estimation In The Wild Rıza Alp Güler, Natalia Neverova, Iasonas Kokkinos [densepose.org] [arXiv] [BibTeX] Dense human pos

Meta Research 6.4k Jan 01, 2023