GeoTransformer - Geometric Transformer for Fast and Robust Point Cloud Registration

Overview

Geometric Transformer for Fast and Robust Point Cloud Registration

PyTorch implementation of the paper:

Geometric Transformer for Fast and Robust Point Cloud Registration.

Zheng Qin, Hao Yu, Changjian Wang, Yulan Guo, Yuxing Peng, and Kai Xu

Introduction

We study the problem of extracting accurate correspondences for point cloud registration. Recent keypoint-free methods bypass the detection of repeatable keypoints which is difficult in low-overlap scenarios, showing great potential in registration. They seek correspondences over downsampled superpoints, which are then propagated to dense points. Superpoints are matched based on whether their neighboring patches overlap. Such sparse and loose matching requires contextual features capturing the geometric structure of the point clouds. We propose Geometric Transformer to learn geometric feature for robust superpoint matching. It encodes pair-wise distances and triplet-wise angles, making it robust in low-overlap cases and invariant to rigid transformation. The simplistic design attains surprisingly high matching accuracy such that no RANSAC is required in the estimation of alignment transformation, leading to $100$ times acceleration. Our method improves the inlier ratio by $17% \sim 30%$ and the registration recall by over $7%$ on the challenging 3DLoMatch benchmark. Code will be released for paper reproduction.

News

Installation

Please use the following command for installation.

# It is recommended to create a new environment
conda create -n geotransformer python==3.8
conda activate geotransformer

# [Optional] If you are using CUDA 11.0 or newer, please install `torch==1.7.1+cu110`
pip install torch==1.7.1+cu110 -f https://download.pytorch.org/whl/torch_stable.html

# Install packages and other dependencies
python setup.py build develop

# Compile c++ wrappers
cd geotransformer/cpp_wrappers
sh ./compile_wrappers.sh

Code has been tested with Ubuntu 20.04, GCC 9.3.0, Python 3.8, PyTorch 1.7.1, CUDA 11.1 and cuDNN 8.1.0.

Data preparation

We provide code for training and testing on 3DMatch.

The dataset can be download from PREDATOR. The data should be organized as follows:

--data--3DMatch--metadata
              |--data--train--7-scenes-chess--cloud_bin_0.pth
                    |      |               |--...
                    |      |--...
                    |--test--7-scenes-redkitchen--cloud_bin_0.pth
                          |                    |--...
                          |--...

Training

The code for GeoTransformer is in experiments/geotransformer.3dmatch. Use the following command for training.

CUDA_VISIBLE_DEVICES=0 python trainval.py
# use "--snapshot=path/to/snapshot" to resume training.

Testing

Use the following command for testing.

# 3DMatch
CUDA_VISIBLE_DEVICES=0 ./eval.sh EPOCH 3DMatch
# 3DLoMatch
CUDA_VISIBLE_DEVICES=0 ./eval.sh EPOCH 3DLoMatch

EPOCH is the epoch id.

We also provide pretrained weights in weights, use the following command to test the pretrained weights.

CUDA_VISIBLE_DEVICES=0 python test.py --snapshot=../../weights/geotransformer-3dmatch.pth.tar --benchmark=3DMatch
CUDA_VISIBLE_DEVICES=0 python eval.py --run_matching --run_registration --benchmark=3DMatch

Replace 3DMatch with 3DLoMatch to evaluate on 3DLoMatch.

Results

Benchmark FMR IR RR
3DMatch 97.7 70.3 91.5
3DLoMatch 88.1 43.3 74.0
Comments
  • Test and visualize on two given point clouds

    Test and visualize on two given point clouds

    Hi, thanks for sharing your work. Is it easy to give some guidance or do you have any tutorial to test/evaluate the pre-trained models on two individual given point clouds?

    opened by ttsesm 10
  • Question about kitti result reproduce.

    Question about kitti result reproduce.

    Hi ,thanks for your amazing work.

    However , I have met a problem of kitti dataset result reproduce. I followed your code to train and test on kitti dataset , but my result seems always lower than the result in your paper. I can achieve Registration recall of 99.8 and RTE 6.8 same as yours ,but my result of RRE is about 0.33 which is much higher than 0.24 in your paper.

    Can you give me some idea of how to reproduce the same result ?

    opened by vision507 9
  • How to draw the point cloud like this ?

    How to draw the point cloud like this ?

    Hi , thanks for your great work. And I would like to know how can I get this kind of visualization result in your paper. 屏幕截图 2022-08-04 081115 When I use open3d to draw the point clouds , it seems very ugly, so I'm wondering how can I get your visualization result. It would be very grateful.

    opened by vision507 9
  • How to group data sets after downloading

    How to group data sets after downloading

    Hello, author: I ran the python files in the Kitti dataset and ModelNet dataset folders according to your requirements, but they did not work and were not classified into the ones you described. Is there a specific method? thank you!

    opened by yangsihai97 9
  • Some questions about downsampling of point clouds

    Some questions about downsampling of point clouds

    Hi there,

    Really appreciate releasing your amazing work! I'm quite new to point clouds and I have some questions regarding downsampling of the point clouds.

    Take 3DMatch as an example. The original raw dataset downloaded from PREDATOR is quite dense, and is preprocessed so that each grid size is 2.5cm. In your work, you further subsample the point clouds using KPConv (the first level downsampled points). My first question is what's the purpose of subsampling? Would a denser point cloud be more informative for registration?

    Also, is there a specific reason you downsample the points further for dense point correspondences as I believe the preprocessed input data is already of density 2.5cm?

    Last, is there a quick way to visualize the registration result for the 3DMatch benchmark? By running eval.py and test.py it seems to only produce the quantitative metrics.

    Thanks for your patience and reply in advance!

    opened by jinhong-ni 5
  • 3DMatch dataset

    3DMatch dataset

    Hi,

    I have a question regarding the D.2. section from the Supplementary material.

    Could you please share which 3DMatch files you use for the evaluation?

    • Do you use those with overlap > 30%?
    • Do you use consecutive pairs of point cloud indices (the original 3DMatch filters the pairs (i,j) with i+1 < j)?

    You mention that you use the files from PREDATOR, but PREDATOR has more pairs than the original 3DMatch dataset. Can you please elaborate?

    Thank you in advance for you response. Hopefully we can clear all this confusion of the registration pairs used in the datasets. Every paper seems to do it a bit differently.

    opened by DavidBoja 4
  • why need gt transoform in demo.py

    why need gt transoform in demo.py

    in line 74 in experiments/geotransformer.3dmatch.stage4.gse.k3.max.oacl.stage2.sinkhorn/model.py , transform = data_dict['transform'].detach(), and in line 110 you use gt transform in get_node_correspondences. however If i don't know how the transform matrix of src to ref, how can I use demo.py

    opened by MaybeOjbk 4
  • The different RR in test.py and eval.py

    The different RR in test.py and eval.py

    Hello, thank you for you amazing work again. When I evaluated on 3DMatch, I found the RR reported by test.py and eval.py are very different, what makes this difference?

    opened by Hui-design 4
  • Question about radius search

    Question about radius search

    First of all, thanks for your excellent work! Radius search in KPConv is quiet slow for large scale point clouds and it usually takes seconds in my own experiments. Is there a more efficient way for radius search, such as a version that can run on GPU. Hope to get your answer!

    opened by Fleurrr 4
  • Questions about RANSAC

    Questions about RANSAC

    Dear authors,

    Thanks for open-sourcing your amazing work. I'm experimenting with RANSAC and I noticed some unexpected results compared to the paper.

    What I've done is first change the number of iterations in RANSAC to 50k. The evaluation runs extremely slowly (approx. 16 hrs in contrast to about 1 hr for local to global), and I only got performance similar to local to global. I also tried to change the number of points in RANSAC to a larger number (by default it is set to 3), such as 250, 500, 1000, etc. I did not achieve a significant gain in performance, even the recall collapses to zero if the number of points is set too big (like 5k). This is contradicting to what's reported in the paper, as the performance is reported to keep improving as the number of samples increases.

    I would be super grateful if you could guide me through what I've done wrong here. Thanks in advance for your patience and help!

    opened by jinhong-ni 4
  • RuntimeError: Error compiling objects for extension

    RuntimeError: Error compiling objects for extension

    Hello, sir. The question is about creating the code runtime environment. After reinstalling CUDA and downloading torch, I executed 'python setup.py build develop' and it still report the error, 'RuntimeError: Error compiling objects for extension'.

    Traceback (most recent call last): File "setup.py", line 5, in setup( File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_init_.py", line 87, in setup return distutils.core.setup(**attrs) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\core.py", line 148, in setup return run_commands(dist) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\core.py", line 163, in run_commands dist.run_commands() File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\dist.py", line 967, in run_commands self.run_command(cmd) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools\dist.py", line 1214, in run_command super().run_command(command) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\dist.py", line 986, in run_command cmd_obj.run() File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\command\build.py", line 135, in run self.run_command(cmd_name) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\cmd.py", line 313, in run_command self.distribution.run_command(command) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools\dist.py", line 1214, in run_command super().run_command(command) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\dist.py", line 986, in run_command cmd_obj.run() File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools\command\build_ext.py", line 79, in run _build_ext.run(self) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\command\build_ext.py", line 339, in run self.build_extensions() File "F:\Anaconda3\envs\geotransformer\lib\site-packages\torch\utils\cpp_extension.py", line 670, in build_extensions build_ext.build_extensions(self) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\command\build_ext.py", line 448, in build_extensions self._build_extensions_serial() File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\command\build_ext.py", line 473, in _build_extensions_serial self.build_extension(ext) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools\command\build_ext.py", line 202, in build_extension _build_ext.build_extension(self, ext) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\command\build_ext.py", line 528, in build_extension objects = self.compiler.compile(sources, File "F:\Anaconda3\envs\geotransformer\lib\site-packages\torch\utils\cpp_extension.py", line 643, in win_wrap_ninja_compile _write_ninja_file_and_compile_objects( File "F:\Anaconda3\envs\geotransformer\lib\site-packages\torch\utils\cpp_extension.py", line 1250, in _write_ninja_file_and_compile_objects _run_ninja_build( File "F:\Anaconda3\envs\geotransformer\lib\site-packages\torch\utils\cpp_extension.py", line 1555, in _run_ninja_build raise RuntimeError(message) from e RuntimeError: Error compiling objects for extension

    Is this because I use win10? Or other issues caused the error?

    opened by ChaunceyQ 4
  • The training model is not effective

    The training model is not effective

    Thank you for your excellent work. When I use the LGR training model without changing the code, the result on 3DMatch (about 91.0%) was still quite different from the model you provided (92.5%)

    opened by awdx551 3
Owner
Zheng Qin
computer vision, deep learning
Zheng Qin
This is the official Pytorch implementation of "Lung Segmentation from Chest X-rays using Variational Data Imputation", Raghavendra Selvan et al. 2020

README This is the official Pytorch implementation of "Lung Segmentation from Chest X-rays using Variational Data Imputation", Raghavendra Selvan et a

Raghav 42 Dec 15, 2022
Rethinking of Pedestrian Attribute Recognition: A Reliable Evaluation under Zero-Shot Pedestrian Identity Setting

Pytorch Pedestrian Attribute Recognition: A strong PyTorch baseline of pedestrian attribute recognition and multi-label classification.

Jian 79 Dec 18, 2022
Notebooks for my "Deep Learning with TensorFlow 2 and Keras" course

Deep Learning with TensorFlow 2 and Keras – Notebooks This project accompanies my Deep Learning with TensorFlow 2 and Keras trainings. It contains the

Aurélien Geron 1.9k Dec 15, 2022
PyTorch implementation of Federated Learning with Non-IID Data, and federated learning algorithms, including FedAvg, FedProx.

Federated Learning with Non-IID Data This is an implementation of the following paper: Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, Vik

Youngjoon Lee 48 Dec 29, 2022
An algorithm study of the 6th iOS 10 set of Boost Camp Web Mobile

알고리즘 스터디 🔥 부스트캠프 웹모바일 6기 iOS 10조의 알고리즘 스터디 입니다. 개인적인 사정 등으로 S034, S055만 참가하였습니다. 스터디 목적 상진: 코테 합격 + 부캠끝나고 아침에 일어나기 위해 필요한 사이클 기완: 꾸준하게 자리에 앉아 공부하기 +

2 Jan 11, 2022
Alphabetical Letter Recognition

BayeesNetworks-Image-Classification Alphabetical Letter Recognition In these demo we are using "Bayees Networks" Our database is composed by Learning

Mohammed Firass 4 Nov 30, 2021
🕹️ Official Implementation of Conditional Motion In-betweening (CMIB) 🏃

Conditional Motion In-Betweening (CMIB) Official implementation of paper: Conditional Motion In-betweeening. Paper(arXiv) | Project Page | YouTube in-

Jihoon Kim 81 Dec 22, 2022
disentanglement_lib is an open-source library for research on learning disentangled representations.

disentanglement_lib disentanglement_lib is an open-source library for research on learning disentangled representation. It supports a variety of diffe

Google Research 1.3k Dec 28, 2022
IMBENS: class-imbalanced ensemble learning in Python.

IMBENS: class-imbalanced ensemble learning in Python. Links: [Documentation] [Gallery] [PyPI] [Changelog] [Source] [Download] [知乎/Zhihu] [中文README] [a

Zhining Liu 176 Jan 04, 2023
Replication Code for "Self-Supervised Bug Detection and Repair" NeurIPS 2021

Self-Supervised Bug Detection and Repair This is the reference code to replicate the research in Self-Supervised Bug Detection and Repair in NeurIPS 2

Microsoft 85 Dec 24, 2022
Projecting interval uncertainty through the discrete Fourier transform

Projecting interval uncertainty through the discrete Fourier transform This repo

1 Mar 02, 2022
Repository for Driving Style Recognition algorithms for Autonomous Vehicles

Driving Style Recognition Using Interval Type-2 Fuzzy Inference System and Multiple Experts Decision Making Created by Iago Pachêco Gomes at USP - ICM

Iago Gomes 9 Nov 28, 2022
Test-Time Personalization with a Transformer for Human Pose Estimation, NeurIPS 2021

Transforming Self-Supervision in Test Time for Personalizing Human Pose Estimation This is an official implementation of the NeurIPS 2021 paper: Trans

41 Nov 28, 2022
This is code of book "Learn Deep Learning with PyTorch"

深度学习入门之PyTorch Learn Deep Learning with PyTorch 非常感谢您能够购买此书,这个github repository包含有深度学习入门之PyTorch的实例代码。由于本人水平有限,在写此书的时候参考了一些网上的资料,在这里对他们表示敬意。由于深度学习的技术在

Xingyu Liao 2.5k Jan 04, 2023
Non-Imaging Transient Reconstruction And TEmporal Search (NITRATES)

Non-Imaging Transient Reconstruction And TEmporal Search (NITRATES) This repo contains the full NITRATES pipeline for maximum likelihood-driven discov

13 Nov 08, 2022
Official code for the paper: Deep Graph Matching under Quadratic Constraint (CVPR 2021)

QC-DGM This is the official PyTorch implementation and models for our CVPR 2021 paper: Deep Graph Matching under Quadratic Constraint. It also contain

Quankai Gao 55 Nov 14, 2022
Drone detection using YOLOv5

This drone detection system uses YOLOv5 which is a family of object detection architectures and we have trained the model on Drone Dataset. Overview I

Tushar Sarkar 27 Dec 20, 2022
Numerical-computing-is-fun - Learning numerical computing with notebooks for all ages.

As much as this series is to educate aspiring computer programmers and data scientists of all ages and all backgrounds, it is also a reminder to mysel

EKA foundation 758 Dec 25, 2022
High performance distributed framework for training deep learning recommendation models based on PyTorch.

High performance distributed framework for training deep learning recommendation models based on PyTorch.

340 Dec 30, 2022
PyTorch-lightning implementation of the ESFW module proposed in our paper Edge-Selective Feature Weaving for Point Cloud Matching

Edge-Selective Feature Weaving for Point Cloud Matching This repository contains a PyTorch-lightning implementation of the ESFW module proposed in our

5 Feb 14, 2022