Self-supervised Deep LiDAR Odometry for Robotic Applications

Overview

DeLORA: Self-supervised Deep LiDAR Odometry for Robotic Applications

Overview

This is the corresponding code to the above paper ("Self-supervised Learning of LiDAR Odometry for Robotic Applications") which is published at the International Conference on Robotics and Automation (ICRA) 2021. The code is provided by the Robotics Systems Lab at ETH Zurich, Switzerland.

** Authors:** Julian Nubert ([email protected]) , Shehryar Khattak , Marco Hutter

title_img

Copyright IEEE

Python Setup

We provide a conda environment for running our code.

Conda

The conda environment is very comfortable to use in combination with PyTorch because only NVidia drivers are needed. The Installation of suitable CUDA and CUDNN libraries is all handle by Conda.

  • Install conda: link
  • To set up the conda environment run the following command:
conda env create -f conda/DeLORA-py3.9.yml

This installs an environment including GPU-enabled PyTorch, including any needed CUDA and cuDNN dependencies.

  • Activate the environment:
conda activate DeLORA-py3.9
  • Install the package to set all paths correctly:
pip3 install -e .

ROS Setup

For running ROS code in the ./src/ros_utils/ folder you need to have ROS installed (link). We recommend Ubuntu 20.04 and ROS Noetic due to its native Python3 support. For performing inference in Python2.7, convert your PyTorch model with ./scripts/convert_pytorch_models.py and run an older PyTorch version (<1.3).

ros-numpy

In any case you need to install ros-numpy if you want to make use of the provided rosnode:

sudo apt install ros-<distro>-ros-numpy

Datasets and Preprocessing

Instructions on how to use and preprocess the datasets can be found in the ./datasets/ folder. We provide scripts for doing the preprocessing for:

  1. general rosbags containing LiDAR scans,
  2. and for the KITTI dataset in its own format.

Example: KITTI Dataset

LiDAR Scans

Download the "velodyne laster data" from the official KITTI odometry evaluation ( 80GB): link. Put it to <delora_ws>/datasets/kitti, where kitti contains /data_odometry_velodyne/dataset/sequences/00..21.

Groundtruth poses

Please also download the groundtruth poses here. Make sure that the files are located at <delora_ws>/datasets/kitti, where kitti contains /data_odometry_poses/dataset/poses/00..10.txt.

Preprocessing

In the file ./config/deployment_options.yaml make sure to set datasets: ["kitti"]. Then run

preprocess_data.py

Custom Dataset

If you want to add an own dataset please add its sensor specifications to ./config/config_datasets.yaml and ./config/config_datasets_preprocessing.yaml. Information that needs to be added is the dataset name, its sequences and its sensor specifications such as vertical field of view and number of rings.

Deploy

After preprocessing, for each dataset we assume the following hierarchical structure: dataset_name/sequence/scan (see previous dataset example). Our code natively supports training and/or testing on various datasets with various sequences at the same time.

Training

Run the training with the following command:

run_training.py

The training will be executed for the dataset(s) specified in ./config/deployment_options.yaml. You will be prompted to enter a name for this training run, which will be used for reference in the MLFlow logging.

Custom Settings

For custom settings and hyper-parameters please have a look in ./config/.

By default loading from RAM is disabled. If you have enough memory, enable it in ./config/deployment_options.yaml. When loading from disk, the first few iterations are sometimes slow due to I/O, but it should accelerate quite quickly. For storing the KITTI training set entirely in memory, roughly 50GB of RAM are required.

Continuing Training

For continuing training provide the --checkpoint flag with a path to the model checkpoint to the script above.

Visualizing progress and results

For visualizing progress we use MLFlow. It allows for simple logging of parameters, metrics, images, and artifacts. Artifacts could e.g. also be whole TensorBoard logfiles. To visualize the training progress execute (from DeLORA folder):

mlflow ui 

The MLFlow can then be visualized in your browser following the link in the terminal.

Testing

Testing can be run along the line:

run_testing.py --checkpoint <path_to_checkpoint>

The checkpoint can be found in MLFlow after training. It runs testing for the dataset specified in ./config/deployment_options.yaml.

We provide an exemplary trained model in ./checkpoints/kitti_example.pth.

ROS-Node

This ROS-node takes the pretrained model at location <model_location> and performs inference; i.e. it predicts and publishes the relative transformation between incoming point cloud scans. The variable <dataset> should contain the name of the dataset in the config files, e.g. kitti, in order to load the corresponding parameters. Topic and frame names can be specified in the following way:

run_rosnode.py --checkpoint <model_location> --dataset <dataset> --lidar_topic=<name_of_lidar_topic> --lidar_frame=<name_of_lidar_frame>

The resulting odometry will be published as a nav_msgs.msg.Odometry message under the topic /delora/odometry .

Example: DARPA Dataset

For the darpa dataset this could look as follows:

run_rosnode.py --checkpoint ~/Downloads/checkpoint_epoch_0.pth --dataset darpa --lidar_topic "/sherman/lidar_points" --lidar_frame sherman/ouster_link

Comfort Functions

Additional functionalities are provided in ./bin/ and ./scripts/.

Visualization of Normals (mainly for debugging)

Located in ./bin/, see the readme-file ./dataset/README.md for more information.

Creation of Rosbags for KITTI Dataset

After starting a roscore, conversion from KITTI dataset format to a rosbag can be done using the following command:

python scripts/convert_kitti_to_rosbag.py

The point cloud scans will be contained in the topic "/velodyne_points", located in the frame velodyne. E.g. for the created rosbag, our provided rosnode can be run using the following command:

run_rosnode.py --checkpoint ~/Downloads/checkpoint_epoch_30.pth --lidar_topic "/velodyne_points" --lidar_frame "velodyne"

Convert PyTorch Model to older PyTorch Compatibility

Converion of the new model <path_to_model>/model.pth to old (compatible with < PyTorch1.3) <path_to_model>/model_py27.pth can be done with the following:

python scripts/convert_pytorch_models.py --checkpoint <path_to_model>/model

Note that there is no .pth ending in the script.

Time The Network

The execution time of the network can be timed using:

python scripts/time_network.py

Paper

Thank you for citing DeLORA (ICRA-2021) if you use any of this code.

@inproceedings{nubert2021self,
  title={Self-supervised Learning of LiDAR Odometry for Robotic Applications},
  author={Nubert, Julian and Khattak, Shehryar and Hutter, Marco},
  booktitle={IEEE International Conference on Robotics and Automation (ICRA)},
  year={2021},
  organization={IEEE}
}

Dependencies

Dependencies are specified in ./conda/DeLORA-py3.9.yml and ./pip/requirements.txt.

Tuning

If the result does not achieve the desired performance, please have a look at the normal estimation, since the loss is usually dominated by the plane-to-plane loss, which is impacted by noisy normal estimates. For the results presented in the paper we picked some reasonable parameters without further fine-tuning, but we are convinced that less noisy normal estimates would lead to an even better convergence.

Owner
Robotic Systems Lab - Legged Robotics at ETH Zürich
The Robotic Systems Lab investigates the development of machines and their intelligence to operate in rough and challenging environments.
Robotic Systems Lab - Legged Robotics at ETH Zürich
[ArXiv 2021] Data-Efficient Instance Generation from Instance Discrimination

InsGen - Data-Efficient Instance Generation from Instance Discrimination Data-Efficient Instance Generation from Instance Discrimination Ceyuan Yang,

GenForce: May Generative Force Be with You 93 Dec 25, 2022
An LSTM based GAN for Human motion synthesis

GAN-motion-Prediction An LSTM based GAN for motion synthesis has a few issues reading H3.6M data from A.Jain et al , will fix soon. Prediction of the

Amogh Adishesha 9 Jun 17, 2022
A Parameter-free Deep Embedded Clustering Method for Single-cell RNA-seq Data

A Parameter-free Deep Embedded Clustering Method for Single-cell RNA-seq Data Overview Clustering analysis is widely utilized in single-cell RNA-seque

AI-Biomed @NSCC-gz 3 May 08, 2022
Tensorflow implementation of soft-attention mechanism for video caption generation.

SA-tensorflow Tensorflow implementation of soft-attention mechanism for video caption generation. An example of soft-attention mechanism. The attentio

Paul Chen 153 Nov 14, 2022
Art Project "Schrödinger's Game of Life"

Repo of the project "Team Creative Quantum AI: Schrödinger's Game of Life" Installation new conda env: conda create --name qcml python=3.8 conda activ

ℍ◮ℕℕ◭ℍ ℝ∈ᛔ∈ℝ 2 Sep 15, 2022
Implementation of CSRL from the AAAI2022 paper: Constraint Sampling Reinforcement Learning: Incorporating Expertise For Faster Learning

CSRL Implementation of CSRL from the AAAI2022 paper: Constraint Sampling Reinforcement Learning: Incorporating Expertise For Faster Learning Python: 3

4 Apr 14, 2022
TorchFlare is a simple, beginner-friendly, and easy-to-use PyTorch Framework train your models effortlessly.

TorchFlare TorchFlare is a simple, beginner-friendly and an easy-to-use PyTorch Framework train your models without much effort. It provides an almost

Atharva Phatak 85 Dec 26, 2022
Gym environment for FLIPIT: The Game of "Stealthy Takeover"

gym-flipit Gym environment for FLIPIT: The Game of "Stealthy Takeover" invented by Marten van Dijk, Ari Juels, Alina Oprea, and Ronald L. Rivest. Desi

Lisa Oakley 2 Dec 15, 2021
Restricted Boltzmann Machines in Python.

How to Use First, initialize an RBM with the desired number of visible and hidden units. rbm = RBM(num_visible = 6, num_hidden = 2) Next, train the m

Edwin Chen 928 Dec 30, 2022
TuckER: Tensor Factorization for Knowledge Graph Completion

TuckER: Tensor Factorization for Knowledge Graph Completion This codebase contains PyTorch implementation of the paper: TuckER: Tensor Factorization f

Ivana Balazevic 296 Dec 06, 2022
The project is an official implementation of our CVPR2019 paper "Deep High-Resolution Representation Learning for Human Pose Estimation"

Deep High-Resolution Representation Learning for Human Pose Estimation (CVPR 2019) News [2020/07/05] A very nice blog from Towards Data Science introd

Leo Xiao 3.9k Jan 05, 2023
Captcha-tensorflow - Image Captcha Solving Using TensorFlow and CNN Model. Accuracy 90%+

Captcha Solving Using TensorFlow Introduction Solve captcha using TensorFlow. Learn CNN and TensorFlow by a practical project. Follow the steps, run t

Jackon Yang 869 Jan 06, 2023
CausaLM: Causal Model Explanation Through Counterfactual Language Models

CausaLM: Causal Model Explanation Through Counterfactual Language Models Authors: Amir Feder, Nadav Oved, Uri Shalit, Roi Reichart Abstract: Understan

Amir Feder 39 Jul 10, 2022
we propose EfficientDerain for high-efficiency single-image deraining

EfficientDerain we propose EfficientDerain for high-efficiency single-image deraining Requirements python 3.6 pytorch 1.6.0 opencv-python 4.4.0.44 sci

Qing Guo 126 Dec 07, 2022
Hippocampal segmentation using the UNet network for each axis

Hipposeg Hippocampal segmentation using the UNet network for each axis, inspired by https://github.com/MICLab-Unicamp/e2dhipseg Red: False Positive Gr

Juan Carlos Aguirre Arango 0 Sep 02, 2021
Anonymous implementation of KSL

k-Step Latent (KSL) Implementation of k-Step Latent (KSL) in PyTorch. Representation Learning for Data-Efficient Reinforcement Learning [Paper] Code i

1 Nov 10, 2021
[TIP 2021] SADRNet: Self-Aligned Dual Face Regression Networks for Robust 3D Dense Face Alignment and Reconstruction

SADRNet Paper link: SADRNet: Self-Aligned Dual Face Regression Networks for Robust 3D Dense Face Alignment and Reconstruction Requirements python

Multimedia Computing Group, Nanjing University 99 Dec 30, 2022
A python tutorial on bayesian modeling techniques (PyMC3)

Bayesian Modelling in Python Welcome to "Bayesian Modelling in Python" - a tutorial for those interested in learning how to apply bayesian modelling t

Mark Regan 2.4k Jan 06, 2023
Capstone-Project-2 - A game program written in the Python language

Capstone-Project-2 My Pygame Game Information: Description This Pygame project i

Nhlakanipho Khulekani Hlophe 1 Jan 04, 2022
Deep Learning for humans

Keras: Deep Learning for Python Under Construction In the near future, this repository will be used once again for developing the Keras codebase. For

Keras 57k Jan 09, 2023