We utilize deep reinforcement learning to obtain favorable trajectories for visual-inertial system calibration.

Overview

Unified Data Collection for Visual-Inertial Calibration via Deep Reinforcement Learning

Update: The lastest code will be updated in this branch. Please switch to CORL2020 branch if you are looking for the Model-based Heuristic Deep RL approach.

Developed by Le Chen and Yunke Ao from Autonomous Systems Lab (ASL) at ETH Zurich.

1 Introduction

In this work we presents a novel formulation to learn a motion policy to be executed on a robot arm for automatic data collection for calibrating intrinsics and extrinsics jointly. Our approach models the calibration process compactly using model-free deep reinforcement learning to derive a policy that guides the motions of a robotic arm holding the sensor to efficiently collect measurements that can be used for both camera intrinsic calibration and camera-IMU extrinsic calibration. Given the current pose and collected measurements, the learned policy generates the subsequent transformation that optimizes sensor calibration accuracy. The evaluations in simulation and on a real robotic system show that our learned policy generates favorable motion trajectories and collects enough measurements efficiently that yield the desired intrinsics and extrinsics with short path lengths. In simulation we are able to perform calibrations $10\times$ faster than hand-crafted policies, which transfers to a real-world speed up of $3\times$ over a human expert.

2 Usage

Our code is tested on Ubuntu 18.04 LTS (Bionic Beaver) and ROS Melodic Morenia with GPU GTX 1660 Ti and CUDA 11.2.

2.1 Build Instructions

  • Install required dependencies:
sudo apt-get install build-essential software-properties-common
sudo apt-get install bc curl ca-certificates fakeroot gnupg2 libssl-dev lsb-release libelf-dev bison flex
sudo apt-get install ros-melodic-moveit, ros-melodic-moveit-visual-tools, ros-melodic-cmake-modules
sudo apt-get install ros-melodic-libfranka ros-melodic-franka-ros, ros-melodic-joint-trajectory-controller
sudo apt-get install ros-melodic-vision-opencv ros-melodic-image-transport-plugins
sudo apt-get install python-setuptools python-rosinstall ipython libeigen3-dev libboost-all-dev doxygen
sudo apt-get install libopencv-dev libgtk-3-dev python-catkin-tools
sudo apt-get install python-matplotlib python-scipy python-git python-pip ipython
sudo apt-get install libtbb-dev libblas-dev liblapack-dev libv4l-dev, libpoco-dev

pip install opencv-python
pip install opencv-contrib-python
pip install --upgrade tensorflow
pip install python-igraph --upgrade
pip install pyyaml
pip install rospkg
pip install matplotlib
pip install pandas
pip install pytorch
pip install wandb
pip install PyKDL
pip install gym
  • Clone the repository and catkin build:
cd ~/catkin_ws
git clone https://github.com/clthegoat/Learn-to-Calibrate.git
cd Learn-to-Calibrate
git checkout master
cd ../
mv Learn-to-Calibrate src
catkin build
source ~/catkin_ws/devel/setup.bash

2.2 Configuration

  • Please change the file saving directory in franka_cal_sim_single/config/config.yaml before training or testing!

2.3 Running the code

2.3.1 Training:

  • In terminal 1:
source ~/catkin_ws/devel/setup.bash
roslaunch franka_cal_sim_single cam_imu_ext_che.launch
  • In terminal 2:
source ~/catkin_ws/devel/setup.bash
cd src/franka_cal_sim/python/algorithms
python RL_algo_sac_int_ext.py

2.3.2 Testing:

  • In terminal 1:
source ~/catkin_ws/devel/setup.bash
roslaunch franka_cal_sim_single cam_imu_ext_che.launch
  • In terminal 2:
source ~/catkin_ws/devel/setup.bash
cd src/franka_cal_sim/python/test_policies/
python RL_algo_sac_ext_int_test.py

3 Citing

Please cite the following paper when using our code for your research:

@article{chen2020learning,
  title={Learning Trajectories for Visual-Inertial System Calibration via Model-based Heuristic Deep Reinforcement Learning},
  author={Chen, Le and Ao, Yunke and Tschopp, Florian and Cramariuc, Andrei and Breyer, Michel and Chung, Jen Jen and Siegwart, Roland and Cadena, Cesar},
  journal={arXiv preprint arXiv:2011.02574},
  year={2020}
}

4 Code reference:

Our code is based on the following repositories:

Owner
ETHZ ASL
ETHZ ASL
Implementation of ViViT: A Video Vision Transformer

ViViT: A Video Vision Transformer Unofficial implementation of ViViT: A Video Vision Transformer. Notes: This is in WIP. Model 2 is implemented, Model

Rishikesh (ऋषिकेश) 297 Jan 06, 2023
Build fully-functioning computer vision models with PyTorch

Detecto is a Python package that allows you to build fully-functioning computer vision and object detection models with just 5 lines of code. Inferenc

Alan Bi 576 Dec 29, 2022
BT-Unet: A-Self-supervised-learning-framework-for-biomedical-image-segmentation-using-Barlow-Twins

BT-Unet: A-Self-supervised-learning-framework-for-biomedical-image-segmentation-using-Barlow-Twins Deep learning has brought most profound contributio

Narinder Singh Punn 12 Dec 04, 2022
EfficientNetv2 TensorRT int8

EfficientNetv2_TensorRT_int8 EfficientNetv2模型实现来自https://github.com/d-li14/efficientnetv2.pytorch 环境配置 ubuntu:18.04 cuda:11.0 cudnn:8.0 tensorrt:7

34 Apr 24, 2022
Tree Nested PyTorch Tensor Lib

DI-treetensor treetensor is a generalized tree-based tensor structure mainly developed by OpenDILab Contributors. Almost all the operation can be supp

OpenDILab 167 Dec 29, 2022
Existing Literature about Machine Unlearning

Machine Unlearning Papers 2021 Brophy and Lowd. Machine Unlearning for Random Forests. In ICML 2021. Bourtoule et al. Machine Unlearning. In IEEE Symp

Jonathan Brophy 213 Jan 08, 2023
DEMix Layers for Modular Language Modeling

DEMix This repository contains modeling utilities for "DEMix Layers: Disentangling Domains for Modular Language Modeling" (Gururangan et. al, 2021). T

Suchin 43 Nov 11, 2022
Keras udrl - Keras implementation of Upside Down Reinforcement Learning

keras_udrl Keras implementation of Upside Down Reinforcement Learning This is me

Eder Santana 7 Jan 24, 2022
Unofficial pytorch implementation of 'Image Inpainting for Irregular Holes Using Partial Convolutions'

pytorch-inpainting-with-partial-conv Official implementation is released by the authors. Note that this is an ongoing re-implementation and I cannot f

Naoto Inoue 525 Jan 01, 2023
PyTorch code for our paper "Image Super-Resolution with Non-Local Sparse Attention" (CVPR2021).

Image Super-Resolution with Non-Local Sparse Attention This repository is for NLSN introduced in the following paper "Image Super-Resolution with Non-

143 Dec 28, 2022
A Light in the Dark: Deep Learning Practices for Industrial Computer Vision

A Light in the Dark: Deep Learning Practices for Industrial Computer Vision This is the repository for our Paper/Contribution to the WI2022 in Nürnber

Maximilian Harl 6 Jan 17, 2022
VIsually-Pivoted Audio and(N) Text

VIP-ANT: VIsually-Pivoted Audio and(N) Text Code for the paper Connecting the Dots between Audio and Text without Parallel Data through Visual Knowled

Yän.PnG 16 Nov 04, 2022
Info and sample codes for "NTU RGB+D Action Recognition Dataset"

"NTU RGB+D" Action Recognition Dataset "NTU RGB+D 120" Action Recognition Dataset "NTU RGB+D" is a large-scale dataset for human action recognition. I

Amir Shahroudy 578 Dec 30, 2022
Official Pytorch implementation of paper "Reverse Engineering of Generative Models: Inferring Model Hyperparameters from Generated Images"

Reverse_Engineering_GMs Official Pytorch implementation of paper "Reverse Engineering of Generative Models: Inferring Model Hyperparameters from Gener

100 Dec 18, 2022
CLIP (Contrastive Language–Image Pre-training) for Italian

Italian CLIP CLIP (Radford et al., 2021) is a multimodal model that can learn to represent images and text jointly in the same space. In this project,

Italian CLIP 114 Dec 29, 2022
Wide Residual Networks (WideResNets) in PyTorch

Wide Residual Networks (WideResNets) in PyTorch WideResNets for CIFAR10/100 implemented in PyTorch. This implementation requires less GPU memory than

Jason Kuen 296 Dec 27, 2022
Circuit Training: An open-source framework for generating chip floor plans with distributed deep reinforcement learning

Circuit Training: An open-source framework for generating chip floor plans with distributed deep reinforcement learning. Circuit Training is an open-s

Google Research 479 Dec 25, 2022
Scene-Text-Detection-and-Recognition (Pytorch)

Scene-Text-Detection-and-Recognition (Pytorch) Competition URL: https://tbrain.t

Gi-Luen Huang 9 Jan 02, 2023
Code for HodgeNet: Learning Spectral Geometry on Triangle Meshes, in SIGGRAPH 2021.

HodgeNet | Webpage | Paper | Video HodgeNet: Learning Spectral Geometry on Triangle Meshes Dmitriy Smirnov, Justin Solomon SIGGRAPH 2021 Set-up To ins

Dima Smirnov 61 Nov 27, 2022
Pytorch implementation of Integrating Tree Path in Transformer for Code Representation

This is an official Pytorch implementation of the approaches proposed in: Han Peng, Ge Li, Wenhan Wang, Yunfei Zhao, Zhi Jin “Integrating Tree Path in

Han Peng 16 Dec 23, 2022