Pip-package for trajectory benchmarking from "Be your own Benchmark: No-Reference Trajectory Metric on Registered Point Clouds", ECMR'21

Related tags

Deep Learningmap-metrics
Overview

Map Metrics for Trajectory Quality

Map metrics toolkit provides a set of metrics to quantitatively evaluate trajectory quality via estimating consistency of the map aggregated from point clouds.

GPS or Motion Capture systems are not always available in perception systems, or their quality is not enough (GPS on small-scale distances) for use as ground truth trajectory. Thus, common full-reference trajectory metrics (APE, RPE, and their modifications) could not be applied to evaluate trajectory quality. When 3D sensing technologies (depth camera, LiDAR) are available on the perception system, one can alternatively assess trajectory quality --- estimate the consistency of the map from registered point clouds via the trajectory.

Documentation: https://map-metrics.readthedocs.io.

Documentation Status Updates

Features

Our toolkit provides implementation of the next metrics:

  • Mean Map Entropy (MME), Mean Plane Variance(MPV) [1] [2]
  • Mutually Orthogonal Metric (MOM) [3] -- has strong correlation with RPE

Citation

If you use this toolkit or MOM-metric results, please, cite our work:

@misc{kornilova2021benchmark,
    title={Be your own Benchmark: No-Reference Trajectory Metric on Registered Point Clouds},
    author={Anastasiia Kornilova and Gonzalo Ferrer},
    year={2021},
    eprint={2106.11351},
    archivePrefix={arXiv},
    primaryClass={cs.RO}
}

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

Links

[1] Droeschel, David, Jörg Stückler, and Sven Behnke. "Local multi-resolution representation for 6D motion estimation and mapping with a continuously rotating 3D laser scanner." 2014 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2014.
[2] Razlaw, Jan, et al. "Evaluation of registration methods for sparse 3D laser scans." 2015 European Conference on Mobile Robots (ECMR). IEEE, 2015.
[3] Kornilova, Anastasiia, and Gonzalo Ferrer. "Be your own Benchmark: No-Reference Trajectory Metric on Registered Point Clouds." arXiv preprint arXiv:2106.11351 (2021).
Comments
  • Cross-Platform Wheels

    Cross-Platform Wheels

    Linux and Windows32-64 wheels.

    MacOS is running into from map_metrics import map_metrics E ImportError: dlopen(/Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/map_metrics/map_metrics.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libboost_graph-mt-x64.dylib E Referenced from: /Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/map_metrics/map_metrics.cpython-36m-darwin.so E Reason: image not found

    opened by achains 1
  • Speed up Github Actions workflow

    Speed up Github Actions workflow

    It takes an hour to build Open3D

    Possible solutions:

    • [ ] Add -j flag
    • [ ] Specify Open3D build options
    • [ ] Store Open3D files in Github Actions Cache
    enhancement 
    opened by achains 1
  • Python version interface improvements

    Python version interface improvements

    Tasks:

    • [x] Break down large functions
    • [x] Break functions into modules
    • [x] Build pip-package, configure CI
    • [x] Add more functions, e.g. I/O processing
    opened by achains 0
  • pip-package. MPV. MME

    pip-package. MPV. MME

    Overview

    Python package wheels with implemented basic metrics (MPV, MME) on C++

    Platform tags

    • manylinux2010_x86_64
    • macosx_10_14_x86_64
    • win32 / win_amd64

    Python version

    • Python >=3.6
    opened by achains 0
  • Cosmetic fixes

    Cosmetic fixes

    What was done:

    • Removed redundant comments
    • Implemented general interface for methods
    • Added missing headers
    • Unsigned long -> int
    • All functions have CamelCase now
    opened by achains 0
  • CMake project structure. Baseline MME and MPV

    CMake project structure. Baseline MME and MPV

    Progress:

    • Configured CMake files
    • Baseline of mme and mpv algorithms

    Notes (Tasks for next PR)

    • Algorithms need to be tested
    • CI should be configured
    opened by achains 0
  • A drawback of mom when dealing drafting walls in indoor envs.

    A drawback of mom when dealing drafting walls in indoor envs.

    • Map Metrics 0.0.1:
    • Python version 3.8:
    • Operating System win10: 1

    Description

    It tired the metirc on a small scale data, likely a room with differential chassis and a RS-16. When the point cloud map consists of multiple wall (actually one) caused by drifting, the mom metric may have a smaller value than a normal map. I think it is caused by the orthogonal walls (both the real one and drifted one) and floor.

    Specifically, Fig.1 is generated by LOAM (without drift), and Fig. 2 is gererated by Lio-sam (with draft).

    Fig.1

    Fig.2

    I also considered to downsample the two maps into the same scale with voxel downsampling. The results are listed below.

    1640311688(1)

    I think this kind of problem is very familiar for indoor, I have seen many drafting wall during daily usage of lidar based slam, without semantic labels, it is very hard for distinguishing them automatically.

    What I Did

    I will try the 0.0.2 version later.

    Paste the command(s) you ran and the output.
If there was a crash, please include the traceback here.
    opened by hahakid 2
  • Add C++ executable with console interface

    Add C++ executable with console interface

    It will be useful for quick testing and performance measurments if library is able to run from command-line interface.

    For example, ./map-metrics --pc="data/kitti_00" --tj="data/Tj_0" --metric="mme" [--"some common hyperparameters"]

    It's build should be disabled by default

    enhancement 
    opened by achains 0
  • Support 10.9 MacOS

    Support 10.9 MacOS

    • Map Metrics version: 0.0.3
    • Python version: >=3.6
    • Operating System: MacOS

    Description

    Pip-package supports MacOS >=10.14 due to C++17 features. We can downgrade to C++14, but a bit later :)

    enhancement 
    opened by achains 0
  • editoring error

    editoring error

    A \delta seems missing in Eq.6 of the paper. Have you tested on all the sequences and other dataset. Recommend to ref LIBRE: The Multiple 3D LiDAR Dataset. VLP64 may have a low precise for long distance measurement. clipboard

    opened by hahakid 1
Owner
Mobile Robotics Lab. at Skoltech
Mobile Robotics Lab. at Skoltech
GitHub Repository
lightweight python wrapper for vowpal wabbit

vowpal_porpoise Lightweight python wrapper for vowpal_wabbit. Why: Scalable, blazingly fast machine learning. Install Install vowpal_wabbit. Clone and

Joseph Reisinger 163 Nov 24, 2022
A parametric soroban written with CADQuery.

A parametric soroban written in CADQuery The purpose of this project is to demonstrate how "code CAD" can be intuitive to learn. See soroban.py for a

Lee 4 Aug 13, 2022
A curated list of awesome game datasets, and tools to artificial intelligence in games

🎮 Awesome Game Datasets In computer science, Artificial Intelligence (AI) is intelligence demonstrated by machines. Its definition, AI research as th

Leonardo Mauro 454 Jan 03, 2023
"Learning Free Gait Transition for Quadruped Robots vis Phase-Guided Controller"

PhaseGuidedControl The current version is developed based on the old version of RaiSim series, and possibly requires further modification. It will be

X-Mechanics 12 Oct 21, 2022
Linear image-to-image translation

Linear (Un)supervised Image-to-Image Translation Examples for linear orthogonal transformations in PCA domain, learned without pairing supervision. Tr

Eitan Richardson 40 Aug 31, 2022
FMA: A Dataset For Music Analysis

FMA: A Dataset For Music Analysis Michaël Defferrard, Kirell Benzi, Pierre Vandergheynst, Xavier Bresson. International Society for Music Information

Michaël Defferrard 1.8k Dec 29, 2022
Character-Input - Create a program that asks the user to enter their name and their age

Character-Input Create a program that asks the user to enter their name and thei

PyLaboratory 0 Feb 06, 2022
API for RL algorithm design & testing of BCA (Building Control Agent) HVAC on EnergyPlus building energy simulator by wrapping their EMS Python API

RL - EmsPy (work In Progress...) The EmsPy Python package was made to facilitate Reinforcement Learning (RL) algorithm research for developing and tes

20 Jan 05, 2023
Pytorch Lightning Implementation of SC-Depth Methods.

SC_Depth_pl: This is a pytorch lightning implementation of SC-Depth (V1, V2) for self-supervised learning of monocular depth from video. In the V1 (IJ

JiaWang Bian 216 Dec 30, 2022
Single Image Super-Resolution (SISR) with SRResNet, EDSR and SRGAN

Single Image Super-Resolution (SISR) with SRResNet, EDSR and SRGAN Introduction Image super-resolution (SR) is the process of recovering high-resoluti

8 Apr 15, 2022
Unofficial PyTorch Implementation of UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation

UnivNet UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation This is an unofficial PyTorch

MINDs Lab 170 Jan 04, 2023
[ICML 2021, Long Talk] Delving into Deep Imbalanced Regression

Delving into Deep Imbalanced Regression This repository contains the implementation code for paper: Delving into Deep Imbalanced Regression Yuzhe Yang

Yuzhe Yang 568 Dec 30, 2022
DualGAN-tensorflow: tensorflow implementation of DualGAN

ICCV paper of DualGAN DualGAN: unsupervised dual learning for image-to-image translation please cite the paper, if the codes has been used for your re

Jack Yi 252 Nov 10, 2022
This program creates a formatted excel file which highlights the undervalued stock according to Graham's number.

Over-and-Undervalued-Stocks Of Nepse Using Graham's Number Scrap the latest data using different websites and creates a formatted excel file that high

6 May 03, 2022
A collection of resources and papers on Diffusion Models, a darkhorse in the field of Generative Models

This repository contains a collection of resources and papers on Diffusion Models and Score-based Models. If there are any missing valuable resources

5.1k Jan 08, 2023
This package proposes simplified exporting pytorch models to ONNX and TensorRT, and also gives some base interface for model inference.

PyTorch Infer Utils This package proposes simplified exporting pytorch models to ONNX and TensorRT, and also gives some base interface for model infer

Alex Gorodnitskiy 11 Mar 20, 2022
Pytorch Implementation of the paper "Cross-domain Correspondence Learning for Exemplar-based Image Translation"

CoCosNet Pytorch Implementation of the paper "Cross-domain Correspondence Learning for Exemplar-based Image Translation" (CVPR 2020 oral). Update: 202

Lingbo Yang 38 Sep 22, 2021
Code for ICCV 2021 paper: ARAPReg: An As-Rigid-As Possible Regularization Loss for Learning Deformable Shape Generators..

ARAPReg Code for ICCV 2021 paper: ARAPReg: An As-Rigid-As Possible Regularization Loss for Learning Deformable Shape Generators.. Installation The cod

Bo Sun 132 Nov 28, 2022
"Domain Adaptive Semantic Segmentation without Source Data" (ACM MM 2021)

LDBE Pytorch implementation for two papers (the paper will be released soon): "Domain Adaptive Semantic Segmentation without Source Data", ACM MM2021.

benfour 16 Sep 28, 2022
The repository for the paper "When Do You Need Billions of Words of Pretraining Data?"

pretraining-learning-curves This is the repository for the paper When Do You Need Billions of Words of Pretraining Data? Edge Probing We use jiant1 fo

ML² AT CILVR 19 Nov 25, 2022