This package is for running the semantic SLAM algorithm using extracted planar surfaces from the received detection

Last update: Dec 02, 2022

Related tags

Overview

Semantic SLAM

This package can perform optimization of pose estimated from VO/VIO methods which tend to drift over time. It uses planar surfaces extracted from object detections in order to create a sparse semantic map of the environment, thus optimizing the drift of the VO/VIO algorithms.

In order to run this package you will need two additional modules

A VO/VIO algorithm: ROVIO, VINS_MONO, OKVIS etc.
An Object Detector: Yolo, Shape Color Detector

Currently it can extract planar surfaces and create a semantic map from from the following objects:

chair
tvmonitor
book
keyboard
laptop
bucket
car

Related Paper:

@ARTICLE{9045978,
  author={Bavle, Hriday and De La Puente, Paloma and How, Jonathan P. and Campoy, Pascual},
  journal={IEEE Access}, 
  title={VPS-SLAM: Visual Planar Semantic SLAM for Aerial Robotic Systems}, 
  year={2020},
  volume={8},
  number={},
  pages={60704-60718},
  doi={10.1109/ACCESS.2020.2983121}}

Video

How do I set it up?

First install g2o following these instructions (Tested on Kinetic and Melodic Distributions):

- sudo apt-get install ros-$ROS_DISTRO-libg2o
- sudo cp -r /opt/ros/$ROS_DISTRO/lib/libg2o_* /usr/local/lib
- sudo cp -r /opt/ros/$ROS_DISTRO/include/g2o /usr/local/include

Install OctopMap server for map generation capabilities:

- sudo apt install ros-$ROS_DISTRO-octomap*

Try a simple example with pre-recorded VIO pose and a blue bucket detector:

Create a ros workspace and clone the following packages:

Download the rosbag:

    wget -P ~/Downloads/ https://www.dropbox.com/s/jnywuvcn2m9ubu2/entire_lab_3_rounds.bag

Create a workspace, clone the repo and compile:

    mkdir -p workspace/ros/semantic_slam_ws/src/ && cd workspace/ros/semantic_slam_ws/src/    
    git clone https://github.com/hridaybavle/semantic_slam && git clone https://bitbucket.org/hridaybavle/bucket_detector.git   
    cd .. && catkin build --cmake-args -DCMAKE_BUILD_TYPE=Release

Launch and visualize

    source devel/setup.bash
    roslaunch semantic_SLAM ps_slam_with_snap_pose_bucket_det_lab_data_with_octomap.launch bagfile:=${HOME}/Downloads/entire_lab_3_rounds.bag show_rviz:=true

Using Docker Image

If the code is giving problems with you local machine, you can try the docker image created with the repo and the required settings.

Download Docker from: Docker

Follow the commands to run the algorithm with the docker

  docker pull hridaybavle/semantic_slam:v1 	
  docker run --rm -it --net="host" -p 11311:11311 hridaybavle/semantic_slam:v1 bash
  cd ~/workspace/ros/semantic_slam_ws/
  source devel/setup.bash
  roslaunch semantic_SLAM ps_slam_with_snap_pose_bucket_det_lab_data_with_octomap.launch bagfile:=${HOME}/Downloads/entire_lab_3_rounds.bag show_rviz:=false

Open a new terminal and rviz in local machine

  cd ~/Downloads/ && wget https://raw.githubusercontent.com/hridaybavle/semantic_slam/master/rviz/graph_semantic_slam.rviz
  rviz -d graph_semantic_slam.rviz

Subsribed Topics

/SQ04/snap_vislam/vislam/pose (geometry_msgs/PoseStamped)
The default snapdragon VIO pose published in NED in frame. This message can be remapped remapped to any other VO pose message publishing in NED frame. (See frame conventions)
/rovio/odometry (geometry_msgs/PoseStamped)
The VIO odometry published in ENU frame. Can be remapped to the desired topic name in the launcher.
/depth_registered/points (sensor_msgs/PointCloud2) The point cloud required for planar surface extraction.
/darknet_ros/bounding_boxes(darknet_msgs_ros/BoundingBoxes)
The detection bounding boxes published by yolo if using the yolo detector ros package.
/image_processed/bounding_boxes(ShapeColor_ObjectDetection/DetectedObjects) The detection bounding boxes if using the bucket detector. It can be downloaded from the link above.

Published Topics

robot_pose(geometry_msgs/PoseStamped) The pose of the robot estimated by the algo.
robot_path(nav_msgs/Path)
The path of the robot psoe for visualization.
keyframe_poses(geometry_msgs/PoseArray)
The poses of the keyframe being added to the g2o graph.
mapped_landmarks(visualization_msgs/MarkerArray)
The mapped semantic planar surfaces.
detected_landmarks(visualization_msgs/MarkerArray)
The detected landmarks in the current frame.

The configurations of the algorithms can be found inside the cfg folder in order to be changed accordingly.

Published TFs

map to odom transform: The transform published between the map frame and the odom frame after the corrections from the semantic SLAM.
base_link to odom transform: The transform published between the base_link (on the robot) frame and the odom frame as estimated by the VO/VIO algorithm.

You might also like...

Indoor Panorama Planar 3D Reconstruction via Divide and Conquer

HV-plane reconstruction from a single 360 image Code for our paper in CVPR 2021: Indoor Panorama Planar 3D Reconstruction via Divide and Conquer (pape

36 Jan 3, 2023

[ICCV 2021 (oral)] Planar Surface Reconstruction from Sparse Views

Planar Surface Reconstruction From Sparse Views Linyi Jin, Shengyi Qian, Andrew Owens, David F. Fouhey University of Michigan ICCV 2021 (Oral) This re

89 Jan 5, 2023

PyTorch implementation of HDN(Homography Decomposition Networks) for planar object tracking

Homography Decomposition Networks for Planar Object Tracking This project is the offical PyTorch implementation of HDN(Homography Decomposition Networ

48 Dec 15, 2022

Pytorch implementation of paper: "NeurMiPs: Neural Mixture of Planar Experts for View Synthesis"

NeurMips: Neural Mixture of Planar Experts for View Synthesis This is the official repo for PyTorch implementation of paper "NeurMips: Neural Mixture

101 Dec 13, 2022

Sequence lineage information extracted from RKI sequence data repo

Pango lineage information for German SARS-CoV-2 sequences This repository contains a join of the metadata and pango lineage tables of all German SARS-

24 Oct 26, 2022

Official page of Struct-MDC (RA-L'22 with IROS'22 option); Depth completion from Visual-SLAM using point & line features

Struct-MDC (click the above buttons for redirection!) Official page of "Struct-MDC: Mesh-Refined Unsupervised Depth Completion Leveraging Structural R

37 Dec 22, 2022

Code for "Diffusion is All You Need for Learning on Surfaces"

Source code for "Diffusion is All You Need for Learning on Surfaces", by Nicholas Sharp Souhaib Attaiki Keenan Crane Maks Ovsjanikov NOTE: the linked

247 Dec 28, 2022

Code for Iso-Points: Optimizing Neural Implicit Surfaces with Hybrid Representations

Implementation for Iso-Points (CVPR 2021) Official code for paper Iso-Points: Optimizing Neural Implicit Surfaces with Hybrid Representations paper |

66 Nov 8, 2022

Neural-Pull: Learning Signed Distance Functions from Point Clouds by Learning to Pull Space onto Surfaces(ICML 2021)

Neural-Pull: Learning Signed Distance Functions from Point Clouds by Learning to Pull Space onto Surfaces(ICML 2021) This repository contains the code

149 Dec 15, 2022

Comments

errors at last step

Hi, I have finished all the steps following the instructions and nothing goes wrong. But when I run

roslaunch semantic_SLAM ps_slam_with_snap_pose_bucket_det_lab_data.launch bagfile:=${HOME}/Downloads/entire_lab_3_rounds.bag show_rviz:=true

I get errors like this and it stucks for a while.

# Using CSparse poseDim -1 landMarkDim -1 blockordering 0
done
keyframe_delta_trans 0.5
keyframe_delta_angle 0.5
keyframe_delta_time 1
use_const_inf_matrix: 1
const_stddev_x: 0.00667
const_stddev_q: 1e-05
Initialized mapping thread 
camera angle in radians: 0.59219
update keyframe every detection: 1
add first landmark: 0
[semantic_graph_slam_node-9] process has died [pid 23067, exit code -11, cmd /home/nrc/workspace/ros/semantic_slam_ws/devel/lib/semantic_SLAM/semantic_graph_SLAM_node __name:=semantic_graph_slam_node __log:=/home/nrc/.ros/log/ccaf4b14-a47a-11ea-b300-000c29c39525/semantic_graph_slam_node-9.log].
log file: /home/nrc/.ros/log/ccaf4b14-a47a-11ea-b300-000c29c39525/semantic_graph_slam_node-9*.log

then I get this. It seems that the visualization program doesn't go right.

[rosbag-2] process has finished cleanly
log file: /home/nrc/.ros/log/ccaf4b14-a47a-11ea-b300-000c29c39525/rosbag-2*.log

Is there something I have missed? Thank you!

opened by ZhengXinyue 8

[semantic_graph_slam_node-9] process has died

Hi, I have finished all the steps following the instructions and nothing goes wrong. But when I run

roslaunch semantic_SLAM ps_slam_with_snap_pose_bucket_det_lab_data_with_octomap.launch bagfile:=${HOME}/Downloads/entire_lab_3_rounds.bag show_rviz:=true

I get errors like this.

done
keyframe_delta_trans 0.5
keyframe_delta_angle 0.5
keyframe_delta_time 1
use_const_inf_matrix: 1
const_stddev_x: 0.00667
const_stddev_q: 1e-05
camera angle in radians: 0.59219
update keyframe every detection: 1
add first landmark: 0
[ INFO] [1591944956.099907360, 1661396775.076756992]: waitForService: Service [/depth_rectifier_manager/load_nodelet] is now available.
[ INFO] [1591944956.100243666, 1661396775.076756992]: waitForService: Service [/depth_manager/load_nodelet] is now available.
[ INFO] [1591944956.545617511, 1661396775.518832629]: Stereo is NOT SUPPORTED
[ INFO] [1591944956.545842654, 1661396775.518832629]: OpenGl version: 4.5 (GLSL 4.5).
[pcl::OrganizedNeighbor::radiusSearch] Input dataset is not from a projective device!
Residual (MSE) 0.000614, using 1248 valid points
[pcl::OrganizedNeighbor::radiusSearch] Input dataset is not from a projective device!
Residual (MSE) 0.000748, using 1444 valid points
[pcl::OrganizedNeighbor::radiusSearch] Input dataset is not from a projective device!
Residual (MSE) 0.001710, using 2303 valid points
[semantic_graph_slam_node-9] process has died [pid 27314, exit code -9, cmd /home/nrc/hd/workspace/ros/semantic_slam_ws/devel/lib/semantic_SLAM/semantic_graph_SLAM_node __name:=semantic_graph_slam_node __log:=/home/nrc/.ros/log/c2c4ddd8-ac79-11ea-96ed-8ca982ff1833/semantic_graph_slam_node-9.log].
log file: /home/nrc/.ros/log/c2c4ddd8-ac79-11ea-96ed-8ca982ff1833/semantic_graph_slam_node-9*.log

When it occurs

[pcl::OrganizedNeighbor::radiusSearch] Input dataset is not from a projective device!
Residual (MSE) 0.000614, using 1248 valid points

the program is still mapping , so I think the problem is not caused by 'pcl'.

I tried to run the launchfile seperately :

ROS_NAMESPACE=camera/color rosrun image_proc image_proc 
roslaunch semantic_SLAM shape.launch  
rosrun semantic_SLAM  semantic_graph_SLAM_node

But at the last step i got 'Segmentation fault :

add first landmark: 0
Segmentation fault (core dumped)

Do you have any idea about it? Thanks a lot !!!

opened by He-Rong 6

Dataset download failure problem

Hello, when I run the sample code, I always encounter network interruptions or unknown errors at the last moment when downloading the dataset entire_lab_3_rounds.bag. Can you provide a new way to download the bag?

opened by kycwx 2
Problemas de incompatibilidad de opencv en el bucket detector

Hola, he conseguido que ambos paquetes en conjunto (semantic slam y bucket detector) funciones bien en una distro de ubuntu virgen con ROS melodic, sin embargo, cuando migro al pc donde trabajo habitualmente y que tiene ya instaladas dependencias anteriores y demás me encuentro con estos errores referentes a opencv: Imagino que se deben a incompatibilidades entre versiones de opencv, podrías confirmarme esto último? Sería posible trabajar con una versión de opencv diferente? Gracias, un saludo!

opened by iandresolares 2

Releases(2.0.0)

2.0.0(Aug 10, 2021)

This release contains all the tf's published from map-->odom-->base_link
Source code(tar.gz)
Source code(zip)

Owner

Hriday Bavle

Postdoctoral Researcher at the University of Luxembourg. My research interests are VO/VIO, SLAM, Perception and Planning applied to Mobile Robots.

GitHub Repository

Pytorch implementation of the paper "Enhancing Content Preservation in Text Style Transfer Using Reverse Attention and Conditional Layer Normalization"

4 Sep 18, 2022

Auto-Encoding Score Distribution Regression for Action Quality Assessment

DAE-AQA It is an open source program reference to paper Auto-Encoding Score Distribution Regression for Action Quality Assessment. 1.Introduction DAE

13 Nov 16, 2022

Rayvens makes it possible for data scientists to access hundreds of data services within Ray with little effort.

Rayvens augments Ray with events. With Rayvens, Ray applications can subscribe to event streams, process and produce events. Rayvens leverages Apache

32 Dec 25, 2022

A curated list of references for MLOps

9.3k Jan 07, 2023

Leaf: Multiple-Choice Question Generation

Leaf: Multiple-Choice Question Generation Easy to use and understand multiple-choice question generation algorithm using T5 Transformers. The applicat

62 Dec 20, 2022

Multi-Objective Loss Balancing for Physics-Informed Deep Learning

Multi-Objective Loss Balancing for Physics-Informed Deep Learning Code for ReLoBRaLo. Abstract Physics Informed Neural Networks (PINN) are algorithms

16 Dec 12, 2022

Lighthouse: Predicting Lighting Volumes for Spatially-Coherent Illumination

Lighthouse: Predicting Lighting Volumes for Spatially-Coherent Illumination Pratul P. Srinivasan, Ben Mildenhall, Matthew Tancik, Jonathan T. Barron,

65 Dec 14, 2022

Official implementation of the paper Chunked Autoregressive GAN for Conditional Waveform Synthesis

PyEmits, a python package for easy manipulation in time-series data. Time-series data is very common in real life. Engineering FSI industry (Financial

150 Dec 06, 2022

Pytorch cuda extension of grid_sample1d

Grid Sample 1d pytorch cuda extension of grid sample 1d. Since pytorch only supports grid sample 2d/3d, I extend the 1d version for efficiency. The fo

24 Dec 03, 2022

The undersampled DWI image using Slice-Interleaved Diffusion Encoding (SIDE) method can be reconstructed by the UNet network.

UNet-SIDE The undersampled DWI image using Slice-Interleaved Diffusion Encoding (SIDE) method can be reconstructed by the UNet network. For Super Reso

1 Jan 13, 2022