Skip to content

Semi-supervised Implicit Scene Completion from Sparse LiDAR

Notifications You must be signed in to change notification settings

OPEN-AIR-SUN/SISC

Repository files navigation

Semi-supervised Implicit Scene Completion from Sparse LiDAR

New Repo

Paper

Created by Pengfei Li, Yongliang Shi, Tianyu Liu, Hao Zhao, Guyue Zhou and YA-QIN ZHANG from Institute for AI Industry Research(AIR), Tsinghua University.

demo

For complete video, click HERE.

We use the proposed model trained on the KITTI dataset to predict implicit completion results on the novel DAIR-V2X dataset. The results are impressive:

china china_1

teaser

sup0

sup1

sup2

sup3

sup4

Introduction

Recent advances show that semi-supervised implicit representation learning can be achieved through physical constraints like Eikonal equations. However, this scheme has not yet been successfully used for LiDAR point cloud data, due to its spatially varying sparsity.

In this repository, we develop a novel formulation that conditions the semi-supervised implicit function on localized shape embeddings. It exploits the strong representation learning power of sparse convolutional networks to generate shape-aware dense feature volumes, while still allows semi-supervised signed distance function learning without knowing its exact values at free space. With extensive quantitative and qualitative results, we demonstrate intrinsic properties of this new learning system and its usefulness in real-world road scenes. Notably, we improve IoU from 26.3% to 51.0% on SemanticKITTI. Moreover, we explore two paradigms to integrate semantic label predictions, achieving implicit semantic completion. Codes and data are publicly available.

Citation

If you find our work useful in your research, please consider citing:

@misc{li2021semisupervised,
    title={Semi-supervised Implicit Scene Completion from Sparse LiDAR}, 
    author={Pengfei Li and Yongliang Shi and Tianyu Liu and Hao Zhao and Guyue Zhou and Ya-Qin Zhang},
    year={2021},
    eprint={2111.14798},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}

Installation

Requirements

CUDA=11.1
python>=3.8
Pytorch>=1.8
numpy
ninja
MinkowskiEngine
tensorboard
pyyaml
configargparse
scripy
open3d
h5py
plyfile
scikit-image

Clone the repository:

git clone https://github.com/OPEN-AIR-SUN/SISC.git

Data preparation

Download the SemanticKITTI dataset from HERE. Unzip it into the same directory as SISC.

Training and inference

The configuration for training/inference is stored in opt.yaml, which can be modified as needed.

Scene Completion

Run the following command for a certain task (train/valid/visualize):

CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --nproc_per_node=1 main_sc.py --task=[task] --experiment_name=[experiment_name]

Semantic Scene Completion

SSC option A

Run the following command for a certain task (ssc_pretrain/ssc_valid/train/valid/visualize):

CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --nproc_per_node=1 main_ssc_a.py --task=[task] --experiment_name=[experiment_name]

Here, use ssc_pretrain/ssc_valid to train/validate the SSC part. Then the pre-trained model can be used to further train the whole model.

SSC option B

Run the following command for a certain task (train/valid/visualize):

CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --nproc_per_node=1 main_ssc_b.py --task=[task] --experiment_name=[experiment_name]

Model Zoo

Our pre-trained models can be downloaded here:

Ablation Pretrained Checkpoints
data augmentation no aug rotate & flip
Dnet input radial distance radial distance & height
Dnet structure last1 pruning last2 pruning last3 pruning last4 pruning Dnet relu 4convs output
Gnet structure width128 depth4 width512 depth4 width256 depth3 width256 depth5 Gnet relu
point sample on:off=1:2 on:off=2:3
positional encoding no encoding incF level10 incT level5 incT level15
sample strategy nearest
scale size scale 2 scale 4 scale 8 scale 16 scale 32
shape size shape 128 shape 512
SSC SSC option A SSC option B
Seg-Net&SSC-Net for SSC-A ssc_pretrain

These models correspond to the ablation study in our paper. The Scale 4 works as our baseline.

Ablation Corresponding Configs
data augmentation no aug rotate & flip
Dnet input radial distance radial distance & height
Dnet structure last1 pruning last2 pruning last3 pruning last4 pruning Dnet relu 4convs output
Gnet structure width128 depth4 width512 depth4 width256 depth3 width256 depth5 Gnet relu
point sample on:off=1:2 on:off=2:3
positional encoding no encoding incF level10 incT level5 incT level15
sample strategy nearest
scale size scale 2 scale 4 scale 8 scale 16 scale 32
shape size shape 128 shape 512
SSC SSC option A SSC option B

About

Semi-supervised Implicit Scene Completion from Sparse LiDAR

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published