Official Implementation of SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-Training for Spatial-Aware Visual Representations

Related tags

Deep LearningSimIPU
Overview

Official Implementation of SimIPU

  • SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-Training for Spatial-Aware Visual Representations
  • Since the code is still waiting for release, if you have any question with reproduction, feel free to contact us. We will try our best to help you.
  • Currently, the core code of SimIPU is implemented in the commercial project. We are trying our best to make the code publicly available.
Comments
  • Question about augmentation

    Question about augmentation

    Hi, I'm a little confused about the data augmentation.

    1. How did you set img_aug when img_moco=True? It seems that we need an 'img_pipeline' in 'simipu_kitti.py', right?
    2. For 3D augmentation, it seems that it is done in this line. So the 3D augmentation is done based on the point features instead the raw points, right? If I want to try moco=True, how to set 3D augmentation? should I do this in the dataset building part? https://github.com/zhyever/SimIPU/blob/5b346e392c161a5e9fdde09b1692656bc7cd3faf/project_cl/decorator/inter_intro_decorator_moco_better.py#L394

    Looking forward to your reply. Many thanks.

    opened by sunnyHelen 2
  • error for env setup:ImportError: cannot import name 'ball_query_ext' from 'mmdet3d.ops.ball_query'

    error for env setup:ImportError: cannot import name 'ball_query_ext' from 'mmdet3d.ops.ball_query'

    Thanks for your insightful paper and clear code repo!

    Hi, I met with the ImportError: cannot import name 'ball_query_ext' from 'mmdet3d.ops.ball_query' when run the command bash tools/dist_train.sh project_cl/configs/simipu/simipu_kitti.py 1 --work_dir ./

    Do you know how to solve it?

    Traceback (most recent call last): File "tools/train.py", line 16, in from mmdet3d.apis import train_model File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/apis/init.py", line 1, in from .inference import (convert_SyncBN, inference_detector, File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/apis/inference.py", line 10, in from mmdet3d.core import (Box3DMode, DepthInstance3DBoxes, File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/core/init.py", line 2, in from .bbox import * # noqa: F401, F403 File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/core/bbox/init.py", line 4, in from .iou_calculators import (AxisAlignedBboxOverlaps3D, BboxOverlaps3D, File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/core/bbox/iou_calculators/init.py", line 1, in from .iou3d_calculator import (AxisAlignedBboxOverlaps3D, BboxOverlaps3D, File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/core/bbox/iou_calculators/iou3d_calculator.py", line 5, in from ..structures import get_box_type File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/core/bbox/structures/init.py", line 1, in from .base_box3d import BaseInstance3DBoxes File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/core/bbox/structures/base_box3d.py", line 5, in from mmdet3d.ops.iou3d import iou3d_cuda File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/ops/init.py", line 5, in from .ball_query import ball_query File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/ops/ball_query/init.py", line 1, in from .ball_query import ball_query File "/mnt/lustre/xxh/SimIPU-main/mmdet3d/ops/ball_query/ball_query.py", line 4, in from . import ball_query_ext ImportError: cannot import name 'ball_query_ext' from 'mmdet3d.ops.ball_query' (/mnt/lustre/xxh/SimIPU-main/mmdet3d/ops/ball_query/init.py)

    I noticed that you once met with the same error. https://github.com/open-mmlab/mmdetection3d/issues/503#issuecomment-847618114

    So, I would like to ask for your help~ Hopefully you have a good solution. :)

    opened by JerryX1110 2
  • A question about eq5 and eq6

    A question about eq5 and eq6

    Thanks for your inspiring work. I have some wonder about eq5 and eq6. As far as I know, After eq5, f should be a tensor which is a global feature with shape (batchsize * 2048 * 1 * 1), how can you sample corresponding image features by projection location? After all, there's no spatial information in f anymore. Or maybe you got features from a previous layer of ResNet? Looking forward to your reply.

    opened by lianchengmingjue 2
  • A question about Tab.5 in Ablation Study

    A question about Tab.5 in Ablation Study

    Thanks for your excellent work first! I have a question about Tab.5 in Ablation Study. Why "Scratch" equals "SimIPU w/o inter-module ", which means that the intra-module is useless?

    opened by Trent-tangtao 1
  • Have you tried not to crop gradient of f^{\alpha} in eq7?

    Have you tried not to crop gradient of f^{\alpha} in eq7?

    Hi, I like your good work! I am wondering have you tried not to crop the gradient of $f^{\alpha}$ in eq7? If you crop the gradient, it seems like the pertaining of the point branch cannot learn anything from the image branch.

    opened by Hiusam 1
  • issues about create_data

    issues about create_data

    Hi, thanks for sharing your great work. I encounter some issues during creating data by running create_data.py First create reduced point cloud for training set [ ] 0/3712, elapsed: 0s, ETA:Traceback (most recent call last): File "tools/create_data.py", line 247, in
    out_dir=args.out_dir)
    File "tools/create_data.py", line 24, in kitti_data_prep
    kitti.create_reduced_point_cloud(root_path, info_prefix)
    File "/mnt/lustre/chenzhuo1/hzha/SimIPU/tools/data_converter/kitti_converter.py", line 374, in create_reduced_point_cloud
    _create_reduced_point_cloud(data_path, train_info_path, save_path)
    File "/mnt/lustre/chenzhuo1/hzha/SimIPU/tools/data_converter/kitti_converter.py", line 314, in _create_reduced_point_cloud
    count=-1).reshape([-1, num_features])
    ValueError: cannot reshape array of size 461536 into shape (6)

    It seems to set the num_features=4 and front_camera_id=2? in this line: https://github.com/zhyever/SimIPU/blob/5b346e392c161a5e9fdde09b1692656bc7cd3faf/tools/data_converter/kitti_converter.py#L291

    I assume doing this can solve the problem but encounter another problem when Create GT Database of KittiDataset
    [ ] 0/3712, elapsed: 0s, ETA:Traceback (most recent call last):
    File "tools/create_data.py", line 247, in
    out_dir=args.out_dir)
    File "tools/create_data.py", line 44, in kitti_data_prep
    with_bbox=True) # for moca
    File "/mnt/lustre/chenzhuo1/hzha/SimIPU/tools/data_converter/create_gt_database.py", line 275, in create_groundtruth_database
    P0 = np.array(example['P0']).reshape(4, 4)
    KeyError: 'P0'

    Can you help me figure out how to solve these issues?

    opened by sunnyHelen 21
Owner
Zhyever
Keep going.
Zhyever
[CVPR-2021] UnrealPerson: An adaptive pipeline for costless person re-identification

UnrealPerson: An Adaptive Pipeline for Costless Person Re-identification In our paper (arxiv), we propose a novel pipeline, UnrealPerson, that decreas

ZhangTianyu 70 Oct 10, 2022
MatchGAN: A Self-supervised Semi-supervised Conditional Generative Adversarial Network

MatchGAN: A Self-supervised Semi-supervised Conditional Generative Adversarial Network This repository is the official implementation of MatchGAN: A S

Justin Sun 12 Dec 27, 2022
A transformer which can randomly augment VOC format dataset (both image and bbox) online.

VocAug It is difficult to find a script which can augment VOC-format dataset, especially the bbox. Or find a script needs complex requirements so it i

Coder.AN 1 Mar 05, 2022
Title: Heart-Failure-Classification

This Notebook is based off an open source dataset available on where I have created models to classify patients who can potentially witness heart failure on the basis of various parameters. The best

Akarsh Singh 2 Sep 13, 2022
Classification Modeling: Probability of Default

Credit Risk Modeling in Python Introduction: If you've ever applied for a credit card or loan, you know that financial firms process your information

Aktham Momani 2 Nov 07, 2022
ShuttleNet: Position-aware Fusion of Rally Progress and Player Styles for Stroke Forecasting in Badminton (AAAI 2022)

ShuttleNet: Position-aware Rally Progress and Player Styles Fusion for Stroke Forecasting in Badminton (AAAI 2022) Official code of the paper ShuttleN

Wei-Yao Wang 11 Nov 30, 2022
SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking

SPLADE 🍴 + 🥄 = 🔎 This repository contains the weights for four models as well as the code for running inference for our two papers: [v1]: SPLADE: S

NAVER 170 Dec 28, 2022
Structure Information is the Key: Self-Attention RoI Feature Extractor in 3D Object Detection

Structure Information is the Key: Self-Attention RoI Feature Extractor in 3D Object Detection abstract:Unlike 2D object detection where all RoI featur

DK. Zhang 2 Oct 07, 2022
Sentiment analysis translations of the Bhagavad Gita

Sentiment and Semantic Analysis of Bhagavad Gita Translations It is well known that translations of songs and poems not only breaks rhythm and rhyming

Machine learning and Bayesian inference @ UNSW Sydney 3 Aug 01, 2022
A set of tests for evaluating large-scale algorithms for Wasserstein-2 transport maps computation.

Continuous Wasserstein-2 Benchmark This is the official Python implementation of the NeurIPS 2021 paper Do Neural Optimal Transport Solvers Work? A Co

Alexander 22 Dec 12, 2022
An efficient PyTorch implementation of the winning entry of the 2017 VQA Challenge.

Bottom-Up and Top-Down Attention for Visual Question Answering An efficient PyTorch implementation of the winning entry of the 2017 VQA Challenge. The

Hengyuan Hu 731 Jan 03, 2023
Official implementation of NPMs: Neural Parametric Models for 3D Deformable Shapes - ICCV 2021

NPMs: Neural Parametric Models Project Page | Paper | ArXiv | Video NPMs: Neural Parametric Models for 3D Deformable Shapes Pablo Palafox, Aljaz Bozic

PabloPalafox 109 Nov 22, 2022
object detection; robust detection; ACM MM21 grand challenge; Security AI Challenger Phase VII

赛题背景 在商品知识产权领域,知识产权体现为在线商品的设计和品牌。不幸的是,在每一天,存在着非法商户通过一些对抗手段干扰商标识别来逃避侵权,这带来了很高的知识产权风险和财务损失。为了促进先进的多媒体人工智能技术的发展,以保护企业来之不易的创作和想法免受恶意使用和剽窃,因此提出了鲁棒性标识检测挑战赛

65 Dec 22, 2022
SimpleDepthEstimation - An unified codebase for NN-based monocular depth estimation methods

SimpleDepthEstimation Introduction This is an unified codebase for NN-based monocular depth estimation methods, the framework is based on detectron2 (

8 Dec 13, 2022
Use Python, OpenCV, and MediaPipe to control a keyboard with facial gestures

CheekyKeys A Face-Computer Interface CheekyKeys lets you control your keyboard using your face. View a fuller demo and more background on the project

69 Nov 09, 2022
Ray tracing of a Schwarzschild black hole written entirely in TensorFlow.

TensorGeodesic Ray tracing of a Schwarzschild black hole written entirely in TensorFlow. Dependencies: Python 3 TensorFlow 2.x numpy matplotlib About

5 Jan 15, 2022
Pytorch reimplementation of PSM-Net: "Pyramid Stereo Matching Network"

This is a Pytorch Lightning version PSMNet which is based on JiaRenChang/PSMNet. use python main.py to start training. PSM-Net Pytorch reimplementatio

XIAOTIAN LIU 1 Nov 25, 2021
Source code for the BMVC-2021 paper "SimReg: Regression as a Simple Yet Effective Tool for Self-supervised Knowledge Distillation".

SimReg: A Simple Regression Based Framework for Self-supervised Knowledge Distillation Source code for the paper "SimReg: Regression as a Simple Yet E

9 Oct 15, 2022
Occlusion robust 3D face reconstruction model in CFR-GAN (WACV 2022)

Occlusion Robust 3D face Reconstruction Yeong-Joon Ju, Gun-Hee Lee, Jung-Ho Hong, and Seong-Whan Lee Code for Occlusion Robust 3D Face Reconstruction

Yeongjoon 31 Dec 19, 2022
How Effective is Incongruity? Implications for Code-mix Sarcasm Detection.

Code for the paper: How Effective is Incongruity? Implications for Code-mix Sarcasm Detection - ICON ACL 2021

2 Jun 05, 2022