Code release for our paper, "SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo"

Last update: Dec 14, 2022

Related tags

Overview

SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo

Thomas Kollar, Michael Laskey, Kevin Stone, Brijen Thananjeyan, Mark Tjersland

This repo contains the code to train the SimNet architecture on procedurally generated simulation data from scratch (no transfer learning required). We also provide a small set of in-house manually labelled validation data containing 3d oriented bounding box labels.

Training the model

Requirements

You will need a Nvidia GPU with at least 12GB of RAM. All code was tested and developed on Ubuntu 20.04.

All commands are assumed to be run from the root of the simnet repo directory (represented by $SIMNET_REPO in commands below).

Setup

Python

Create a python 3.8 virtual environment and install requirements:

cd $SIMNET_REPO
conda create -y --prefix ./env python=3.8
./env/bin/python -m pip install --upgrade pip
./env/bin/python -m pip install -r frozen_requirements.txt

Docker

Make sure docker is installed and working without requiring sudo. If it is not installed, follow the official instructions for setting it up.

docker ps

Wandb

Launch wandb local server for logging training results (you do not need to do this if you already have a wandb account setup). This will launch a local webserver http://localhost:8080 using docker that you can use to visualize training progress and validation images. You will have to visit the http://localhost:8080/authorize page to get the local API access token (this can take a few minutes the first time). Once you get the key you can paste it into the terminal to continue.

cd $SIMNET_REPO
./env/bin/wandb local

Datasets

Download and untar train+val datasets simnet2021a.tar (18GB, md5 checksum:b8e1d3cb7200b44b1de223e87141f14b). This file contains all the training and validation you need to replicate our small objects results.

cd $SIMNET_REPO
wget https://tri-robotics-public.s3.amazonaws.com/github/simnet/datasets/simnet2021a.tar -P datasets
tar xf datasets/simnet2021a.tar -C datasets

Train and Validate

Overfit test:

./runner.sh net_train.py @config/net_config_overfit.txt

Full training run (requires 12GB GPU memory)

./runner.sh net_train.py @config/net_config.txt

Results

Check wandb (http://localhost:8080) to see training progress. On a Titan V, it takes about 48 hours for training to converge, but decent validation results can be seen around 24 hours.

Example validation image visualization:

Example 3D oriented bounding box mAP on validation dataset:

Licenses

The source code is released under the MIT license.

The datasets are released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

You might also like...

The code release of paper Low-Light Image Enhancement with Normalizing Flow

[AAAI 2022] Low-Light Image Enhancement with Normalizing Flow Paper | Project Page Low-Light Image Enhancement with Normalizing Flow Yufei Wang, Renji

176 Jan 6, 2023

PyTorch implementation of our Adam-NSCL algorithm from our CVPR2021 (oral) paper "Training Networks in Null Space for Continual Learning"

Adam-NSCL This is a PyTorch implementation of Adam-NSCL algorithm for continual learning from our CVPR2021 (oral) paper: Title: Training Networks in N

34 Dec 21, 2022

Code release for NeX: Real-time View Synthesis with Neural Basis Expansion

Comments

depth noise model

I was looking through the code and was curious about the depth noise model. I found this: https://github.com/ToyotaResearchInstitute/simnet/blob/main/simnet/lib/camera.py but I can't seem to find camera_noise. Is it in the repository?

opened by seann999 1
Pre-trained Models

Hi Kevin and the team,

Thanks for making the data and code available, really impressive work on the paper.

Is there any plans to make the pre-trained model available, especially the SimNet benchmarked in the paper.

Thanks,

opened by ppyht2 0

Code release for our paper, "SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo"

Related tags

Overview

SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo

Training the model

Requirements

Setup

Python

Docker

Wandb

Datasets

Train and Validate

Results

Licenses

You might also like...

The code release of paper Low-Light Image Enhancement with Normalizing Flow

PyTorch implementation of our Adam-NSCL algorithm from our CVPR2021 (oral) paper "Training Networks in Null Space for Continual Learning"

Code release for NeX: Real-time View Synthesis with Neural Basis Expansion

Code release for "Transferable Semantic Augmentation for Domain Adaptation" (CVPR 2021)

Code release for "COTR: Correspondence Transformer for Matching Across Images"

We will release the code of "ConTNet: Why not use convolution and transformer at the same time?" in this repo

This is the dataset and code release of the OpenRooms Dataset.

Code release for DS-NeRF (Depth-supervised Neural Radiance Fields)

Code release for BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images

Comments

depth noise model

Pre-trained Models

Releases(v0.0.1)

v0.0.1(Jul 19, 2021)

Owner

PyTorch implementation of Deep HDR Imaging via A Non-Local Network (TIP 2020).

Simple and Effective Few-Shot Named Entity Recognition with Structured Nearest Neighbor Learning

Supplementary materials for ISMIR 2021 LBD paper "Evaluation of Latent Space Disentanglement in the Presence of Interdependent Attributes"

code for ICCV 2021 paper 'Generalized Source-free Domain Adaptation'

Code for How To Create A Fully Automated AI Based Trading System With Python

High-quality single file implementation of Deep Reinforcement Learning algorithms with research-friendly features

This repository is the official implementation of Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (NeurIPS21).

Image-generation-baseline - MUGE Text To Image Generation Baseline

Py4fi2nd - Jupyter Notebooks and code for Python for Finance (2nd ed., O'Reilly) by Yves Hilpisch.

A PyTorch implementation: "LASAFT-Net-v2: Listen, Attend and Separate by Attentively aggregating Frequency Transformation"

This is a GUI interface which can process forest fire detection, smoke detection and fire segmentation

Pairwise Learning for Neural Link Prediction for OGB (PLNLP-OGB)

An implementation for Neural Architecture Search with Random Labels (CVPR 2021 poster) on Pytorch.

The world's largest toxicity dataset.

Codebase of deep learning models for inferring stability of mRNA molecules

The repository offers the official implementation of our paper in PyTorch.

[ICCV '21] In this repository you find the code to our paper Keypoint Communities

​TextWorld is a sandbox learning environment for the training and evaluation of reinforcement learning (RL) agents on text-based games.

Denoising Diffusion Probabilistic Models

Rocket-recycling with Reinforcement Learning

TextWorld is a sandbox learning environment for the training and evaluation of reinforcement learning (RL) agents on text-based games.