Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time

Overview

Semi Hand-Object

Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time (CVPR 2021). report

Project Page with Videos Teaser

Installation

  • Clone this repository:
    git clone https://github.com/stevenlsw/Semi-Hand-Object.git
  • Install the dependencies by the following command:
    pip install -r requirements.txt

Quick Demo (update soon)

Training and Evaluation on HO3D Dataset

Preparation

  • Download the MANO model files (mano_v1_2.zip) from MANO website. Unzip and put mano/models/MANO_RIGHT.pkl into assets/mano_models.

  • Download the YCB-Objects used in HO3D dataset. Put unzipped folder object_models under assets.

  • The structure should look like this:

Semi-Hand-Object/
  assets/
    mano_models/
      MANO_RIGHT.pkl
    object_models/
      006_mustard_bottle/
        points.xyz
        textured_simple.obj
      ......
  • Download and unzip HO3D dataset to path you like, the unzipped path is referred as $HO3D_root.

Evaluation

The hand & object pose estimation performance on HO3D dataset. We evaluate hand pose results on the official CodaLab challenge. The hand metric below is mean joint/mesh error after procrustes alignment, the object metric is average object vertices error within 10% of object diameter (ADD-0.1D).

In our model, we use transformer architecture to perform hand-object contextual reasoning.

Please download the trained model and save to path you like, the model path is refered as $resume.

trained-model joint↓ mesh↓ cleanser↑ bottle↑ can↑ ave↑
link 0.99 0.95 92.2 80.4 55.7 76.1
  • Testing with trained model

   python traineval.py --evaluate --HO3D_root={path to the dataset} --resume={path to the model} --test_batch=24 --host_folder=exp_results

The testing results will be saved in the $host_folder, which contains the following files:

  • option.txt (saved options)
  • object_result.txt (object pose evaluation performance)
  • pred.json (zip -j pred.zip pred.json and submit to the offical challenge for hand evaluation)

Training

Please download the preprocessed files to train HO3D dataset. The downloaded files contains training list and labels generated from the original dataset to accelerate training. Please put the unzipped folder ho3d-process to current directory.

    python traineval.py --HO3D_root={path to the dataset} --train_batch=24 --host_folder=exp_results

The models will be automatically saved in $host_folder

Citation

@inproceedings{liu2021semi,
  title={Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time},
  author={Liu, Shaowei and Jiang, Hanwen and Xu, Jiarui and Liu, Sifei and Wang, Xiaolong},
  booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition},
  year={2021}
}

TODO

  • Google colab demo

Acknowledgments

We thank:

List of all dependencies affected by node-ipc malicious commit

node-ipc-dependencies-list List of all dependencies affected by node-ipc malicious commit as of 17/3/2022 - 19/3/2022 (timestamp) Please improve upon

99 Oct 15, 2022
Official Implementation of HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation

HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation by Lukas Hoyer, Dengxin Dai, and Luc Van Gool [Arxiv] [Paper] Overview Unsup

Lukas Hoyer 149 Dec 28, 2022
Brax is a differentiable physics engine that simulates environments made up of rigid bodies, joints, and actuators

Brax is a differentiable physics engine that simulates environments made up of rigid bodies, joints, and actuators. It's also a suite of learning algorithms to train agents to operate in these enviro

Google 1.5k Jan 02, 2023
Mind the Trade-off: Debiasing NLU Models without Degrading the In-distribution Performance

Models for natural language understanding (NLU) tasks often rely on the idiosyncratic biases of the dataset, which make them brittle against test cases outside the training distribution.

Ubiquitous Knowledge Processing Lab 22 Jan 02, 2023
Imaginaire - NVIDIA's Deep Imagination Team's PyTorch Library

Imaginaire Docs | License | Installation | Model Zoo Imaginaire is a pytorch library that contains optimized implementation of several image and video

NVIDIA Research Projects 3.6k Dec 29, 2022
Two-Stage Peer-Regularized Feature Recombination for Arbitrary Image Style Transfer

Two-Stage Peer-Regularized Feature Recombination for Arbitrary Image Style Transfer Paper on arXiv Public PyTorch implementation of two-stage peer-reg

NNAISENSE 38 Oct 14, 2022
Official implementation of UTNet: A Hybrid Transformer Architecture for Medical Image Segmentation

UTNet (Accepted at MICCAI 2021) Official implementation of UTNet: A Hybrid Transformer Architecture for Medical Image Segmentation Introduction Transf

110 Jan 01, 2023
Aggragrating Nested Transformer Official Jax Implementation

NesT is a simple method, which aggragrates nested local transformers on image blocks. The idea makes vision transformers attain better accuracy, data efficiency, and convergence on the ImageNet bench

Google Research 169 Dec 20, 2022
A curated list of awesome resources related to Semantic Search🔎 and Semantic Similarity tasks.

A curated list of awesome resources related to Semantic Search🔎 and Semantic Similarity tasks.

224 Jan 04, 2023
Easy to use and customizable SOTA Semantic Segmentation models with abundant datasets in PyTorch

Semantic Segmentation Easy to use and customizable SOTA Semantic Segmentation models with abundant datasets in PyTorch Features Applicable to followin

sithu3 530 Jan 05, 2023
Predict bus arrival time using VertexAI and Nvidia's Jetson Nano

bus_prediction predict bus arrival time using VertexAI and Nvidia's Jetson Nano imagenet the command for imagenet.py look like this python3 /path/to/i

10 Dec 22, 2022
Code for visualizing the loss landscape of neural nets

Visualizing the Loss Landscape of Neural Nets This repository contains the PyTorch code for the paper Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer

Tom Goldstein 2.2k Jan 09, 2023
IAST: Instance Adaptive Self-training for Unsupervised Domain Adaptation (ECCV 2020)

This repo is the official implementation of our paper "Instance Adaptive Self-training for Unsupervised Domain Adaptation". The purpose of this repo is to better communicate with you and respond to y

CVSM Group - email: <a href=[email protected]"> 84 Dec 12, 2022
Lightweight library to build and train neural networks in Theano

Lasagne Lasagne is a lightweight library to build and train neural networks in Theano. Its main features are: Supports feed-forward networks such as C

Lasagne 3.8k Dec 29, 2022
Implementation of Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning

advantage-weighted-regression Implementation of Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning, by Peng et al. (

Omar D. Domingues 1 Dec 02, 2021
[ACM MM 2021] Yes, "Attention is All You Need", for Exemplar based Colorization

Transformer for Image Colorization This is an implemention for Yes, "Attention Is All You Need", for Exemplar based Colorization, and the current soft

Wang Yin 30 Dec 07, 2022
TransReID: Transformer-based Object Re-Identification

TransReID: Transformer-based Object Re-Identification [arxiv] The official repository for TransReID: Transformer-based Object Re-Identification achiev

569 Dec 30, 2022
Group project for MFIN7036. Our goal is to predict firm profitability with text-based competition measures.

NLP_0-project Group project for MFIN7036. Our goal is to predict firm profitability with text-based competition measures1. We are a "democratic" and c

3 Mar 16, 2022
A large-scale video dataset for the training and evaluation of 3D human pose estimation models

ASPset-510 ASPset-510 (Australian Sports Pose Dataset) is a large-scale video dataset for the training and evaluation of 3D human pose estimation mode

Aiden Nibali 36 Oct 30, 2022
Official and maintained implementation of the paper "OSS-Net: Memory Efficient High Resolution Semantic Segmentation of 3D Medical Data" [BMVC 2021].

OSS-Net: Memory Efficient High Resolution Semantic Segmentation of 3D Medical Data Christoph Reich, Tim Prangemeier, Özdemir Cetin & Heinz Koeppl | Pr

Christoph Reich 23 Sep 21, 2022