(ICONIP 2020) MobileHand: Real-time 3D Hand Shape and Pose Estimation from Color Image

Overview

MobileHand: Real-time 3D Hand Shape and Pose Estimation from Color Image

This repo contains the source code for MobileHand, real-time estimation of 3D hand shape and pose from a single color image running at over 110 Hz on a GPU or 75 Hz on a CPU.

Paper | Project | Video

If you find MobileHand useful for your work, please consider citing

@inproceedings{MobileHand:2020,
  title     = {MobileHand: Real-time 3D Hand Shape and Pose Estimation from Color Image},
  author    = {Guan Ming, Lim and Prayook, Jatesiktat and Wei Tech, Ang},
  booktitle = {27th International Conference on Neural Information Processing (ICONIP)},
  year      = {2020}
}

Setup

The simplest way to run our implementation is to use anaconda and create an environment called mobilehand

conda env create -f environment.yaml
conda activate mobilehand

Next, download MANO right hand model

  • Go to MANO project page
  • Click on Sign In and register for your account
  • Download Models & Code (mano_v1_2.zip)
  • Unzip and copy the file mano_v1_2/models/MANO_RIGHT.pkl into the mobilehand/model folder

Demo

cd code/ # Change directory to the folder `mobilehand/code/`

python demo.py -m image -d stb      # Test on sample image (STB dataset)
python demo.py -m image -d freihand # Test on sample image (FreiHAND dataset)
python demo.py -m video             # Test on sample video
python demo.py -m camera            # Test with webcam
python demo.py -m camera -c         # Add -c to enable GPU processing

Dataset

[2017 ICIP] A Hand Pose Tracking Benchmark from Stereo Matching. [PDF] [Project] [Code]

Jiawei Zhang, Jianbo Jiao, Mingliang Chen, Liangqiong Qu, Xiaobin Xu, and Qingxiong Yang

[ICCV 2019] FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images. [PDF] [Project] [Code]

Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, Thomas Brox

Related works

[CVPR 2019] Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Rendering. [PDF]

Seungryul Baek, Kwang In Kim, Tae-Kyun Kim

[CVPR 2019] 3D Hand Shape and Pose from Images in the Wild. [PDF] [Code]

Adnane Boukhayma, Rodrigo de Bem, Philip H.S. Torr

[CVPR 2019] 3D Hand Shape and Pose Estimation from a Single RGB Image. [PDF] [Project] [Code] (Oral)

Liuhao Ge, Zhou Ren, Yuncheng Li, Zehao Xue, Yingying Wang, Jianfei Cai, Junsong Yuan

[CVPR 2019] Learning joint reconstruction of hands and manipulated objects. [PDF] [Code] [Code] [Project]

Yana Hasson, Gül Varol, Dimitris Tzionas, Igor Kalevatykh, Michael J. Black, Ivan Laptev, and Cordelia Schmid

[ICCV 2019] End-to-end Hand Mesh Recovery from a Monocular RGB Image. [PDF] [Code]

Xiong Zhang*, Qiang Li*, Wenbo Zhang, Wen Zheng

[CVPR 2020] Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild. [PDF] [Project] (Oral)

Dominik Kulon, Riza Alp Güler, Iasonas Kokkinos, Michael Bronstein, Stefanos Zafeiriou

[CVPR 2020] Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data. [PDF] [Project] [Code]

Yuxiao Zhou, Marc Habermann, Weipeng Xu, Ikhsanul Habibie, Christian Theobalt, Feng Xu

Key references

[MVA 2019] Accurate Hand Keypoint Localization on Mobile Devices. [PDF] [Code]

Filippos Gouidis, Paschalis Panteleris, Iason Oikonomidis, Antonis Argyros

[CVPR 2018] End-to-end Recovery of Human Shape and Pose. [PDF] [Project] [Code]

Angjoo Kanazawa, Michael J Black, David W. Jacobs, Jitendra Malik

[SIGGRAPH ASIA 2017] Embodied Hands:Modeling and Capturing Hands and Bodies Together. [PDF] [Project]

Javier Romero, Dimitrios Tzionas, Michael J Black

A strongly-typed genetic programming framework for Python

monkeys "If an army of monkeys were strumming on typewriters they might write all the books in the British Museum." monkeys is a framework designed to

H. Chase Stevens 115 Nov 27, 2022
Image-retrieval-baseline - MUGE Multimodal Retrieval Baseline

MUGE Multimodal Retrieval Baseline This repo is implemented based on the open_cl

47 Dec 16, 2022
Source code for ZePHyR: Zero-shot Pose Hypothesis Rating @ ICRA 2021

ZePHyR: Zero-shot Pose Hypothesis Rating ZePHyR is a zero-shot 6D object pose estimation pipeline. The core is a learned scoring function that compare

R-Pad - Robots Perceiving and Doing 18 Aug 22, 2022
Imagededup - 😎 Finding duplicate images made easy

imagededup is a python package that simplifies the task of finding exact and near duplicates in an image collection.

idealo 4.3k Jan 07, 2023
The official PyTorch code for NeurIPS 2021 ML4AD Paper, "Does Thermal data make the detection systems more reliable?"

MultiModal-Collaborative (MMC) Learning Framework for integrating RGB and Thermal spectral modalities This is the official code for NeurIPS 2021 Machi

NeurAI 12 Nov 02, 2022
Simulation of the solar system using various nummerical methods

solar-system Simulation of the solar system using various nummerical methods Download the repo Make shure matplotlib, scipy etc. are installed execute

Caspar 7 Jul 15, 2022
RIM: Reliable Influence-based Active Learning on Graphs.

RIM: Reliable Influence-based Active Learning on Graphs. This repository is the official implementation of RIM. Requirements To install requirements:

Wentao Zhang 4 Aug 29, 2022
CARMS: Categorical-Antithetic-REINFORCE Multi-Sample Gradient Estimator

CARMS: Categorical-Antithetic-REINFORCE Multi-Sample Gradient Estimator This is the official code repository for NeurIPS 2021 paper: CARMS: Categorica

Alek Dimitriev 1 Jul 09, 2022
Code to reproduce the results for Compositional Attention

Compositional-Attention This repository contains the official implementation for the paper Compositional Attention: Disentangling Search and Retrieval

Sarthak Mittal 58 Nov 30, 2022
MVGCN: a novel multi-view graph convolutional network (MVGCN) framework for link prediction in biomedical bipartite networks.

MVGCN MVGCN: a novel multi-view graph convolutional network (MVGCN) framework for link prediction in biomedical bipartite networks. Developer: Fu Hait

13 Dec 01, 2022
Marine debris detection with commercial satellite imagery and deep learning.

Marine debris detection with commercial satellite imagery and deep learning. Floating marine debris is a global pollution problem which threatens mari

Inter Agency Implementation and Advanced Concepts 56 Dec 16, 2022
Code for testing convergence rates of Lipschitz learning on graphs

📈 LipschitzLearningRates The code in this repository reproduces the experimental results on convergence rates for k-nearest neighbor graph infinity L

2 Dec 20, 2021
Jingju baseline - A baseline model of our project of Beijing opera script generation

Jingju Baseline It is a baseline of our project about Beijing opera script gener

midon 1 Jan 14, 2022
A Nim frontend for pytorch, aiming to be mostly auto-generated and internally using ATen.

Master Release Pytorch - Py + Nim A Nim frontend for pytorch, aiming to be mostly auto-generated and internally using ATen. Because Nim compiles to C+

Giovanni Petrantoni 425 Dec 22, 2022
🦙 LaMa Image Inpainting, Resolution-robust Large Mask Inpainting with Fourier Convolutions, WACV 2022

🦙 LaMa Image Inpainting, Resolution-robust Large Mask Inpainting with Fourier Convolutions, WACV 2022

Advanced Image Manipulation Lab @ Samsung AI Center Moscow 4.7k Dec 31, 2022
Sample Code for "Pessimism Meets Invariance: Provably Efficient Offline Mean-Field Multi-Agent RL"

Sample Code for "Pessimism Meets Invariance: Provably Efficient Offline Mean-Field Multi-Agent RL" This is the official codebase for Pessimism Meets I

3 Sep 19, 2022
A PyTorch implementation of PointRend: Image Segmentation as Rendering

PointRend A PyTorch implementation of PointRend: Image Segmentation as Rendering [arxiv] [Official Implementation: Detectron2] This repo for Only Sema

AhnDW 336 Dec 26, 2022
(NeurIPS '21 Spotlight) IQ-Learn: Inverse Q-Learning for Imitation

Inverse Q-Learning (IQ-Learn) Official code base for IQ-Learn: Inverse soft-Q Learning for Imitation, NeurIPS '21 Spotlight IQ-Learn is an easy-to-use

Divyansh Garg 102 Dec 20, 2022
Deformable DETR is an efficient and fast-converging end-to-end object detector.

Deformable DETR: Deformable Transformers for End-to-End Object Detection.

2k Jan 05, 2023
LaneDetectionAndLaneKeeping - Lane Detection And Lane Keeping

LaneDetectionAndLaneKeeping This project is part of my bachelor's thesis. The go

5 Jun 27, 2022