pytorch implementation of openpose including Hand and Body Pose Estimation.

Overview

pytorch-openpose

pytorch implementation of openpose including Body and Hand Pose Estimation, and the pytorch model is directly converted from openpose caffemodel by caffemodel2pytorch. You could implement face keypoint detection in the same way if you are interested in. Pay attention to that the face keypoint detector was trained using the procedure described in [Simon et al. 2017] for hands.

openpose detects hand by the result of body pose estimation, please refer to the code of handDetector.cpp. In the paper, it states as:

This is an important detail: to use the keypoint detector in any practical situation, 
we need a way to generate this bounding box. 
We directly use the body pose estimation models from [29] and [4], 
and use the wrist and elbow position to approximate the hand location, 
assuming the hand extends 0.15 times the length of the forearm in the same direction.

If anybody wants a pure python wrapper, please refer to my pytorch implementation of openpose, maybe it helps you to implement a standalone hand keypoint detector.

Don't be mean to star this repo if it helps your research.

Getting Started

Install Requriements

Create a python 3.7 environement, eg:

conda create -n pytorch-openpose python=3.7
conda activate pytorch-openpose

Install pytorch by following the quick start guide here (use pip) https://download.pytorch.org/whl/torch_stable.html

Install other requirements with pip

pip install -r requirements.txt

Download the Models

*.pth files are pytorch model, you could also download caffemodel file if you want to use caffe as backend.

Download the pytorch models and put them in a directory named model in the project root directory

Run the Demo

Run:

python demo_camera.py

to run a demo with a feed from your webcam or run

python demo.py

to use a image from the images folder or run

python demo_video.py <video-file>

to process a video file (requires ffmpeg-python).

Todo list

  • convert caffemodel to pytorch.
  • Body Pose Estimation.
  • Hand Pose Estimation.
  • Performance test.
  • Speed up.

Demo

Skeleton

Body Pose Estimation

Hand Pose Estimation

Body + Hand

Video Body

Attribution: this video.

Video Hand

Attribution: this video.

Citation

Please cite these papers in your publications if it helps your research (the face keypoint detector was trained using the procedure described in [Simon et al. 2017] for hands):

@inproceedings{cao2017realtime,
  author = {Zhe Cao and Tomas Simon and Shih-En Wei and Yaser Sheikh},
  booktitle = {CVPR},
  title = {Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields},
  year = {2017}
}

@inproceedings{simon2017hand,
  author = {Tomas Simon and Hanbyul Joo and Iain Matthews and Yaser Sheikh},
  booktitle = {CVPR},
  title = {Hand Keypoint Detection in Single Images using Multiview Bootstrapping},
  year = {2017}
}

@inproceedings{wei2016cpm,
  author = {Shih-En Wei and Varun Ramakrishna and Takeo Kanade and Yaser Sheikh},
  booktitle = {CVPR},
  title = {Convolutional pose machines},
  year = {2016}
}
Issues
  • Add script to process video files

    Add script to process video files

    I wrote a script to process video files and add the pose annotations. It processes frame by frame and it's not very fast but it works. Also, I had to add a line for numerical stability in src/body.py because I hit zero division errors.

    opened by gngdb 3
  • ZeroDivisionError: float division by zero in body.py

    ZeroDivisionError: float division by zero in body.py

    Thanks for your great work!

    When I run the demo.py with a change of test_image = 'images/090932040.jpg', there is an error :

    body.py:123: RuntimeWarning: invalid value encountered in true_divide
      vec = np.divide(vec, norm)
    Traceback (most recent call last):
      File "demo.py", line 16, in <module>
        candidate, subset = body_estimation(oriImg)
      File "src\body.py", line 135, in __call__
        0.5 * oriImg.shape[0] / norm - 1, 0)
    ZeroDivisionError: float division by zero
    

    090932040

    This is the image I used. Could you help me with the error?

    opened by hankroldd 3
  • Is this available for commercial work?

    Is this available for commercial work?

    Hi, Appreciations for such a nice work!!

    I was looking this project and noticed that you didn't add any license to this project. I want to know whether this can be used for commercial projects or not.

    I searched in official Openpose and found out that they don't offer commercial license for free.

    Since your project is a derivative of their work, so it's owned by CMU based on the license of openpose. Could you look this issue from OpenPose repo for more detail?

    Also, we can't use their pretrained model which follows the same license as their code, this is the issue, I'm referring to. Since you converted their Caffe model to PyTorch model, it's kind of derivative of their work only and as per their license, your work is CMU's work if used commercially and CMU's Openpose license apply by default.

    So, this or any other implementation of Openpose can't be used commercially!

    What do you think about this?

    opened by kartikeyporwal 3
  • 3D real-time single-person keypoint detection using 3D triangulation from multiple single views

    3D real-time single-person keypoint detection using 3D triangulation from multiple single views

    I'm especially interested in the 3D real-time single-person keypoint detection using 3D triangulation which is available in the original OpenPose Caffe. I read in the README that you used caffemodel2pytorch for converting this code from Caffe. Could you maybe give some more details on this? So I could also convert the 3D triangulation code from the original repo?

    Thanks in advance!

    opened by JeremyKeusters 2
  • Understading the candiate indexes

    Understading the candiate indexes

    Hi

    Thanks for the great work. Can you tell me what are 3rd and 4th column in the candidate vector returned from body_estimation() model. The first two are x and y coordinates I guess.

    opened by saniazahan 1
  • ZeroDivisionError: float division by zero in body.py

    ZeroDivisionError: float division by zero in body.py

    RuntimeWarning: invalid value encountered in true_divide vec = np.divide(vec, norm) Traceback (most recent call last): File "demo.py", line 16, in candidate, subset = body_estimation(oriImg) File "src\body.py", line 135, in call 0.5 * oriImg.shape[0] / norm - 1, 0) ZeroDivisionError: float division by zero

    opened by Adeel-Intizar 1
  • Error in hand tracking on video

    Error in hand tracking on video

    Hey i am running into a very weird issue where i used the model to estimate poses from a video stream it works on the first frame but fails after that with this error..

    label_img, label_numbers = label(binary, return_num=True, connectivity = binary.ndim)
    TypeError: 'str' object is not callable
    

    Any guidance you can provide on the most probable cause of the error would be very helpful thanks!

    opened by mehulsuresh 1
  • 样例代码里面,有个candidate字段无法解析。

    样例代码里面,有个candidate字段无法解析。

    https://www.paddlepaddle.org.cn/hubdetail?name=openpose_body_estimation&en_category=KeyPointDetection

    人体关键点检测的服务部署部分代码,

    canvas = base64_to_cv2(r.json()["results"]['data']) cv2.imwrite('keypoint_body.png', canvas)

    确实可以把data字段解析出来,但是candidate用 base64_to_cv2 解析出来,压根没有可读性。candidate本来表示的是21个关键点的位置,可是用canvas = base64_to_cv2(r.json()["results"]['candidate']) 算出来,数据很奇怪,压根不是预测出来的关键点的位置序列。

    各位大佬,是如何解析出来这个 r.json()["results"]['candidate'] 字段并且可解释里面字段的含义呢?

    opened by austinxln 0
  • Add section to readme how to setup the environment and improve imports

    Add section to readme how to setup the environment and improve imports

    Hi,

    thank you for your repository!

    I added info on how to install the requirements and get started with the demo to the readme. I also improved the imports by removing the sys.path.insert() line and adding a __init__.py, so the directory gets recognized as a module by python. I also renamed the python directory to src as it might be confusing when running from python import ...

    Cheers, Fabian

    opened by FabianHertwig 0
  • 'Digraph' object has no attribute '_repr_svg_' in network_graph.ipynb

    'Digraph' object has no attribute '_repr_svg_' in network_graph.ipynb

    1. modify add src. before .model otherwise fail to import bodypose_model and handpose_model from src.model import bodypose_model, handpose_model but when run

    bodymodel = bodypose_model() hl.build_graph(bodymodel, torch.zeros([1, 3, 368, 368]))

    it occurs error:

    AttributeError Traceback (most recent call last)

    ~/miniconda3/envs/yolov5/lib/python3.9/site-packages/IPython/core/formatters.py in call(self, obj) 343 method = get_real_method(obj, self.print_method) 344 if method is not None: --> 345 return method() 346 return None 347 else:

    ~/miniconda3/envs/yolov5/lib/python3.9/site-packages/hiddenlayer/graph.py in repr_svg(self) 357 def repr_svg(self): 358 """Allows Jupyter notebook to render the graph automatically.""" --> 359 return self.build_dot().repr_svg() 360 361 def save(self, path, format="pdf"):

    AttributeError: 'Digraph' object has no attribute 'repr_svg'

    12 <hiddenlayer.graph.Graph at 0x7fe8c272be50>

    opened by justinge 0
  • How can I get one person date,not multi?

    How can I get one person date,not multi?

    I found that the FPS is just 1 ot 2, so I want to get one person date only. Maybe it will increase FPS? If it doesn't work, what should I do to increase?

    opened by GYPgalaxy 7
  • How do I increase GPU utilization?

    How do I increase GPU utilization?

    First, thank you for your contribution. When I was detecting the video, each frame cost 0.7s~1.7s. I observed that the GPU utilization was only about 10%, while the CPU100%.

    opened by notsong 1
Owner
Hzzone
To be talented & positive.
Hzzone
FrankMocap: A Strong and Easy-to-use Single View 3D Hand+Body Pose Estimator

FrankMocap pursues an easy-to-use single view 3D motion capture system developed by Facebook AI Research (FAIR). FrankMocap provides state-of-the-art 3D pose estimation outputs for body, hand, and body+hands in a single system. The core objective of FrankMocap is to democratize the 3D human pose estimation technology, enabling anyone (researchers, engineers, developers, artists, and others) can easily obtain 3D motion capture outputs from videos and images.

Facebook Research 1.6k Apr 21, 2022
Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors, CVPR 2021

Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors Human POSEitioning System (H

Aymen Mir 62 Apr 20, 2022
HPRNet: Hierarchical Point Regression for Whole-Body Human Pose Estimation

HPRNet: Hierarchical Point Regression for Whole-Body Human Pose Estimation Official PyTroch implementation of HPRNet. HPRNet: Hierarchical Point Regre

Nermin Samet 38 Apr 22, 2022
Demo for Real-time RGBD-based Extended Body Pose Estimation paper

Real-time RGBD-based Extended Body Pose Estimation This repository is a real-time demo for our paper that was published at WACV 2021 conference The ou

Renat Bashirov 81 Apr 18, 2022
A gesture recognition system powered by OpenPose, k-nearest neighbours, and local outlier factor.

OpenHands OpenHands is a gesture recognition system powered by OpenPose, k-nearest neighbours, and local outlier factor. Currently the system can iden

Paul Treanor 12 Jan 10, 2022
Expressive Body Capture: 3D Hands, Face, and Body from a Single Image

Expressive Body Capture: 3D Hands, Face, and Body from a Single Image [Project Page] [Paper] [Supp. Mat.] Table of Contents License Description Fittin

Vassilis Choutas 1.1k Apr 21, 2022
Full body anonymization - Realistic Full-Body Anonymization with Surface-Guided GANs

Realistic Full-Body Anonymization with Surface-Guided GANs This is the official

Håkon Hukkelås 21 Apr 19, 2022
WormMovementSimulation - 3D Simulation of Worm Body Movement with Neurons attached to its body

Generate 3D Locomotion Data This module is intended to create 2D video trajector

null 2 Feb 3, 2022
Re-implementation of the Noise Contrastive Estimation algorithm for pyTorch, following "Noise-contrastive estimation: A new estimation principle for unnormalized statistical models." (Gutmann and Hyvarinen, AISTATS 2010)

Noise Contrastive Estimation for pyTorch Overview This repository contains a re-implementation of the Noise Contrastive Estimation algorithm, implemen

Denis Emelin 40 Nov 14, 2021
SE3 Pose Interp - Interpolate camera pose or trajectory in SE3, pose interpolation, trajectory interpolation

SE3 Pose Interpolation Pose estimated from SLAM system are always discrete, and

Ran Cheng 2 Feb 18, 2022
Simple Pose: Rethinking and Improving a Bottom-up Approach for Multi-Person Pose Estimation

SimplePose Code and pre-trained models for our paper, “Simple Pose: Rethinking and Improving a Bottom-up Approach for Multi-Person Pose Estimation”, a

Jia Li 239 Apr 7, 2022
Repository for the paper "PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation", CVPR 2021.

PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation Code repository for the paper: PoseAug: A Differentiable Pose Augme

Pyjcsx 280 Apr 21, 2022
This repository contains codes of ICCV2021 paper: SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation

SO-Pose This repository contains codes of ICCV2021 paper: SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation This paper is basically an

shangbuhuan 46 Apr 24, 2022
Python scripts for performing 3D human pose estimation using the Mobile Human Pose model in ONNX.

Python scripts for performing 3D human pose estimation using the Mobile Human Pose model in ONNX.

Ibai Gorordo 75 Apr 9, 2022
A hand tracking demo made with mediapipe where you can control lights with pinching your fingers and moving your hand up/down.

HandTrackingBrightnessControl A hand tracking demo made with mediapipe where you can control lights with pinching your fingers and moving your hand up

Teemu Laurila 19 Feb 12, 2022
MohammadReza Sharifi 21 Apr 28, 2022
Web service for facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation based on OpenFace 2.0

OpenGaze: Web Service for OpenFace Facial Behaviour Analysis Toolkit Overview OpenFace is a fantastic tool intended for computer vision and machine le

Sayom Shakib 2 Mar 26, 2022
OpenFace – a state-of-the art tool intended for facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation.

OpenFace 2.2.0: a facial behavior analysis toolkit Over the past few years, there has been an increased interest in automatic facial behavior analysis

Tadas Baltrusaitis 5.6k Apr 29, 2022
Hand-distance-measurement-game - Hand Distance Measurement Game

Hand Distance Measurement Game This is program is made to calculate the distance

Priyansh 2 Jan 12, 2022