EasyMocap is an open-source toolbox for markerless human motion capture from RGB videos.

Overview

EasyMocap

EasyMocap is an open-source toolbox for markerless human motion capture from RGB videos.

In this project, we provide the basic code for fitting SMPL[1]/SMPL+H[2]/SMPLX[3] model to capture body+hand+face poses from multiple views.

Input ✔️ Skeleton ✔️ SMPL
input repro smpl

We plan to intergrate more interesting algorithms, please stay tuned!

  1. [CVPR19] Multi-Person from Multiple Views
  2. [ECCV20] Mocap from Multiple Uncalibrated and Unsynchronized Videos
  3. Dense Reconstruction and View Synthesis from Sparse Views

Installation

1. Download SMPL models

This step is the same as smplx.

To download the SMPL model go to this (male and female models, version 1.0.0, 10 shape PCs) and this (gender neutral model) project website and register to get access to the downloads section.

To download the SMPL+H model go to this project website and register to get access to the downloads section.

To download the SMPL-X model go to this project website and register to get access to the downloads section.

Place them as following:

data
└── smplx
    ├── J_regressor_body25.npy
    ├── J_regressor_body25_smplh.txt
    ├── J_regressor_body25_smplx.txt
    ├── smpl
    │   ├── SMPL_FEMALE.pkl
    │   ├── SMPL_MALE.pkl
    │   └── SMPL_NEUTRAL.pkl
    ├── smplh
    │   ├── MANO_LEFT.pkl
    │   ├── MANO_RIGHT.pkl
    │   ├── SMPLH_FEMALE.pkl
    │   └── SMPLH_MALE.pkl
    └── smplx
        ├── SMPLX_FEMALE.pkl
        ├── SMPLX_MALE.pkl
        └── SMPLX_NEUTRAL.pkl

2. Requirements

  • torch==1.4.0
  • torchvision==0.5.0
  • opencv-python
  • pyrender: for visualization
  • chumpy: for loading SMPL model
  • OpenPose[4]: for 2D pose

Some of python libraries can be found in requirements.txt. You can test different version of PyTorch.

Quick Start

We provide an example multiview dataset[dropbox][BaiduDisk(vg1z)], which has 800 frames from 23 synchronized and calibrated cameras. After downloading the dataset, you can run the following example scripts.

data=path/to/data
out=path/to/output
# 0. extract the video to images
python3 scripts/preprocess/extract_video.py ${data}
# 1. example for skeleton reconstruction
python3 code/demo_mv1pmf_skel.py ${data} --out ${out} --vis_det --vis_repro --undis --sub_vis 1 7 13 19
# 2.1 example for SMPL reconstruction
python3 code/demo_mv1pmf_smpl.py ${data} --out ${out} --end 300 --vis_smpl --undis --sub_vis 1 7 13 19 --gender male
# 2.2 example for SMPL-X reconstruction
python3 code/demo_mv1pmf_smpl.py ${data} --out ${out} --undis --body bodyhandface --sub_vis 1 7 13 19 --start 400 --model smplx --vis_smpl --gender male
# 3.1 example for rendering SMPLX to ${out}/smpl
python3 code/vis_render.py ${data} --out ${out} --skel ${out}/smpl --model smplx --gender male --undis --start 400 --sub_vis 1
# 3.2 example for rendering skeleton of SMPL to ${out}/smplskel
python3 code/vis_render.py ${data} --out ${out} --skel ${out}/smpl --model smplx --gender male --undis --start 400 --sub_vis 1 --type smplskel --body bodyhandface

Not Quick Start

0. Prepare Your Own Dataset

zju-ls-feng
├── intri.yml
├── extri.yml
└── videos
    ├── 1.mp4
    ├── 2.mp4
    ├── ...
    ├── 8.mp4
    └── 9.mp4

The input videos are placed in videos/.

Here intri.yml and extri.yml store the camera intrinsici and extrinsic parameters. For example, if the name of a video is 1.mp4, then there must exist K_1, dist_1 in intri.yml, and R_1((3, 1), rotation vector of camera), T_1(3, 1) in extri.yml. The file format is following OpenCV format.

1. Run OpenPose

data=path/to/data
out=path/to/output
python3 scripts/preprocess/extract_video.py ${data} --openpose <openpose_path> --handface
  • --openpose: specify the openpose path
  • --handface: detect hands and face keypoints

2. Run the code

# 1. example for skeleton reconstruction
python3 code/demo_mv1pmf_skel.py ${data} --out ${out} --vis_det --vis_repro --undis --sub_vis 1 7 13 19
# 2. example for SMPL reconstruction
python3 code/demo_mv1pmf_smpl.py ${data} --out ${out} --end 300 --vis_smpl --undis --sub_vis 1 7 13 19

The input flags:

  • --undis: use to undistort the images
  • --start, --end: control the begin and end number of frames.

The output flags:

  • --vis_det: visualize the detection
  • --vis_repro: visualize the reprojection
  • --sub_vis: use to specify the views to visualize. If not set, the code will use all views
  • --vis_smpl: use to render the SMPL mesh to images.

3. Output

The results are saved in json format.

<output_root>
├── keypoints3d
│   ├── 000000.json
│   └── xxxxxx.json
└── smpl
    ├── 000000.jpg
    ├── 000000.json
    └── 000004.json

The data in keypoints3d/000000.json is a list, each element represents a human body.

{
    'id': <id>,
    'keypoints3d': [[x0, y0, z0, c0], [x1, y1, z0, c1], ..., [xn, yn, zn, cn]]
}

The data in smpl/000000.json is also a list, each element represents the SMPL parameters which is slightly different from official model.

{
    "id": <id>,
    "Rh": <(1, 3)>,
    "Th": <(1, 3)>,
    "poses": <(1, 72/78/87)>,
    "expression": <(1, 10)>,
    "shapes": <(1, 10)>
}

We set the first 3 dimensions of poses to zero, and add a new parameter Rh to represents the global oritentation, the vertices of SMPL model V = RX(theta, beta) + T.

If you use SMPL+H model, the poses contains 22x3+6+6. We use 6 pca coefficients for each hand. 3(jaw, left eye, right eye)x3 poses of head are added for SMPL-X model.

Evaluation

In our code, we do not set the best weight parameters, you can adjust these according your data. If you find a set of good weights, feel free to tell me.

We will add more quantitative reports in doc/evaluation.md

Acknowledgements

Here are the great works this project is built upon:

  • SMPL models and layer are from MPII SMPL-X model.
  • Some functions are borrowed from SPIN, VIBE, SMPLify-X
  • The method for fitting 3D skeleton and SMPL model is similar to TotalCapture, without using point cloud.

We also would like to thank Wenduo Feng who is the performer in the sample data.

Contact

Please open an issue if you have any questions.

Citation

This project is a part of our work iMocap and Neural Body

Please consider citing these works if you find this repo is useful for your projects.

@inproceedings{dong2020motion,
  title={Motion capture from internet videos},
  author={Dong, Junting and Shuai, Qing and Zhang, Yuanqing and Liu, Xian and Zhou, Xiaowei and Bao, Hujun},
  booktitle={European Conference on Computer Vision},
  pages={210--227},
  year={2020},
  organization={Springer}
}

@article{peng2020neural,
  title={Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans},
  author={Peng, Sida and Zhang, Yuanqing and Xu, Yinghao and Wang, Qianqian and Shuai, Qing and Bao, Hujun and Zhou, Xiaowei},
  journal={arXiv preprint arXiv:2012.15838},
  year={2020}
}

Reference

[1] Loper, Matthew, et al. "SMPL: A skinned multi-person linear model." ACM transactions on graphics (TOG) 34.6 (2015): 1-16.
[2] Romero, Javier, Dimitrios Tzionas, and Michael J. Black. "Embodied hands: Modeling and capturing hands and bodies together." ACM Transactions on Graphics (ToG) 36.6 (2017): 1-17.
[3] Pavlakos, Georgios, et al. "Expressive body capture: 3d hands, face, and body from a single image." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
Bogo, Federica, et al. "Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image." European conference on computer vision. Springer, Cham, 2016.
[4] Cao, Z., Hidalgo, G., Simon, T., Wei, S.E., Sheikh, Y.: Openpose: real-time multi-person 2d pose estimation using part affinity fields. arXiv preprint arXiv:1812.08008 (2018)
Issues
  • Hand/finger pose estimation

    Hand/finger pose estimation

    Hi, I saw that openpose does this pose estimation, but, from what I saw, doesnt seems that keypose3d has this points. Is is difficult to add there?

    opened by carlosedubarreto 22
  • mode openpose change yolo-hrnet  error

    mode openpose change yolo-hrnet error

    When I ran the script extract_video-.py, I changed the mode default openpose to yolo-hrnet and this error occurred. I'm not sure if this is caused by missing files 54

    opened by zhanghongyong123456 21
  • The output is all zeros

    The output is all zeros

    Hi. After rendering, the output keypoints are all 0. Is there any possible error that could cause this problem?

    opened by BoMingZhao 18
  • Converting Output SMPL  to the  standard SMPL parameters

    Converting Output SMPL to the standard SMPL parameters

    Hi, As it is mentioned in thedocumentation, the output representation of SMPL is slightly different from the original SMPL parameters, is there any way to convert output SMPL with Rh and Th to the standard SMPL model parameters? Thanks very much

    opened by leilaUEA 18
  • Conversion to FBX

    Conversion to FBX

    I saw the great work you all done to have the bvh conversion working.

    here is my approach to convert to FBX adapting the code from VIBE mocap:

    install pandas pip install pandas

    install bpy via pip pip install bpy

    if pip install bpy doesnt work, install using conda (if you have conda installed) conda install -c kitsune.one python-blender

    copy blender 2.82 files to the python executable. If you use conda on the base enviroment or virtual enviroment, you can fid tha directory using

    conda env list

    Copy the 2.82 folder from blender download nd paste to the enviroment you are using.

    and save the "fbx_output_EasyMocap" at this folder:"scripts/postprocess"

    and finish doing these instructions of downloading the SMPL for unity https://github.com/mkocabas/VIBE#fbx-and-gltf-output-new-feature

    then you can test using the demo video file

    python code/demo_mv1pmf_smpl.py demo_test --out out_demo_test --end 300 --vis_smpl --undis --sub_vis 1 7 13 19 --gender male

    then test the fbx conversion with python scripts/postprocess/fbx_output_EasyMocap.py --input out_demo_test/smpl/ --output out_demo_test/easymocap.fbx --fps_source 30 --fps_target 30 --gender male

    The fbx conversion script: fbx_output_EasyMocap.zip

    opened by carlosedubarreto 16
  • Different cameras

    Different cameras

    Hello,

    I was wondering if you can use different camera models for each camera, or do they all have to be the same model?

    Thanks!

    opened by missphyrgames 16
  • Keypoints annotator release

    Keypoints annotator release

    Hello,

    thanks for the amazing work!

    Are you planning to release the keypoints annotator any time soon?

    opened by tfederico 15
  • adding tabulate

    adding tabulate

    adding package tabulate, needed for the calibration script

    opened by carlosedubarreto 15
  • Badly fitted SMPL model

    Badly fitted SMPL model

    null

    opened by tejaswivg 14
  • Visualisizer with SMPLX-model possible?

    Visualisizer with SMPLX-model possible?

    Hello, after trying different options inside of o3d_scene.yml i could only get the skeleton model to run. I assume i must change some arguments for:

    body_model: module: "easymocap.visualize.skelmodel.SkelModel" args: body_type: "body25" <- bodyhand? (but its not working, i was checking config.py for this but i guess i made some mistakes) joint_radius: 0.02 gender: "neutral" model_type: "smpl" <- smplh or smplx? (not working either)

    On your example-page there is an animated fullbody-geo (smplx?) shown so i guess its possible to visualize a whole body right? Thanks again ...

    opened by FrankSpalteholz 10
  • easymocap work in unity game engine

    easymocap work in unity game engine

    Eaymocap is real good solution for motion capture. can it will easily run in unity game engine for games.

    opened by kapilmathur123 0
  • [Question] Did you support MVPose (CVPR 2019) in PyTorch here?

    [Question] Did you support MVPose (CVPR 2019) in PyTorch here?

    Thanks for your great work! I was wondering whether you have implemented your work MVPose on CVPR 2019 in the PyTorch version? If you have done, I will be very glad if you can offer the source code! Thanks agan.

    opened by liqikai9 1
  • Mediapipe predict strange results

    Mediapipe predict strange results

    微信截图_20211228135835 But Openpose succeed...
    opened by JACKYLUO1991 1
  • Some data are missing in the  Mirrored-Human dataset

    Some data are missing in the Mirrored-Human dataset

    I find that the provided mirrored-human dataset is incompleted, the some of the videos and frame annotations are missing, how can I get them?

    opened by puppet101 2
  • Lightstage SMPLX parameters quality?

    Lightstage SMPLX parameters quality?

    Hello!

    First of all thanks so much for releasing code and data to the research community, awesome work!

    I have dowloaded pre-processed sequences from https://chingswy.github.io/Dataset-Demo/ (LightStage). When I visualise SMPLX parameters, however, quality seems inferior to what shown in the preview videos on the website (see attached visualisations, especially for hands).

    Screenshot 2021-12-16 at 6 03 32 PM Screenshot 2021-12-16 at 6 03 40 PM

    Is this intended, or am I doing something wrong when loading the provided SMPLX parameters?

    Thanks!

    opened by edoRemelli 2
  • How to visualize the smpl reprojection results for mvmp

    How to visualize the smpl reprojection results for mvmp

    Thanks for your perfect work. I would like to ask how to visualize the smpl reprojection results for mvmp. The links is failed in the mvmp webpage.

    Looking forward to your reply. Thx~

    opened by Xianjin111 7
  • Twisting occuring

    Twisting occuring

    hi there

    This model works well, but I have found that there is some 'twisting' that happens when the mesh twists 360 degrees. I suspect that more cameras solves this (in the example attached I'm using 4 cameras)

    Are there any known solutions for this?

    Thanks

    https://user-images.githubusercontent.com/38970401/145704581-94de841f-4232-44c1-8b95-f3fe9d7351aa.mp4

    opened by jamalknight 0
  • question about smpl issue (smpl twisted in some frames)

    question about smpl issue (smpl twisted in some frames)

    I have used a 3 camera dataset as an input, the smpl model in most of the frames are fine, in some frames it get twisted, does it happen because of openpose problem and occolusion?I wonder if we have two camera on each side (i.e. 5 cameras instead of 3) that would solve the issue?

    here is smpl reprojection of continuous frames, the left arm openpose key points are miscalculated in the second camera for few frames

    000438 000439 000440 000441 000442 000443 000444 000445 000446 000447 000448

    000438 000439 000440 000441 000442 000443 000444 000445 000446 000447 000448

    opened by leilaUEA 5
  • some problems about mvmp test

    some problems about mvmp test

    Hi, thanks for sharing this wonderful work. I have test the mv1p success on my own datasets, but for mvmp, the repro can only reproject one person. I use the same camera parameters to test only one person, it works fine, so it seems the intri or extrin parameters are right. Hope you can give some advice,thanks~

    opened by visonpon 2
  • RuntimeError: The size of tensor a (42) must match the size of tensor b (0) at non-singleton dimension 1

    RuntimeError: The size of tensor a (42) must match the size of tensor b (0) at non-singleton dimension 1

    Hi, I want to run the SMPLX model with body25 however I receive the following error. Also, I deleted the keypoints3d folder and run the code again but it did not help.

    `python3 apps/demo/mv1p.py ${data} --out ${data}/output/smplx --vis_det --vis_repro --undis --body body25 --model smplx --gender male --vis_smpl

    Demo code for multiple views and one person:

    - Input : seq => 1, 2, 3
    - Output: seq/output/smplx
    - Body  : smplx=>male, body25
    

    loading: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4756/4756 [00:06<00:00, 695.93it/s] -> [Optimize global RT ]: 51.0s -> [Optimize 3D Pose/4756 frames]: 352.4s Traceback (most recent call last): File "apps/demo/mv1p.py", line 117, in mv1pmf_smpl(dataset, args) File "apps/demo/mv1p.py", line 69, in mv1pmf_smpl params = smpl_from_keypoints3d2d(body_model, kp3ds, keypoints2d, bboxes, File "/Users/user/Desktop/KTH_RE/EasyMocap-master/easymocap/pipeline/basic.py", line 77, in smpl_from_keypoints3d2d params = multi_stage_optimize(body_model, params, kp3ds, kp2ds, bboxes, Pall, weight_pose, cfg) File "/Users/user/Desktop/KTH_RE/EasyMocap-master/easymocap/pipeline/basic.py", line 29, in multi_stage_optimize params = optimizePose3D(body_model, params, kp3ds, weight=weight, cfg=cfg) File "/Users/user/Desktop/KTH_RE/EasyMocap-master/easymocap/pyfitting/optimize_simple.py", line 301, in optimizePose3D params = _optimizeSMPL(body_model, params, prepare_funcs, postprocess_funcs, loss_funcs, weight_loss=weight, cfg=cfg) File "/Users/user/Desktop/KTH_RE/EasyMocap-master/easymocap/pyfitting/optimize_simple.py", line 246, in _optimizeSMPL final_loss = fitting.run_fitting(optimizer, closure, opt_params) File "/Users/user/Desktop/KTH_RE/EasyMocap-master/easymocap/pyfitting/optimize.py", line 38, in run_fitting loss = optimizer.step(closure) File "/Users/user/Desktop/KTH_RE/EasyMocap-master/easymocap/pyfitting/lbfgs.py", line 307, in step orig_loss = closure() File "/Users/user/Desktop/KTH_RE/EasyMocap-master/easymocap/pyfitting/optimize_simple.py", line 231, in closure loss_dict = {key:func(kpts_est=kpts_est, **new_params) for key, func in loss_funcs.items()} File "/Users/user/Desktop/KTH_RE/EasyMocap-master/easymocap/pyfitting/optimize_simple.py", line 231, in loss_dict = {key:func(kpts_est=kpts_est, **new_params) for key, func in loss_funcs.items()} File "/Users/user/Desktop/KTH_RE/EasyMocap-master/easymocap/pyfitting/lossfactory.py", line 63, in hand diff_square = (kpts_est[:, 25:25+42, :3] - self.keypoints3d[:, 25:25+42, :3])*self.conf[:, 25:25+42] RuntimeError: The size of tensor a (42) must match the size of tensor b (0) at non-singleton dimension 1`

    Do you have any suggestions for me to solve it?

    opened by mertmerci 0
Releases(v0.1)
  • v0.1(Mar 29, 2021)

    The basic version of EasyMocap contains:

    1. 3D reconstruction of SMPL/SMPL+H/SMPL-X model from multiple views
    2. Visualization of 3D skeletons and human meshes
    3. Conversion to bvh format
    4. Camera's extrinsic parameters calibration
    Source code(tar.gz)
    Source code(zip)
Owner
ZJU3DV
ZJU3DV is a research group of State Key Lab of CAD&CG, Zhejiang University, which maily focuses on the research of 3D computer vision, SLAM and AR.
ZJU3DV
dataset for ECCV 2020 "Motion Capture from Internet Videos"

Motion Capture from Internet Videos Motion Capture from Internet Videos Junting Dong*, Qing Shuai*, Yuanqing Zhang, Xian Liu, Xiaowei Zhou, Hujun Bao

ZJU3DV 82 Dec 28, 2021
Exploring Versatile Prior for Human Motion via Motion Frequency Guidance (3DV2021)

Exploring Versatile Prior for Human Motion via Motion Frequency Guidance This is the codebase for video-based human motion reconstruction in human-mot

Jiachen Xu 3 Dec 9, 2021
DSAC* for Visual Camera Re-Localization (RGB or RGB-D)

DSAC* for Visual Camera Re-Localization (RGB or RGB-D) Introduction Installation Data Structure Supported Datasets 7Scenes 12Scenes Cambridge Landmark

Visual Learning Lab 102 Jan 12, 2022
3DMV jointly combines RGB color and geometric information to perform 3D semantic segmentation of RGB-D scans.

3DMV 3DMV jointly combines RGB color and geometric information to perform 3D semantic segmentation of RGB-D scans. This work is based on our ECCV'18 p

Владислав Молодцов 1 Oct 26, 2021
PhysCap: Physically Plausible Monocular 3D Motion Capture in Real Time

PhysCap: Physically Plausible Monocular 3D Motion Capture in Real Time The implementation is based on SIGGRAPH Aisa'20. Dependencies Python 3.7 Ubuntu

soratobtai 47 Jan 16, 2022
A real-time motion capture system that estimates poses and global translations using only 6 inertial measurement units

TransPose Code for our SIGGRAPH 2021 paper "TransPose: Real-time 3D Human Translation and Pose Estimation with Six Inertial Sensors". This repository

Xinyu Yi 127 Jan 13, 2022
Motion and Shape Capture from Sparse Markers

MoSh++ This repository contains the official chumpy implementation of mocap body solver used for AMASS: AMASS: Archive of Motion Capture as Surface Sh

Nima Ghorbani 71 Jan 10, 2022
A minimal solution to hand motion capture from a single color camera at over 100fps. Easy to use, plug to run.

Minimal Hand A minimal solution to hand motion capture from a single color camera at over 100fps. Easy to use, plug to run. This project provides the

Yuxiao Zhou 710 Jan 20, 2022
Differential rendering based motion capture blender project.

TraceArmature Summary TraceArmature is currently a set of python scripts that allow for high fidelity motion capture through the use of AI pose estima

William Rodriguez 2 Jan 8, 2022
Towards uncontrained hand-object reconstruction from RGB videos

Towards uncontrained hand-object reconstruction from RGB videos Yana Hasson, Gül Varol, Ivan Laptev and Cordelia Schmid Project page Paper Table of Co

Yana 48 Jan 14, 2022
PoseCamera is python based SDK for human pose estimation through RGB webcam.

PoseCamera PoseCamera is python based SDK for human pose estimation through RGB webcam. Install install posecamera package through pip pip install pos

WonderTree 7 Jul 20, 2021
MMDetection3D is an open source object detection toolbox based on PyTorch

MMDetection3D is an open source object detection toolbox based on PyTorch, towards the next-generation platform for general 3D detection. It is a part of the OpenMMLab project developed by MMLab.

OpenMMLab 2k Jan 14, 2022
AdelaiDepth is an open source toolbox for monocular depth prediction.

AdelaiDepth is an open source toolbox for monocular depth prediction.

Adelaide Intelligent Machines (AIM) Group 436 Jan 20, 2022
LaneDet is an open source lane detection toolbox based on PyTorch that aims to pull together a wide variety of state-of-the-art lane detection models

LaneDet is an open source lane detection toolbox based on PyTorch that aims to pull together a wide variety of state-of-the-art lane detection models. Developers can reproduce these SOTA methods and build their own methods.

TuZheng 240 Jan 18, 2022
MMFlow is an open source optical flow toolbox based on PyTorch

Documentation: https://mmflow.readthedocs.io/ Introduction English | 简体中文 MMFlow is an open source optical flow toolbox based on PyTorch. It is a part

OpenMMLab 391 Jan 12, 2022
An open source object detection toolbox based on PyTorch

MMDetection is an open source object detection toolbox based on PyTorch. It is a part of the OpenMMLab project.

Bo Chen 3 Dec 24, 2021
mmfewshot is an open source few shot learning toolbox based on PyTorch

OpenMMLab FewShot Learning Toolbox and Benchmark

OpenMMLab 319 Jan 20, 2022
✨✨✨An awesome open source toolbox for stereo matching.

OpenStereo This is an awesome open source toolbox for stereo matching. Supported Methods: BM SGM(T-PAMI'07) GCNet(ICCV'17) PSMNet(CVPR'18) StereoNet(E

Wang Qingyu 2 Dec 7, 2021
Mmdetection3d Noted - MMDetection3D is an open source object detection toolbox based on PyTorch

MMDetection3D is an open source object detection toolbox based on PyTorch

Jiangjingwen 2 Dec 24, 2021