[CVPR 2022] PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervision (Oral)

Overview

PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervision

Kehong Gong*, Bingbing Li*, Jianfeng Zhang*, Tao Wang*, Jing Huang, Bi Mi, Jiashi Feng, Xinchao Wang

CVPR 2022 (Oral Presentation, arxiv)

Logo

Framework

Pose-triplet contains three components: estimator, imitator and hallucinator

The three components form dual-loop during the training process, complementing and strengthening one another. alt text

Improvement through co-evolving

Here is imitated motion of different rounds, the estimator and imitator get improved over the rounds of training, and thus the imitated motion becomes more accurate and realistic from round 1 to 3. alt text

Video demo

04806-supp.mp4

Comparasion

Here we compared our results with two recent works Yu et al. and Hu et al.

Installation

  • Please refer to README_env.md for the python environment setup.

Data Preparation

Training

Please refer to script-summary for the training process. We also provide a checkpoint folder here with better performance, which support that this framework has the potential to reach the same performance as fully-supervised approaches.
Note: checkpoint for the RL policy is not include due to the size limitation, please following the training code to train the policy.

Inference

We provide an inference code here. Please follow the instruction and download the pretrained model for inference on videos.

Talk

Here is a slidestalk (PPT in english, speak in chinese).

Citation

If you find this code useful for your research, please consider citing the following paper:

@inproceedings{gong2022posetriplet,
  title      = {PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervision},
  author     = {Gong, Kehong and Li, Bingbing and Zhang, Jianfeng and Wang, Tao and Huang, Jing and Mi, Michael Bi and Feng, Jiashi and Wang, Xinchao},
  booktitle  = {CVPR},
  year       = {2022}
}
Issues
  • Can this model inference single image?

    Can this model inference single image?

    I want to find out whether this model can inference online. So does it need the frame from future when it inference current frames? or like the question as title, can this model inference single image?

    opened by tiger990111 8
  • ValueError in Inference

    ValueError in Inference

    When I ran the inference script: python videopose-j16-wild-eval_run.py I got folloing errors. Could you help me?

    -------------- prepare video clip spends 0.03 seconds
    -------------- load keypoint spends 0.05 seconds
    Loading checkpoint ./checkpoint/ckpt_ep_045.bin
    -------------- load 3D model spends 3.81 seconds
    -------------- generate reconstruction 3D data spends 0.53 seconds
    Loading checkpoint ./checkpoint/ckpt_ep_045.bin
    -------------- load 3D Traj model spends 0.16 seconds
    -------------- generate reconstruction 3D data spends 0.02 seconds
    Rendering... save to ./wild_eval/333_scale2D_010/bilibili-clip/kunkun_clip_alpha_pose.mp4
    ===========================> This video get 49 frames in total.
      2%|##2                                                                                                          | 1/49 [00:00<00:10,  4.57it/s]Traceback (most recent call last):
      File "videopose-j16-wild-eval_run.py", line 288, in <module>
        Vis.redering()
      File "videopose-j16-wild-eval_run.py", line 44, in redering
        self.visalizatoin(anim_output)
      File "videopose-j16-wild-eval_run.py", line 232, in visalizatoin
        input_video_skip=args.viz_skip)
      File "/mnt/zhoudeyu/project/save_video/dengyuanzhang/posetriplet/PoseTriplet-main/estimator_inference/common/visualization.py", line 195, in render_animation
        anim.save(output, writer=writer)
      File "/root/miniconda3/envs/alphapose/lib/python3.6/site-packages/matplotlib/animation.py", line 1174, in save
        writer.grab_frame(**savefig_kwargs)
      File "/root/miniconda3/envs/alphapose/lib/python3.6/contextlib.py", line 99, in __exit__
        self.gen.throw(type, value, traceback)
      File "/root/miniconda3/envs/alphapose/lib/python3.6/site-packages/matplotlib/animation.py", line 232, in saving
        self.finish()
      File "/root/miniconda3/envs/alphapose/lib/python3.6/site-packages/matplotlib/animation.py", line 358, in finish
        self.cleanup()
      File "/root/miniconda3/envs/alphapose/lib/python3.6/site-packages/matplotlib/animation.py", line 395, in cleanup
        out, err = self._proc.communicate()
      File "/root/miniconda3/envs/alphapose/lib/python3.6/subprocess.py", line 863, in communicate
        stdout, stderr = self._communicate(input, endtime, timeout)
      File "/root/miniconda3/envs/alphapose/lib/python3.6/subprocess.py", line 1525, in _communicate
        selector.register(self.stdout, selectors.EVENT_READ)
      File "/root/miniconda3/envs/alphapose/lib/python3.6/selectors.py", line 351, in register
        key = super().register(fileobj, events, data)
      File "/root/miniconda3/envs/alphapose/lib/python3.6/selectors.py", line 237, in register
        key = SelectorKey(fileobj, self._fileobj_lookup(fileobj), events, data)
      File "/root/miniconda3/envs/alphapose/lib/python3.6/selectors.py", line 224, in _fileobj_lookup
        return _fileobj_to_fd(fileobj)
      File "/root/miniconda3/envs/alphapose/lib/python3.6/selectors.py", line 39, in _fileobj_to_fd
        "{!r}".format(fileobj)) from None
    ValueError: Invalid file object: <_io.BufferedReader name=30>
      6%|######6                                                                                                      | 3/49 [00:00<00:07,  5.99it/s]
    
    opened by ChawDoe 7
  • Lower body skeleton issue

    Lower body skeleton issue

    Hi I am trying to get the 3 pose on upper body and is there a way to filter or remove lower body keypoints completely. I am getting the following output where the legs are distorted but I don't want lower body joints to be visible in my output.

    Screen Shot 2022-04-25 at 2 23 37 PM
    opened by Abi5678 4
  • The depth info of legs is always bad

    The depth info of legs is always bad

    I found that the depth of leg is always different from other parts, which leads to lean-forward bad case. Like, this is my kpts2d on img, kpt2d

    And this is the result of PoseTriplet, printed with ax.view_init(0, 70) and ax.view_init(0, 0) kpts3d

    So, Have you found this issue when you research in this task? Would you give me some advise to avoid those bad cases?

    opened by tiger990111 1
  • pretrained models of imitator?

    pretrained models of imitator?

    Hi, just wanna ask, how to train on imitator? does there any instructions on how to run? Will there any pretrained weights that can be used to visualize imitator result?

    opened by jinfagang 0
Python scripts for performing 3D human pose estimation using the Mobile Human Pose model in ONNX.

Python scripts for performing 3D human pose estimation using the Mobile Human Pose model in ONNX.

Ibai Gorordo 83 Jun 13, 2022
Repository for the paper "PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation", CVPR 2021.

PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation Code repository for the paper: PoseAug: A Differentiable Pose Augme

Pyjcsx 293 Jun 20, 2022
[CVPR 2022 Oral] EPro-PnP: Generalized End-to-End Probabilistic Perspective-n-Points for Monocular Object Pose Estimation

EPro-PnP EPro-PnP: Generalized End-to-End Probabilistic Perspective-n-Points for Monocular Object Pose Estimation In CVPR 2022 (Oral). [paper] Hanshen

 同济大学智能汽车研究所综合感知研究组 ( Comprehensive Perception Research Group under Institute of Intelligent Vehicles, School of Automotive Studies, Tongji University) 66 Jun 13, 2022
Code for "Human Pose Regression with Residual Log-likelihood Estimation", ICCV 2021 Oral

Human Pose Regression with Residual Log-likelihood Estimation [Paper] [arXiv] [Project Page] Human Pose Regression with Residual Log-likelihood Estima

JeffLi 291 Jun 17, 2022
Mixup for Supervision, Semi- and Self-Supervision Learning Toolbox and Benchmark

OpenSelfSup News Downstream tasks now support more methods(Mask RCNN-FPN, RetinaNet, Keypoints RCNN) and more datasets(Cityscapes). 'GaussianBlur' is

AI Lab, Westlake University 45 Jun 12, 2022
Learning trajectory representations using self-supervision and programmatic supervision.

Trajectory Embedding for Behavior Analysis (TREBA) Implementation from the paper: Jennifer J. Sun, Ann Kennedy, Eric Zhan, David J. Anderson, Yisong Y

null 52 May 15, 2022
Code for "Single-view robot pose and joint angle estimation via render & compare", CVPR 2021 (Oral).

Single-view robot pose and joint angle estimation via render & compare Yann Labbé, Justin Carpentier, Mathieu Aubry, Josef Sivic CVPR: Conference on C

Yann Labbé 48 Jun 12, 2022
Code for "Reconstructing 3D Human Pose by Watching Humans in the Mirror", CVPR 2021 oral

Reconstructing 3D Human Pose by Watching Humans in the Mirror Qi Fang*, Qing Shuai*, Junting Dong, Hujun Bao, Xiaowei Zhou CVPR 2021 Oral The videos a

ZJU3DV 170 May 28, 2022
Code repo for realtime multi-person pose estimation in CVPR'17 (Oral)

Realtime Multi-Person Pose Estimation By Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh. Introduction Code repo for winning 2016 MSCOCO Keypoints Cha

Zhe Cao 4.8k Jun 20, 2022
Code for "PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation" CVPR 2019 oral

Good news! We release a clean version of PVNet: clean-pvnet, including how to train the PVNet on the custom dataset. Use PVNet with a detector. The tr

ZJU3DV 686 Jun 17, 2022
This repository contains codes of ICCV2021 paper: SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation

SO-Pose This repository contains codes of ICCV2021 paper: SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation This paper is basically an

shangbuhuan 46 Apr 24, 2022
[CVPR 2022 Oral] Versatile Multi-Modal Pre-Training for Human-Centric Perception

Versatile Multi-Modal Pre-Training for Human-Centric Perception Fangzhou Hong1  Liang Pan1  Zhongang Cai1,2,3  Ziwei Liu1* 1S-Lab, Nanyang Technologic

Fangzhou Hong 66 Jun 17, 2022
This is an official implementation of our CVPR 2021 paper "Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression" (https://arxiv.org/abs/2104.02300)

Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression Introduction In this paper, we are interested in the bottom-up paradigm of estima

HRNet 302 Jun 17, 2022
This repository is the offical Pytorch implementation of ContextPose: Context Modeling in 3D Human Pose Estimation: A Unified Perspective (CVPR 2021).

Context Modeling in 3D Human Pose Estimation: A Unified Perspective (CVPR 2021) Introduction This repository is the offical Pytorch implementation of

null 33 May 27, 2022
(CVPR 2022 - oral) Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry

Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry Official implementation of the paper Multi-View Depth Est

Bae, Gwangbin 46 Jun 17, 2022
[CVPR 2022] Pytorch implementation of "Templates for 3D Object Pose Estimation Revisited: Generalization to New objects and Robustness to Occlusions" paper

template-pose Pytorch implementation of "Templates for 3D Object Pose Estimation Revisited: Generalization to New objects and Robustness to Occlusions

Van Nguyen Nguyen 60 Jun 21, 2022
CVPR 2021 - Official code repository for the paper: On Self-Contact and Human Pose.

selfcontact This repo is part of our project: On Self-Contact and Human Pose. [Project Page] [Paper] [MPI Project Page] It includes the main function

Lea Müller 61 Jun 11, 2022
CVPR 2021 - Official code repository for the paper: On Self-Contact and Human Pose.

SMPLify-XMC This repo is part of our project: On Self-Contact and Human Pose. [Project Page] [Paper] [MPI Project Page] License Software Copyright Lic

Lea Müller 76 Jun 9, 2022
CVPR 2021 - Official code repository for the paper: On Self-Contact and Human Pose.

TUCH This repo is part of our project: On Self-Contact and Human Pose. [Project Page] [Paper] [MPI Project Page] License Software Copyright License fo

Lea Müller 40 May 7, 2022