CVPR 2021 - Official code repository for the paper: On Self-Contact and Human Pose.

Overview

TUCH

This repo is part of our project: On Self-Contact and Human Pose.
[Project Page] [Paper] [MPI Project Page]

Teaser SMPLify-XMC

License

Software Copyright License for non-commercial scientific research purposes. Please read carefully the following terms and conditions and any accompanying documentation before you download and/or use the TUCH data and software, (the "Data & Software"), including 3D meshes, images, videos, textures, software, scripts, and animations. By downloading and/or using the Data & Software (including downloading, cloning, installing, and any other use of the corresponding github repository), you acknowledge that you have read these terms and conditions, understand them, and agree to be bound by them. If you do not agree with these terms and conditions, you must not download and/or use the Data & Software. Any infringement of the terms of this agreement will automatically terminate your rights under this License.

Description and Demo

TUCH is a network that regresses human pose and shape, while handling self-contact. The network has the same design as SPIN, but uses new loss terms, that encourage self-contact and resolve intersections.

TUCH result
TUCH fits for two poses with self-contact.

Installation

1) Clone this repo

git clone [email protected]:muelea/tuch.git
cd tuch

32) Create python virtual environment and install requirements

mkdir .venv
python3.6 -m venv .venv/tuch
source .venv/tuch/bin/activate
pip install -r requirements.txt --no-cache-dir

The torchgeometry package uses (1 - bool tensor) statement, which is not supported. Since we try to invert a mask, we can exchange lines 301 - 304 in .venv/tuch/lib/python3.6/site-packages/torchgeometry/core/conversions.py,

FROM: 
    mask_c0 = mask_d2 * mask_d0_d1
    mask_c1 = mask_d2 * (1 - mask_d0_d1)
    mask_c2 = (1 - mask_d2) * mask_d0_nd1
    mask_c3 = (1 - mask_d2) * (1 - mask_d0_nd1)
TO:
    mask_c0 = mask_d2 * mask_d0_d1
    mask_c1 = mask_d2 * (~mask_d0_d1)
    mask_c2 = (~mask_d2) * mask_d0_nd1
    mask_c3 = (~mask_d2) * (~mask_d0_nd1)

3) Download the SMPL body model

Get them SMPL body model from https://smpl.is.tue.mpg.de and save it under SMPL_DIR. ln -s SMPL_DIR data/models/smpl

4) Download SPIN and TUCH model

Downlaod the SPIN and TUCH model and save it in data/

chmod 700 scripts/fetch_data.sh
./scripts/fetch_data.sh

5) Download essentials (necessary to run training code and smplify-dc demo; not necessary for the tuch demo)

Download essentials from here and unpack to METADATA_DIR. Then create symlinks between the essentials and this repo:

ln -s $METADATA_DIR/tuch-essentials data/essentials

6) Download the MTP and DSC datasets (necessary to run training code and smplify-dc demo; not necessary for the tuch demo)

To run TUCH training, please download:

For more information on how to prepare the data read me.

TUCH demo

python demo_tuch.py --checkpoint=data/tuch_model_checkpoint.pt  \
--img data/example_input/img_032.jpg --openpose data/example_input/img_032_keypoints.json \
--outdir data/example_output/demo_tuch

This is the link to the demo image.

SMPLify-DC demo

You can use the following command to run SMPLify-DC on our DSC data, after pre-processing it. See readme for instructions. The output are the initial SPIN estimate (columns 2 and 3) and the SMPLify-DC optimized result (column 4 and 5).

python demo_smplify_dc.py --name smplify_dc --log_dir out/demo_smplify_dc --ds_names dsc_df \
--num_smplify_iters 100

TUCH Training

To select the training data, you can use the --ds_names and --ds_composition flags. ds_names are the short names of each dataset, ds_composition their share per batch. --run_smplify uses DSC annotations when available, otherwise it runs SMPLify-DC without L_D term. If you memory is not sufficient, you can try changing the batch size via the --batch_size flag.

Run TUCH training code:

python train.py --name=tuch --log_dir=out --pretrained_checkpoint=data/spin_model_checkpoint.pt \
  --ds_names dsc mtp --ds_composition 0.5 0.5 \
  --run_smplify --num_smplify_iters=10

For a quick sanity check (no optimization and contact losses) you can finetune on MTP data only without pushing and pulling terms. For this, use mtp data only and set contact_loss_weight=0.0, and remove the optimization flag:

python train.py --name=tuch_mtp_nolplc --log_dir=out/ --pretrained_checkpoint=data/spin_model_checkpoint.pt \
  --ds_names mtp --ds_composition 1.0 \
  --contact_loss_weight=0.0 

To train on different data distributions, pass the dsc dataset names to --ds_names and their share per batch in the same order to --ds_composition. For example,
--ds_names dsc mtp --ds_composition 0.5 0.5 uses 50 % dsc and 50% mtp per batch and
--ds_names dsc mtp --ds_composition 0.3 0.7 uses 30 % dsc and 70% mtp per batch.

TUCH Evaluation

python eval.py --checkpoint=data/tuch_model_checkpoint.pt --dataset=mpi-inf-3dhp
python eval.py --checkpoint=data/tuch_model_checkpoint.pt --dataset=3dpw

EFT + Contact Fitting for DSC data

Training with in-the-loop optimization is slow. You can do Exemplar FineTuning (EFT) with Contact. For this, first process the DSC datasets. Then run:

python fit_eft.py --name tucheft --dsname dsc_lsp
python fit_eft.py --name tucheft --dsname dsc_lspet
python fit_eft.py --name tucheft --dsname dsc_df

Afterwards, you can use the eft datasets similar to the DSC data, just add '_eft' to the dataset name: --ds_names dsc_eft mtp --ds_composition 0.5 0.5 uses 50 % dsc eft and 50% mtp per batch. --ds_names dsc_lsp_eft mtp --ds_composition 0.5 0.5 uses 50 % dsc lsp eft and 50% mtp per batch.

Citation

@inproceedings{Mueller:CVPR:2021,
  title = {On Self-Contact and Human Pose},
  author = {M{\"u}ller, Lea and Osman, Ahmed A. A. and Tang, Siyu and Huang, Chun-Hao P. and Black, Michael J.},
  booktitle = {Proceedings IEEE/CVF Conf.~on Computer Vision and Pattern Recogßnition (CVPR)},
  month = jun,
  year = {2021},
  doi = {},
  month_numeric = {6}
}

Acknowledgement

We thank Nikos Kolotouros and Georgios Pavlakos for publishing the SPIN code: https://github.com/nkolot/SPIN. This has allowed us to build our code on top of it and continue to use important features, such as the prior or optimization. Again, special thanks to Vassilis Choutas for his implementation of the generalized winding numbers and the measurements code. We also thank our data capture and admin team for their help with the extensive data collection on Mechanical Turk and in the Capture Hall. Many thanks to all subjects who contributed to this dataset in the scanner and on the Internet. Thanks to all PS members who proofread the script and did not understand it and the reviewers, who helped improving during the rebuttal. Lea Mueller and Ahmed A. A. Osman thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting them. We thank the wonderful PS department for their questions and support.

Contact

For questions, please contact [email protected]

For commercial licensing (and all related questions for business applications), please contact [email protected].

Owner
Lea Müller
PhD student in the Perceiving Systems Department at the Max Planck Institute for Intelligent Systems in Tübingen, Germany.
Lea Müller
FinEAS: Financial Embedding Analysis of Sentiment 📈

FinEAS: Financial Embedding Analysis of Sentiment 📈 (SentenceBERT for Financial News Sentiment Regression) This repository contains the code for gene

LHF Labs 31 Dec 13, 2022
bespoke tooling for offensive security's Windows Usermode Exploit Dev course (OSED)

osed-scripts bespoke tooling for offensive security's Windows Usermode Exploit Dev course (OSED) Table of Contents Standalone Scripts egghunter.py fin

epi 268 Jan 05, 2023
Keras Realtime Multi-Person Pose Estimation - Keras version of Realtime Multi-Person Pose Estimation project

This repository has become incompatible with the latest and recommended version of Tensorflow 2.0 Instead of refactoring this code painfully, I create

M Faber 769 Dec 08, 2022
Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulation

Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulation Official PyTorch implementation for the paper Look

Rishabh Jangir 20 Nov 24, 2022
Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors, CVPR 2021

Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors Human POSEitioning System (H

Aymen Mir 66 Dec 21, 2022
Styleformer - Official Pytorch Implementation

Styleformer -- Official PyTorch implementation Styleformer: Transformer based Generative Adversarial Networks with Style Vector(https://arxiv.org/abs/

Jeeseung Park 159 Dec 12, 2022
RP-GAN: Stable GAN Training with Random Projections

RP-GAN: Stable GAN Training with Random Projections This repository contains a reference implementation of the algorithm described in the paper: Behna

Ayan Chakrabarti 20 Sep 18, 2021
Storage-optimizer - Identify potintial optimizations on the cloud storage accounts

Storage Optimizer Identify potintial optimizations on the cloud storage accounts

Zaher Mousa 1 Feb 13, 2022
Replication Package for "An Empirical Study of the Effectiveness of an Ensemble of Stand-alone Sentiment Detection Tools for Software Engineering Datasets"

Replication Package for "An Empirical Study of the Effectiveness of an Ensemble of Stand-alone Sentiment Detection Tools for Software Engineering Data

2 Oct 06, 2022
Multi-Object Tracking in Satellite Videos with Graph-Based Multi-Task Modeling

TGraM Multi-Object Tracking in Satellite Videos with Graph-Based Multi-Task Modeling, Qibin He, Xian Sun, Zhiyuan Yan, Beibei Li, Kun Fu Abstract Rece

Qibin He 6 Nov 25, 2022
Exploration-Exploitation Dilemma Solving Methods

Exploration-Exploitation Dilemma Solving Methods Medium article for this repo - HERE In ths repo I implemented two techniques for tackling mentioned t

Aman Mishra 6 Jan 25, 2022
The repository contains reproducible PyTorch source code of our paper Generative Modeling with Optimal Transport Maps, ICLR 2022.

Generative Modeling with Optimal Transport Maps The repository contains reproducible PyTorch source code of our paper Generative Modeling with Optimal

Litu Rout 30 Dec 22, 2022
Apply our monocular depth boosting to your own network!

MergeNet - Boost Your Own Depth Boost custom or edited monocular depth maps using MergeNet Input Original result After manual editing of base You can

Computational Photography Lab @ SFU 142 Dec 17, 2022
Single-step adversarial training (AT) has received wide attention as it proved to be both efficient and robust.

Subspace Adversarial Training Single-step adversarial training (AT) has received wide attention as it proved to be both efficient and robust. However,

15 Sep 02, 2022
Medical Image Segmentation using Squeeze-and-Expansion Transformers

Medical Image Segmentation using Squeeze-and-Expansion Transformers Introduction This repository contains the code of the IJCAI'2021 paper 'Medical Im

askerlee 172 Dec 20, 2022
Lowest memory consumption and second shortest runtime in NTIRE 2022 challenge on Efficient Super-Resolution

FMEN Lowest memory consumption and second shortest runtime in NTIRE 2022 on Efficient Super-Resolution. Our paper: Fast and Memory-Efficient Network T

33 Dec 01, 2022
Combinatorial model of ligand-receptor binding

Combinatorial model of ligand-receptor binding The binding of ligands to receptors is the starting point for many import signal pathways within a cell

Mobolaji Williams 0 Jan 09, 2022
Code for Quantifying Ignorance in Individual-Level Causal-Effect Estimates under Hidden Confounding

🍐 quince Code for Quantifying Ignorance in Individual-Level Causal-Effect Estimates under Hidden Confounding 🍐 Installation $ git clone

Andrew Jesson 19 Jun 23, 2022
[ICLR 2021] HW-NAS-Bench: Hardware-Aware Neural Architecture Search Benchmark

HW-NAS-Bench: Hardware-Aware Neural Architecture Search Benchmark Accepted as a spotlight paper at ICLR 2021. Table of content File structure Prerequi

72 Jan 03, 2023
Probabilistic Cross-Modal Embedding (PCME) CVPR 2021

Probabilistic Cross-Modal Embedding (PCME) CVPR 2021 Official Pytorch implementation of PCME | Paper Sanghyuk Chun1 Seong Joon Oh1 Rafael Sampaio de R

NAVER AI 87 Dec 21, 2022