Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On

Last update: Dec 24, 2022

Related tags

Overview

Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On

Abstract

We propose a new generative model for 3D garment deformations that enables us to learn, for the first time, a data-driven method for virtual try-on that effectively addresses garment-body collisions. In contrast to existing methods that require an undesirable postprocessing step to fix garment-body interpenetrations at test time, our approach directly outputs 3D garment configurations that do not collide with the underlying body. Key to our success is a new canonical space for garments that removes pose-and-shape deformations already captured by a new diffused human body model, which extrapolates body surface properties such as skinning weights and blendshapes to any 3D point. We leverage this representation to train a generative model with a novel self-supervised collision term that learns to reliably solve garment-body interpenetrations. We extensively evaluate and compare our results with recently proposed data-driven methods, and show that our method is the first to successfully address garment-body contact in unseen body shapes and motions, without compromising realism and detail.

Running the model

Requirements: python3.8, tensorflow-2.2.1, numpy-1.18.5, scipy-1.7.1, chumpy-0.70

Project structure:

vto-garment-collisions
│
└───assets 
|    └─ images    
|    └─ meshes    
|    └─ CMU       # Not included, see instructions
|    └─ SMPL      # Not included, see instructions
| 
└───rendering     # Code to render meshes 
|
└───src           # Code to run the model
| 
└───trained_models      
|    └─ diffused_body  # Networks of the diffused body model (Not included, see instructions)
|    └─ tshirt         # Networks of tshirt model (Not included, see instructions)
│
└───run_model.py

Download trained models

Download models of the diffused human body: https://github.com/isantesteban/vto-garment-collisions/releases/download/trained-models/trained_models_diffused_body.zip
Download models of the garment: https://github.com/isantesteban/vto-garment-collisions/releases/download/tshirt-trained-models/trained_models_tshirt.zip
Create trained_models directory and extract trained_models_diffused_body.zip and trained_models_tshirt.zip there.

Download human model

Sign in into https://smpl.is.tue.mpg.de
Download SMPL version 1.0.0 for Python 2.7 (10 shape PCs)
Extract SMPL_python_v.1.0.0.zip and copy smpl/models/basicModel_f_lbs_10_207_0_v1.0.0.pkl in assets/SMPL

Download animation sequences

Sign in into https://amass.is.tue.mpg.de
Download the body data for the CMU motions (SMPL+H model)
Extract CMU.tar.bz2 in assets/CMU:

tar -C assets/ -xf ~/Downloads/CMU.tar.bz2 CMU/

Generate garment animation

To generate the deformed garment meshes for a given sequence:

python run_model.py assets/CMU/07/07_02_poses.npz --export_dir results/07_02

Rendering

Requirements: blender-2.93, ffmpeg

To render the meshes:

blender --background rendering/scene.blend --python rendering/render.py --path results/07_02

Citation

If you find this repository useful please cite our work:

@article {santesteban2021garmentcollisions,
    journal = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    title = {{Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On}},
    author = {Santesteban, Igor and Thuerey, Nils and Otaduy, Miguel A and Casas, Dan},
    year = {2021}
}

AI Virtual Calculator: This is a simple virtual calculator based on Artificial intelligence.

AI Virtual Calculator: This is a simple virtual calculator that works with gestures using OpenCV. We will use our hand in the air to click on the calc

1 Jan 13, 2022

Checkout some cool self-projects you can try your hands on to curb your boredom this December!

SoC-Winter Checkout some cool self-projects you can try your hands on to curb your boredom this December! These are short projects that you can do you

29 Nov 8, 2022

[CVPR 2021] "The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models" Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, Zhangyang Wang

The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models Codes for this paper The Lottery Tickets Hypo

59 Dec 28, 2022

Patch Rotation: A Self-Supervised Auxiliary Task for Robustness and Accuracy of Supervised Models

Patch-Rotation(PatchRot) Patch Rotation: A Self-Supervised Auxiliary Task for Robustness and Accuracy of Supervised Models Submitted to Neurips2021 To

4 Jul 12, 2021

The self-supervised goal reaching benchmark introduced in Discovering and Achieving Goals via World Models

Lexa-Benchmark Codebase for the self-supervised goal reaching benchmark introduced in 'Discovering and Achieving Goals via World Models'. Setup Create

1 Oct 14, 2021

This repository is the official implementation of Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (NeurIPS21).

Core-tuning This repository is the official implementation of ``Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regular

18 Dec 17, 2022

Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On

Related tags

Overview

Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On

Abstract

Running the model

Download trained models

Download human model

Download animation sequences

Generate garment animation

Rendering

Citation

You might also like...

AI Virtual Calculator: This is a simple virtual calculator based on Artificial intelligence.

Checkout some cool self-projects you can try your hands on to curb your boredom this December!

[CVPR 2021] "The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models" Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, Zhangyang Wang

Patch Rotation: A Self-Supervised Auxiliary Task for Robustness and Accuracy of Supervised Models

The self-supervised goal reaching benchmark introduced in Discovering and Achieving Goals via World Models

This repository is the official implementation of Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (NeurIPS21).

Try out deep learning models online on Google Colab

Users can free try their models on SIDD dataset based on this code

Unified Pre-training for Self-Supervised Learning and Supervised Learning for ASR

Releases(dress-trained-models)

dress-trained-models(Mar 28, 2022)

tshirt-trained-models(Nov 4, 2021)

trained-models(Nov 4, 2021)

Owner

Apache Flink

PyTorch Implementation of "Non-Autoregressive Neural Machine Translation"

Memory-efficient optimum einsum using opt_einsum planning and PyTorch kernels.

An elaborate and exhaustive paper list for Named Entity Recognition (NER)

paper list in the area of reinforcenment learning for recommendation systems

PyTorch code for our paper "Attention in Attention Network for Image Super-Resolution"

DGL-TreeSearch and the Gurobi-MWIS interface

Learning from Synthetic Data with Fine-grained Attributes for Person Re-Identification

Active and Sample-Efficient Model Evaluation

Unofficial reimplementation of ECAPA-TDNN for speaker recognition (EER=0.86 for Vox1_O when train only in Vox2)

An e-commerce company wants to segment its customers and determine marketing strategies according to these segments.

Source code to accompany Defunctland's video "FASTPASS: A Complicated Legacy"

Task Transformer Network for Joint MRI Reconstruction and Super-Resolution (MICCAI 2021)

Code for 'Self-Guided and Cross-Guided Learning for Few-shot segmentation. (CVPR' 2021)'

Implementation of Pix2Seq in PyTorch

Code for the paper "Controllable Video Captioning with an Exemplar Sentence"

Implementation of ECCV20 paper: the devil is in classification: a simple framework for long-tail object detection and instance segmentation

Stock-history-display - something like a easy yearly review for your stock performance

This is a TensorFlow implementation for C2-Rec

Utilities and information for the signals.numer.ai tournament