Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On

Last update: Dec 24, 2022

Related tags

Overview

Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On

Abstract

We propose a new generative model for 3D garment deformations that enables us to learn, for the first time, a data-driven method for virtual try-on that effectively addresses garment-body collisions. In contrast to existing methods that require an undesirable postprocessing step to fix garment-body interpenetrations at test time, our approach directly outputs 3D garment configurations that do not collide with the underlying body. Key to our success is a new canonical space for garments that removes pose-and-shape deformations already captured by a new diffused human body model, which extrapolates body surface properties such as skinning weights and blendshapes to any 3D point. We leverage this representation to train a generative model with a novel self-supervised collision term that learns to reliably solve garment-body interpenetrations. We extensively evaluate and compare our results with recently proposed data-driven methods, and show that our method is the first to successfully address garment-body contact in unseen body shapes and motions, without compromising realism and detail.

Running the model

Requirements: python3.8, tensorflow-2.2.1, numpy-1.18.5, scipy-1.7.1, chumpy-0.70

Project structure:

vto-garment-collisions
│
└───assets 
|    └─ images    
|    └─ meshes    
|    └─ CMU       # Not included, see instructions
|    └─ SMPL      # Not included, see instructions
| 
└───rendering     # Code to render meshes 
|
└───src           # Code to run the model
| 
└───trained_models      
|    └─ diffused_body  # Networks of the diffused body model (Not included, see instructions)
|    └─ tshirt         # Networks of tshirt model (Not included, see instructions)
│
└───run_model.py

Download trained models

Download models of the diffused human body: https://github.com/isantesteban/vto-garment-collisions/releases/download/trained-models/trained_models_diffused_body.zip
Download models of the garment: https://github.com/isantesteban/vto-garment-collisions/releases/download/tshirt-trained-models/trained_models_tshirt.zip
Create trained_models directory and extract trained_models_diffused_body.zip and trained_models_tshirt.zip there.

Download human model

Sign in into https://smpl.is.tue.mpg.de
Download SMPL version 1.0.0 for Python 2.7 (10 shape PCs)
Extract SMPL_python_v.1.0.0.zip and copy smpl/models/basicModel_f_lbs_10_207_0_v1.0.0.pkl in assets/SMPL

Download animation sequences

Sign in into https://amass.is.tue.mpg.de
Download the body data for the CMU motions (SMPL+H model)
Extract CMU.tar.bz2 in assets/CMU:

tar -C assets/ -xf ~/Downloads/CMU.tar.bz2 CMU/

Generate garment animation

To generate the deformed garment meshes for a given sequence:

python run_model.py assets/CMU/07/07_02_poses.npz --export_dir results/07_02

Rendering

Requirements: blender-2.93, ffmpeg

To render the meshes:

blender --background rendering/scene.blend --python rendering/render.py --path results/07_02

Citation

If you find this repository useful please cite our work:

@article {santesteban2021garmentcollisions,
    journal = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    title = {{Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On}},
    author = {Santesteban, Igor and Thuerey, Nils and Otaduy, Miguel A and Casas, Dan},
    year = {2021}
}

AI Virtual Calculator: This is a simple virtual calculator based on Artificial intelligence.

AI Virtual Calculator: This is a simple virtual calculator that works with gestures using OpenCV. We will use our hand in the air to click on the calc

1 Jan 13, 2022

Checkout some cool self-projects you can try your hands on to curb your boredom this December!

SoC-Winter Checkout some cool self-projects you can try your hands on to curb your boredom this December! These are short projects that you can do you

29 Nov 8, 2022

[CVPR 2021] "The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models" Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, Zhangyang Wang

The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models Codes for this paper The Lottery Tickets Hypo

59 Dec 28, 2022

Patch Rotation: A Self-Supervised Auxiliary Task for Robustness and Accuracy of Supervised Models

Patch-Rotation(PatchRot) Patch Rotation: A Self-Supervised Auxiliary Task for Robustness and Accuracy of Supervised Models Submitted to Neurips2021 To

4 Jul 12, 2021

The self-supervised goal reaching benchmark introduced in Discovering and Achieving Goals via World Models

Lexa-Benchmark Codebase for the self-supervised goal reaching benchmark introduced in 'Discovering and Achieving Goals via World Models'. Setup Create

1 Oct 14, 2021

This repository is the official implementation of Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (NeurIPS21).

Core-tuning This repository is the official implementation of ``Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regular

18 Dec 17, 2022

Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On

Related tags

Overview

Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On

Abstract

Running the model

Download trained models

Download human model

Download animation sequences

Generate garment animation

Rendering

Citation

You might also like...

AI Virtual Calculator: This is a simple virtual calculator based on Artificial intelligence.

Checkout some cool self-projects you can try your hands on to curb your boredom this December!

[CVPR 2021] "The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models" Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, Zhangyang Wang

Patch Rotation: A Self-Supervised Auxiliary Task for Robustness and Accuracy of Supervised Models

The self-supervised goal reaching benchmark introduced in Discovering and Achieving Goals via World Models

This repository is the official implementation of Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (NeurIPS21).

Try out deep learning models online on Google Colab

Users can free try their models on SIDD dataset based on this code

Unified Pre-training for Self-Supervised Learning and Supervised Learning for ASR

Releases(dress-trained-models)

dress-trained-models(Mar 28, 2022)

tshirt-trained-models(Nov 4, 2021)

trained-models(Nov 4, 2021)

Owner

Six - a Python 2 and 3 compatibility library

The implementation of the lifelong infinite mixture model

Code for a seq2seq architecture with Bahdanau attention designed to map stereotactic EEG data from human brains to spectrograms, using the PyTorch Lightning.

A method to perform unsupervised cross-region adaptation of crop classifiers trained with satellite image time series.

A PyTorch implementation of the Transformer model in "Attention is All You Need".

A Comprehensive Empirical Study of Vision-Language Pre-trained Model for Supervised Cross-Modal Retrieval

SEC'21: Sparse Bitmap Compression for Memory-Efficient Training onthe Edge

3D-CariGAN: An End-to-End Solution to 3D Caricature Generation from Normal Face Photos

This library provides an abstraction to perform Model Versioning using Weight & Biases.

Facestar dataset. High quality audio-visual recordings of human conversational speech.

Neighborhood Contrastive Learning for Novel Class Discovery

[CVPR 2022 Oral] Balanced MSE for Imbalanced Visual Regression https://arxiv.org/abs/2203.16427

Implementation of Analyzing and Improving the Image Quality of StyleGAN (StyleGAN 2) in PyTorch

Segmentation Training Pipeline

Plugin adapted from Ultralytics to bring YOLOv5 into Napari

Medical-Image-Triage-and-Classification-System-Based-on-COVID-19-CT-and-X-ray-Scan-Dataset

An adaptive hierarchical energy management strategy for hybrid electric vehicles

Picasso: a methods for embedding points in 2D in a way that respects distances while fitting a user-specified shape.

TransZero++: Cross Attribute-guided Transformer for Zero-Shot Learning

HomeAssitant custom integration for dyson