Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On

Last update: Dec 24, 2022

Related tags

Overview

Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On

Abstract

We propose a new generative model for 3D garment deformations that enables us to learn, for the first time, a data-driven method for virtual try-on that effectively addresses garment-body collisions. In contrast to existing methods that require an undesirable postprocessing step to fix garment-body interpenetrations at test time, our approach directly outputs 3D garment configurations that do not collide with the underlying body. Key to our success is a new canonical space for garments that removes pose-and-shape deformations already captured by a new diffused human body model, which extrapolates body surface properties such as skinning weights and blendshapes to any 3D point. We leverage this representation to train a generative model with a novel self-supervised collision term that learns to reliably solve garment-body interpenetrations. We extensively evaluate and compare our results with recently proposed data-driven methods, and show that our method is the first to successfully address garment-body contact in unseen body shapes and motions, without compromising realism and detail.

Running the model

Requirements: python3.8, tensorflow-2.2.1, numpy-1.18.5, scipy-1.7.1, chumpy-0.70

Project structure:

vto-garment-collisions
│
└───assets 
|    └─ images    
|    └─ meshes    
|    └─ CMU       # Not included, see instructions
|    └─ SMPL      # Not included, see instructions
| 
└───rendering     # Code to render meshes 
|
└───src           # Code to run the model
| 
└───trained_models      
|    └─ diffused_body  # Networks of the diffused body model (Not included, see instructions)
|    └─ tshirt         # Networks of tshirt model (Not included, see instructions)
│
└───run_model.py

Download trained models

Download models of the diffused human body: https://github.com/isantesteban/vto-garment-collisions/releases/download/trained-models/trained_models_diffused_body.zip
Download models of the garment: https://github.com/isantesteban/vto-garment-collisions/releases/download/tshirt-trained-models/trained_models_tshirt.zip
Create trained_models directory and extract trained_models_diffused_body.zip and trained_models_tshirt.zip there.

Download human model

Sign in into https://smpl.is.tue.mpg.de
Download SMPL version 1.0.0 for Python 2.7 (10 shape PCs)
Extract SMPL_python_v.1.0.0.zip and copy smpl/models/basicModel_f_lbs_10_207_0_v1.0.0.pkl in assets/SMPL

Download animation sequences

Sign in into https://amass.is.tue.mpg.de
Download the body data for the CMU motions (SMPL+H model)
Extract CMU.tar.bz2 in assets/CMU:

tar -C assets/ -xf ~/Downloads/CMU.tar.bz2 CMU/

Generate garment animation

To generate the deformed garment meshes for a given sequence:

python run_model.py assets/CMU/07/07_02_poses.npz --export_dir results/07_02

Rendering

Requirements: blender-2.93, ffmpeg

To render the meshes:

blender --background rendering/scene.blend --python rendering/render.py --path results/07_02

Citation

If you find this repository useful please cite our work:

@article {santesteban2021garmentcollisions,
    journal = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    title = {{Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On}},
    author = {Santesteban, Igor and Thuerey, Nils and Otaduy, Miguel A and Casas, Dan},
    year = {2021}
}

AI Virtual Calculator: This is a simple virtual calculator based on Artificial intelligence.

AI Virtual Calculator: This is a simple virtual calculator that works with gestures using OpenCV. We will use our hand in the air to click on the calc

1 Jan 13, 2022

Checkout some cool self-projects you can try your hands on to curb your boredom this December!

SoC-Winter Checkout some cool self-projects you can try your hands on to curb your boredom this December! These are short projects that you can do you

29 Nov 8, 2022

[CVPR 2021] "The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models" Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, Zhangyang Wang

The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models Codes for this paper The Lottery Tickets Hypo

59 Dec 28, 2022

Patch Rotation: A Self-Supervised Auxiliary Task for Robustness and Accuracy of Supervised Models

Patch-Rotation(PatchRot) Patch Rotation: A Self-Supervised Auxiliary Task for Robustness and Accuracy of Supervised Models Submitted to Neurips2021 To

4 Jul 12, 2021

The self-supervised goal reaching benchmark introduced in Discovering and Achieving Goals via World Models

Lexa-Benchmark Codebase for the self-supervised goal reaching benchmark introduced in 'Discovering and Achieving Goals via World Models'. Setup Create

1 Oct 14, 2021

This repository is the official implementation of Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (NeurIPS21).

Core-tuning This repository is the official implementation of ``Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regular

18 Dec 17, 2022

Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On

Related tags

Overview

Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On

Abstract

Running the model

Download trained models

Download human model

Download animation sequences

Generate garment animation

Rendering

Citation

You might also like...

AI Virtual Calculator: This is a simple virtual calculator based on Artificial intelligence.

Checkout some cool self-projects you can try your hands on to curb your boredom this December!

[CVPR 2021] "The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models" Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, Zhangyang Wang

Patch Rotation: A Self-Supervised Auxiliary Task for Robustness and Accuracy of Supervised Models

The self-supervised goal reaching benchmark introduced in Discovering and Achieving Goals via World Models

This repository is the official implementation of Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (NeurIPS21).

Try out deep learning models online on Google Colab

Users can free try their models on SIDD dataset based on this code

Unified Pre-training for Self-Supervised Learning and Supervised Learning for ASR

Releases(dress-trained-models)

dress-trained-models(Mar 28, 2022)

tshirt-trained-models(Nov 4, 2021)

trained-models(Nov 4, 2021)

Owner

An implementation of paper `Real-time Convolutional Neural Networks for Emotion and Gender Classification` with PaddlePaddle.

Python3 / PyTorch implementation of the following paper: Fine-grained Semantics-aware Representation Enhancement for Self-supervisedMonocular Depth Estimation. ICCV 2021 (oral)

SigOpt wrappers for scikit-learn methods

We utilize deep reinforcement learning to obtain favorable trajectories for visual-inertial system calibration.

Bald-to-Hairy Translation Using CycleGAN

DAT4 - General Assembly's Data Science course in Washington, DC

Resources complimenting the Machine Learning Course led in the Faculty of mathematics and informatics part of Sofia University.

Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency[ECCV 2020]

A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.

Yet another video caption

Control-Raspberry-Pi-Robot-using-Hand-Gestures - A 4WD Robot car based on Raspberry Pi that controlled by hand gestures(using openCV and mediapipe)

LVI-SAM: Tightly-coupled Lidar-Visual-Inertial Odometry via Smoothing and Mapping

Raptor-Multi-Tool - Raptor Multi Tool With Python

Project Aquarium is a SUSE-sponsored open source project aiming at becoming an easy to use, rock solid storage appliance based on Ceph.

Test-Time Personalization with a Transformer for Human Pose Estimation, NeurIPS 2021

This provides the R code and data to replicate results in "The USS Trustee’s risky strategy"

Learning to Estimate Hidden Motions with Global Motion Aggregation

Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models (published in ICLR2018)

pix2pix in tensorflow.js

A library for building and serving multi-node distributed faiss indices.