Implementation of the ICCV'21 paper Temporally-Coherent Surface Reconstruction via Metric-Consistent Atlases

Overview

Temporally-Coherent Surface Reconstruction via Metric-Consistent Atlases [Papers 1, 2][Project page] [Video]

The implementation of the papers

Install

The framework was tested with Python 3.8, PyTorch 1.7.0. and CUDA 11.0. The easiest way to work with the code is to create a new virtual Python environment and install the required packages.

  1. Install the virtualenvwrapper.
  2. Create a new environment and install the required packages.
mkvirtualenv --python=python3.8 tcsr
pip install -r requirements.txt
  1. Install Pytorch3d.
cd ~
curl -LO https://github.com/NVIDIA/cub/archive/1.10.0.tar.gz
tar xzf 1.10.0.tar.gz
export CUB_HOME=$PWD/cub-1.10.0
pip install git+https://github.com/facebookresearch/[email protected]
  1. Get the code and prepare the environment as follows:
git clone [email protected]:bednarikjan/temporally_coherent_surface_reconstruction.git
git submodule update --init --recursive
export PYTHONPATH="{PYTHONPATH}:path/to/dir/temporally_coherent_surface_reconstruction"

Get the Data

The project was tested on 6 base datasets (and their derivatives). Each datasets has to be processed so as to generate the input point clouds for training, the GT correspondences for evauluation and other auxilliary data. To do so, please use the individual scripts in tcsr/process_datasets. For each dataset, follow these steps:

  1. Download the data (links below).
  2. Open the script <dataset_name>.py and set the input/output paths.
  3. Run the script: python <dataset_name>.py

1. ANIM

  • Download the sequences horse gallop, horse collapse, camel gallop, camel collapse, and elephant gallop.
  • Download the sequence walking cat.

2. AMA

  • Download all 10 sequences, meshes only.

3. DFAUST

4. CAPE

  • Request the access to the raw scans and download it.
  • At the time of writing the paper (September 2021) four subjects (00032, 00096, 00159, 03223) were available and used in the paper.

5. INRIA

  • Request the access to the dataset and download it.
  • At the time of writing the paper (September 2021), four subjects (s1, s2, s3, s6) were available and used in the paper.

6. CMU

Train

The provided code allows for training our proposed method (OUR) but also the other atlas based approaches Differential Surface Representation (DSR) and AtlasNet (AN). The training is configured using the *.yaml configuration scripts in tcsr/train/configs.

There are 9 sample configuration files our_<dataset_name>.yaml which train OUR on each individual dataset and 2 sample configuration files an_anim.yaml, dsr_anim.yaml which train AN and DSR respectivelly on ANIM dataset.

By default, the trainin uses the exact settings as in the paper, namely it trains for 200'000 iterations using SGD, learning rate of 0.001 and batch size of 4. This can be altered in the configuration files.

Before starting the training, follow these steps:

  • Open the source file tcsr/data/data_loader.py and set the paths to the datasets in each dataset class.
  • Open the desired training configuration *.yaml file in tcsr/train/configs/ and set the output path for the training run data in the attribute path_train_run.

Start the training usint the script tcsr/train/train.py:

python train.py --conf configs/<file_name>.yaml

By default the script saves the training progress each 2000 iterations so you can safely kill it at any point and resume the trianing later using:

python train.py --cont path/to/training_run/root_dir

Evaluate

To evaluate a trianed model on the dense correspondence prediction task, use the script tcsr/evaluate/eval_dataset.py which allows for evaluation of multiple sequences (i.e. individual training runs within one dataset) at once. Please have a look at the command line arguments in the file.

An example of how to run the evaluation for the training runs contained in the root directory train_runs_root corresponding to 2 training runs run for the sequences cat_walk and horse_gallop within ANIM dataset:

python eval_dataset.py /path/to/train_runs_root --ds anim --include_seqs cat_walk horse_gallop  

The script produces a *.csv file in train_runs_root with the 4 measured metrics (see the paper).

Visualize

There are currently two ways to visualize the predictions.

1. Tensorboard

By default, the training script saves the GT and the predicted point clouds (for a couple of random data samples) each 2000 iterations. These can be viewed within Tensorboard. Each patch is visualized with a different color. This visualization is mostly useful as a sanity check during the trianing to see that the model is converging as expected.

  • Navigate to the root directory of the trianing runs and run:
tensorboard --logdir=. --port=8008 --bind_all
  • Open your browser and navigate to http://localhost:8008/

2. Per-sequence reconstruction GIF

You can view the reconstructed surfaces as a patch-wise textured mesh as a video within a GIF file. For this purpose, use the IPython Notebook file tcsr/visualize/render_uv.ipynb and open it in jupyterlab which allows for viewing the GIF right after running the code.

The rendering parameters (such as the camera location, texturing mode, gif speed etc.) are set usin the configuration file tcsr/visualize/conf_patches.yaml. There are sample configurations for the sequence cat_walk, which can be used to write configurations for other sequences/datasets.

Before running the cells, set the variables in the second cell (paths, models, data).

Citation

@inproceedings{bednarik2021temporally_coherent,
   title = {Temporally-Coherent Surface Reconstruction via Metric-Consistent Atlases},
   author = {Bednarik, Jan and Kim, Vladimir G. and Chaudhuri, Siddhartha and Parashar, Shaifali and Salzmann, Mathieu and Fua, Pascal and Aigerman, Noam},
   booktitle = {Proceedings of IEEE International Conference on Computer Vision (ICCV)},
   year = {2021}
}

@inproceedings{bednarik2021temporally_consistent,
   title = {Temporally-Consistent Surface Reconstruction via Metrically-Consistent Atlases},
   author = {Bednarik, Jan and Aigerman, Noam and Kim, Vladimir G. and Chaudhuri, Siddhartha and Parashar, Shaifali and Salzmann, Mathieu and Fua, Pascal},
   booktitle = {arXiv},
   year = {2021}
}

Acknowledgements

This work was partially done while the main author was an intern at Adobe Research.

TODO

  • Add support for visualizing the correspondence error heatmap on the GT mesh.
  • Add support for visualizing the colorcoded correspondences on the GT mesh.
  • Add the support for generating the pre-aligned AMAa dataset using ICP.
  • Add the code for the nonrigid ICP experiments.
An Api for Emotion recognition.

PLAYEMO Playemo was built from the ground-up with Flask, a python tool that makes it easy for developers to build APIs. Use Cases Is Python your langu

greek geek 2 Jul 16, 2022
Algo-burn - Script to configure an Algorand address as a "burn" address for one or more ASA tokens

Algorand Burn Address This is a simple script to illustrate how a "burn address"

GSD 5 May 10, 2022
Metric learning algorithms in Python

metric-learn: Metric Learning in Python metric-learn contains efficient Python implementations of several popular supervised and weakly-supervised met

1.3k Dec 28, 2022
CNN Based Meta-Learning for Noisy Image Classification and Template Matching

CNN Based Meta-Learning for Noisy Image Classification and Template Matching Introduction This master thesis used a few-shot meta learning approach to

Kumar Manas 2 Dec 09, 2021
OBG-FCN - implementation of 'Object Boundary Guided Semantic Segmentation'

OBG-FCN This repository is to reproduce the implementation of 'Object Boundary Guided Semantic Segmentation' in http://arxiv.org/abs/1603.09742 Object

Jiu XU 3 Mar 11, 2019
MADE (Masked Autoencoder Density Estimation) implementation in PyTorch

pytorch-made This code is an implementation of "Masked AutoEncoder for Density Estimation" by Germain et al., 2015. The core idea is that you can turn

Andrej 498 Dec 30, 2022
Revisiting Contrastive Methods for Unsupervised Learning of Visual Representations. [2021]

Revisiting Contrastive Methods for Unsupervised Learning of Visual Representations This repo contains the Pytorch implementation of our paper: Revisit

Wouter Van Gansbeke 80 Nov 20, 2022
Designing a Minimal Retrieve-and-Read System for Open-Domain Question Answering (NAACL 2021)

Designing a Minimal Retrieve-and-Read System for Open-Domain Question Answering Abstract In open-domain question answering (QA), retrieve-and-read mec

Clova AI Research 34 Apr 13, 2022
A fast MoE impl for PyTorch

An easy-to-use and efficient system to support the Mixture of Experts (MoE) model for PyTorch.

Rick Ho 873 Jan 09, 2023
Pseudo lidar - (CVPR 2019) Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving

Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving This paper has been accpeted by Conference o

Yan Wang 881 Dec 27, 2022
High performance Cross-platform Inference-engine, you could run Anakin on x86-cpu,arm, nv-gpu, amd-gpu,bitmain and cambricon devices.

Anakin2.0 Welcome to the Anakin GitHub. Anakin is a cross-platform, high-performance inference engine, which is originally developed by Baidu engineer

514 Dec 28, 2022
Contrastive Learning for Compact Single Image Dehazing, CVPR2021

AECR-Net Contrastive Learning for Compact Single Image Dehazing, CVPR2021. Official Pytorch based implementation. Paper arxiv Pytorch Version TODO: mo

glassy 253 Jan 01, 2023
It's final year project of Diploma Engineering. This project is based on Computer Vision.

Face-Recognition-Based-Attendance-System It's final year project of Diploma Engineering. This project is based on Computer Vision. Brief idea about ou

Neel 10 Nov 02, 2022
Cl datasets - PyTorch image dataloaders and utility functions to load datasets for supervised continual learning

Continual learning datasets Introduction This repository contains PyTorch image

berjaoui 5 Aug 28, 2022
EmoTag helps you train emotion detection model for Chinese audios

emoTag emoTag helps you train emotion detection model for Chinese audios. Environment pip install -r requirement.txt Data We used Emotional Speech Dat

_zza 4 Sep 07, 2022
Kroomsa: A search engine for the curious

Kroomsa A search engine for the curious. It is a search algorithm designed to en

Wingify 7 Jun 20, 2022
[WACV 2022] Contextual Gradient Scaling for Few-Shot Learning

CxGrad - Official PyTorch Implementation Contextual Gradient Scaling for Few-Shot Learning Sanghyuk Lee, Seunghyun Lee, and Byung Cheol Song In WACV 2

Sanghyuk Lee 4 Dec 05, 2022
Collection of generative models, e.g. GAN, VAE in Pytorch and Tensorflow.

Generative Models Collection of generative models, e.g. GAN, VAE in Pytorch and Tensorflow. Also present here are RBM and Helmholtz Machine. Note: Gen

Agustinus Kristiadi 7k Jan 02, 2023
Python wrappers to the C++ library SymEngine, a fast C++ symbolic manipulation library.

SymEngine Python Wrappers Python wrappers to the C++ library SymEngine, a fast C++ symbolic manipulation library. Installation Pip See License section

136 Dec 28, 2022
Clean and readable code for Decision Transformer: Reinforcement Learning via Sequence Modeling

Minimal implementation of Decision Transformer: Reinforcement Learning via Sequence Modeling in PyTorch for mujoco control tasks in OpenAI gym

Nikhil Barhate 104 Jan 06, 2023