Benchmark for the generalization of 3D machine learning models across different remeshing/samplings of a surface.

Overview

Discretization Robust Correspondence Benchmark

One challenge of machine learning on 3D surfaces is that there are many different representations/samplings ("discretizations") which all encode the same underlying shape---consider e.g. different triangle meshes of a surface. We expect models to generalize across these representations; the purpose of this benchmark is to measure generalization of 3D machine learning models across different discretizations

This benchmark contains test meshes of human bodies, derived from the MPI-FAUST dataset, remeshed/resampled according to several policies. The task is to predict correspondence, defined by predicting the nearest vertex index on the template mesh. We intentionally provide test data only. The intent of this benchmark is that methods train on the ordinary FAUST template meshes, then evaluate on this dataset. This measures the ability of the method to generalize to new, unseen discretizations of shapes.

example image of data

From: DiffusionNet: Discretization Agnostic Learning on Surfaces, Nicholas Sharp, Souhaib Attaiki, Keenan Crane, Maks Ovsjanikov, conditionally accepted to ACM ToG 2021.

Please cite this benchmark as:

@article{sharp2021diffusion,
  author = {Sharp, Nicholas and Attaiki, Souhaib and Crane, Keenan and Ovsjanikov, Maks},
  title = {DiffusionNet: Discretization Agnostic Learning on Surfaces},
  journal = {ACM Trans. Graph.},
  volume = {XX},
  number = {X},
  year = {20XX},
  publisher = {ACM},
  address = {New York, NY, USA},
}

Remeshing/sampling policies

  • iso Meshes are isotropically remeshed, to have a roughly uniform distribution of vetices, with approximately equilateral triangles
  • qes Meshes are first refined to have many more vertices, then simplified back to approximately 2x the original resolution using Quadric Error Simplification
  • mc Meshes are volumetrically reconstructed, and a mesh is extracted via the marching cubes algorithm.
  • dense Meshes are refined to have nonuniform density by choosing 5 random faces, refining the mesh in the vicinity of the face, then isotropically remeshing.
  • cloud A point cloud, with normals, sampled uniformly from the mesh

In this repository

  • data/
    • iso/
      • tr_reg_iso_080.ply FAUST test mesh 80, remeshed according to the iso strategy
      • tr_reg_iso_080.txt Ground-truth correspondence indices, per-vertex
      • ...
      • tr_reg_iso_099.ply
      • tr_reg_iso_099.txt
    • qes/
      • tr_reg_qes_080.ply
      • tr_reg_qes_080.txt
      • ...
    • mc/
      • tr_reg_mc_080.ply
      • tr_reg_mc_080.txt
      • ...
    • dense/
      • tr_reg_dense_080.ply
      • tr_reg_dense_080.txt
      • ...
    • cloud/
      • tr_reg_cloud_080.ply A sampled point cloud from FAUST test mesh 80, with normals
      • tr_reg_cloud_080.txt Ground-truth correspondence indices, per-point
      • ...
  • scripts/ Meshlab & Python scripts which were used to generate the data.

Notes about the data

  • The meshes are not necessarily high quality! In particular, the mc meshes have coincident vertices and degenerate leftover from the marching cubes process. Such artifacts are a common occurence in real data.

Benchmark Task

This benchmark is designed for template correspondence via vertex index prediction. That is, for each vertex (resp., point) in a test shape, we predict the corresponding nearest vertex on a template mesh. The FAUST template mesh has 6890 vertices, so this is essentially a segmentation problem with classes from [0, 6899]. Note that although popular in past work, this categorical formulation is surely not the best notion of correspondence between surfaces. However, it is very simple, and exposes a tendancy to overfit to discretization, which makes it a good choice for this benchmark.

The first 80 original MPI-FAUST template meshes should be used as training data: i.e. tr_reg_000.ply-tr_reg_079.ply. The last 20 shapes are taken as the test set, and remeshed/resampled for the purpose of this benchmark. These original meshes are already deformed templates, so the ground truth vertex labels are simply [0,1,2,3,4...]. We do not host the original data here; you must download it from http://faust.is.tue.mpg.de/.

After training on the first 80 original FAUST meshes, we evaluate on the test meshes, predicting corresponding vertices. Error is measured by the geodesic distance along the template mesh between the predicted vertex and the ground-truth vertex. (% of vertices predicted exactly correct is not really a meaningful metric.) See this repo for a full example of training and eval scripts.

Papers using this dataset

(create a pull request to add more!)

License

The scripts which generate the data are available for any use under an MIT license (C) Nicholas Sharp 2021.

The remeshed/sampled meshes are derived from the MPI-FAUST dataset, governed by this license (which allows derivative works).

Owner
Nicholas Sharp
3D geometry researcher: computer graphics/vision, geometry processing, and 3D machine learning
Nicholas Sharp
Official implementation of "Generating 3D Molecules for Target Protein Binding"

Generating 3D Molecules for Target Protein Binding This is the official implementation of the GraphBP method proposed in the following paper. Meng Liu

DIVE Lab, Texas A&M University 74 Dec 07, 2022
Official repository of the AAAI'2022 paper "Contrast and Generation Make BART a Good Dialogue Emotion Recognizer"

CoG-BART Contrast and Generation Make BART a Good Dialogue Emotion Recognizer Quick Start: To run the model on test sets of four datasets, Download th

39 Dec 24, 2022
Code and Experiments for ACL-IJCNLP 2021 Paper Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering.

Code and Experiments for ACL-IJCNLP 2021 Paper Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering.

Sidd Karamcheti 50 Nov 16, 2022
Implementation of "JOKR: Joint Keypoint Representation for Unsupervised Cross-Domain Motion Retargeting"

JOKR: Joint Keypoint Representation for Unsupervised Cross-Domain Motion Retargeting Pytorch implementation for the paper "JOKR: Joint Keypoint Repres

45 Dec 25, 2022
Simple codebase for flexible neural net training

neural-modular Simple codebase for flexible neural net training. Allows for seamless exchange of models, dataset, and optimizers. Uses hydra for confi

Jannik Kossen 7 Apr 05, 2022
This repository contains the official implementation code of the paper Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis, accepted at EMNLP 2021.

MultiModal-InfoMax This repository contains the official implementation code of the paper Improving Multimodal Fusion with Hierarchical Mutual Informa

Deep Cognition and Language Research (DeCLaRe) Lab 89 Dec 26, 2022
Generative Adversarial Text-to-Image Synthesis

###Generative Adversarial Text-to-Image Synthesis Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, Honglak Lee This is the

Scott Ellison Reed 883 Dec 31, 2022
MonoScene: Monocular 3D Semantic Scene Completion

MonoScene: Monocular 3D Semantic Scene Completion MonoScene: Monocular 3D Semantic Scene Completion] [arXiv + supp] | [Project page] Anh-Quan Cao, Rao

298 Jan 08, 2023
Model Serving Made Easy

The easiest way to build Machine Learning APIs BentoML makes moving trained ML models to production easy: Package models trained with any ML framework

BentoML 4.4k Jan 08, 2023
DilatedNet in Keras for image segmentation

Keras implementation of DilatedNet for semantic segmentation A native Keras implementation of semantic segmentation according to Multi-Scale Context A

303 Mar 15, 2022
Neural Radiance Fields Using PyTorch

This project is a PyTorch implementation of Neural Radiance Fields (NeRF) for reproduction of results whilst running at a faster speed.

Vedant Ghodke 1 Feb 11, 2022
A real world application of a Recurrent Neural Network on a binary classification of time series data

What is this This is a real world application of a Recurrent Neural Network on a binary classification of time series data. This project includes data

Josep Maria Salvia Hornos 2 Jan 30, 2022
Open source repository for the code accompanying the paper 'Non-Rigid Neural Radiance Fields Reconstruction and Novel View Synthesis of a Deforming Scene from Monocular Video'.

Non-Rigid Neural Radiance Fields This is the official repository for the project "Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synt

Facebook Research 296 Dec 29, 2022
Tensors and neural networks in Haskell

Hasktorch Hasktorch is a library for tensors and neural networks in Haskell. It is an independent open source community project which leverages the co

hasktorch 920 Jan 04, 2023
Paddle implementation for "Highly Efficient Knowledge Graph Embedding Learning with Closed-Form Orthogonal Procrustes Analysis" (NAACL 2021)

ProcrustEs-KGE Paddle implementation for Highly Efficient Knowledge Graph Embedding Learning with Orthogonal Procrustes Analysis 🙈 A more detailed re

Lincedo Lab 4 Jun 09, 2021
CIFS: Improving Adversarial Robustness of CNNs via Channel-wise Importance-based Feature Selection

CIFS This repository provides codes for CIFS (ICML 2021). CIFS: Improving Adversarial Robustness of CNNs via Channel-wise Importance-based Feature Sel

Hanshu YAN 19 Nov 12, 2022
Efficient and intelligent interactive segmentation annotation software

Efficient and intelligent interactive segmentation annotation software

294 Dec 30, 2022
Pytorch implementation for "Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasets" (ECCV 2020 Spotlight)

Distribution-Balanced Loss [Paper] The implementation of our paper Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasets (

Tong WU 304 Dec 22, 2022
Implementation of paper "Decision-based Black-box Attack Against Vision Transformers via Patch-wise Adversarial Removal"

Patch-wise Adversarial Removal Implementation of paper "Decision-based Black-box Attack Against Vision Transformers via Patch-wise Adversarial Removal

4 Oct 12, 2022
NumQMBasic - A mini-course offered to Undergrad physics students

The best way to use this material is by forking it by click the Fork button at the top, right corner. Then you will get your own copy to play with! Th

Raghu 35 Dec 05, 2022