Latte: Cross-framework Python Package for Evaluation of Latent-based Generative Models

Last update: Sep 08, 2022

Overview

Cross-framework Python Package for Evaluation of Latent-based Generative Models

Latte

Latte (for LATent Tensor Evaluation) is a cross-framework Python package for evaluation of latent-based generative models. Latte supports calculation of disentanglement and controllability metrics in both PyTorch (via TorchMetrics) and TensorFlow.

Installation

For developers working on local clone, cd to the repo and replace latte with .. For example, pip install .[tests]

pip install latte-metrics           # core (numpy only)
pip install latte-metrics[pytorch]  # with torchmetrics wrapper
pip install latte-metrics[keras]    # with tensorflow wrapper
pip install latte-metrics[tests]    # for testing

Running tests locally

pip install .[tests]
pytest tests/ --cov=latte

Example

Functional API

import latte
from latte.functional.disentanglement.mutual_info import mig
import numpy as np

latte.seed(42)

z = np.random.randn(16, 8)
a = np.random.randn(16, 2)

mutual_info_gap = mig(z, a, discrete=False, reg_dim=[4, 3])

Modular API

import latte
from latte.metrics.core.disentanglement import MutualInformationGap
import numpy as np

latte.seed(42)

mig = MutualInformationGap()

# ... 
# initialize data and model
# ...

for data, attributes in range(batches):
  recon, z = model(data)

  mig.update_state(z, attributes)

mig_val = mig.compute()

TorchMetrics API

import latte
from latte.metrics.torch.disentanglement import MutualInformationGap
import torch

latte.seed(42)

mig = MutualInformationGap()

# ... 
# initialize data and model
# ...

for data, attributes in range(batches):
  recon, z = model(data)

  mig.update(z, attributes)

mig_val = mig.compute()

Keras Metric API

import latte
from latte.metrics.keras.disentanglement import MutualInformationGap
from tensorflow import keras as tfk

latte.seed(42)

mig = MutualInformationGap()

# ... 
# initialize data and model
# ...

for data, attributes in range(batches):
  recon, z = model(data)

  mig.update_state(z, attributes)

mig_val = mig.result()

Documentation

https://latte.readthedocs.io/en/latest

Supported metrics

Metric	Latte Functional	Latte Modular	TorchMetrics	Keras Metric
Disentanglement Metrics
📝 Mutual Information Gap (MIG)	🧪	🧪	🧪	??
📝 Dependency-blind Mutual Information Gap (DMIG)	🧪	🧪	🧪	🧪
📝 Dependency-aware Mutual Information Gap (XMIG)	🧪	🧪	🧪	🧪
📝 Dependency-aware Latent Information Gap (DLIG)	🧪	🧪	🧪	🧪
📝 Separate Attribute Predictability (SAP)	🧪	🧪	🧪	🧪
📝 Modularity	🧪	🧪	🧪	🧪
📝 β-VAE Score	👀	👀	👀	👀
📝 FactorVAE Score	👀	👀	👀	👀
📝 DCI Score	👀	👀	👀	👀
📝 Interventional Robustness Score (IRS)	👀	👀	👀	👀
📝 Consistency	👀	👀	👀	👀
📝 Restrictiveness	👀	👀	👀	👀
Interpolatability Metrics
📝 Smoothness	🧪	🧪	🧪	🧪
📝 Monotonicity	🧪	🧪	🧪	🧪
📝 Latent Density Ratio	🕣	🕣	🕣	🕣
📝 Linearity	👀	👀	👀	👀

Bundled metric modules

🧪 Experimental (subject to changes) | ✔️ Stable | 🔨 In Progress | 🕣 In Queue

Metric Bundle	Latte Functional	Latte Modular	TorchMetrics	Keras Metric	Included
Dependency-aware Disentanglement	🧪	🧪	🧪	🧪	MIG, DMIG, XMIG, DLIG
LIAD-based Interpolatability	🧪	🧪	🧪	🧪	Smoothness, Monotonicity

Cite

For individual metrics, please cite the paper according to the link in the 📝 icon in front of each metric.

If you find our package useful please cite our repository and arXiv preprint as

@article{
  watcharasupat2021latte,
  author = {Watcharasupat, Karn N. and Lee, Junyoung and Lerch, Alexander},
  title = {{Latte: Cross-framework Python Package for Evaluation of Latent-based Generative Models}},
  eprint={2112.10638},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url = {https://github.com/karnwatcharasupat/latte}
  doi = {10.5281/zenodo.5786402}
}

Comments

Documentation: Metric Descriptions

Might be nice to provide a short description for each metric in addition to the paper links. The readme might get too long with it, but either some doc in the repo or maybe on a github.io page?
type: documentation priority: high

opened by alexanderlerch 2
Add Smoothness and Monotonicity support
Smoothness

[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

Monotonicity

[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

type: enhancement
opened by karnwatcharasupat 0
Add Modularity support
[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

type: enhancement
opened by karnwatcharasupat 0
Add SAP support
[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

type: enhancement
opened by karnwatcharasupat 0
Add DMIG, DLIG, XMIG support
DMIG

[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

XMIG

[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

DLIG

[ x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

type: enhancement
opened by karnwatcharasupat 0
Add MIG support
[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

type: enhancement
opened by karnwatcharasupat 0
Support issue for on-the-fly computation in TF2 graph mode
The current delegate-to-NumPy technique used in TF is only compatible with TF2 eager mode since Tensor.numpy() would not work in graph mode. As a result, graph-mode users will only be able to use Latte in the evaluation stage when the model weights are no longer changing but not on-the-fly during the training stage.

However, certain computation steps required for some metrics (especially MI-based ones) necessarily require scikit-learn ops and there is no (maintainable) way to create consistent TF mirrors of those functions.

One potential solution is to wrap the core functions in tf.numpy_function or tf.py_function but we will have to figure out a way to make the wrapper less painful to implement/maintain since the variable args/kwargs option currently used by the dtype converter is not allowed in these functions. A naive workaround would be to make a tf.numpy_function wrapper for every highest-possible level function with fixed args but this would be considered a last-resort solution.

Links:

https://www.tensorflow.org/api_docs/python/tf/numpy_function

https://www.tensorflow.org/api_docs/python/tf/py_function

type: enhancement priority: medium !! needs more brains !!
opened by karnwatcharasupat 3

Releases(v0.0.1-alpha5)

v0.0.1-alpha5(Jan 20, 2022)
What's Changed

Add contributing guide by @karnwatcharasupat in https://github.com/karnwatcharasupat/latte/pull/16

[ADD] add example notebooks by @karnwatcharasupat in https://github.com/karnwatcharasupat/latte/pull/18

Full Changelog: https://github.com/karnwatcharasupat/latte/compare/v0.0.1-alpha3...v0.0.1-alpha5
Source code(tar.gz)
Source code(zip)
v0.0.1-alpha3(Dec 16, 2021)
Added metric bundle support

Full Changelog: https://github.com/karnwatcharasupat/latte/compare/v0.0.1-alpha2...v0.0.1-alpha3
Source code(tar.gz)
Source code(zip)
v0.0.1-alpha2(Dec 9, 2021)

Full Changelog: https://github.com/karnwatcharasupat/latte/compare/v0.0.1-alpha1...v0.0.1-alpha2
Source code(tar.gz)
Source code(zip)
v0.0.1-alpha1(Dec 1, 2021)

Full Changelog: https://github.com/karnwatcharasupat/latte/compare/v0.0.1-dev5...v0.0.1-alpha1
Source code(tar.gz)
Source code(zip)

Owner

Karn Watcharasupat

Lab Cat 🐱🌈 | Audio Signal Processing Research Student. NTU EEE Class of 2022. Georgia Tech Music Tech Visiting Researcher.

GitHub Repository https://latte.readthedocs.io/en/latest

Pytorch implementation for "Density-aware Chamfer Distance as a Comprehensive Metric for Point Cloud Completion" (NeurIPS 2021)

Density-aware Chamfer Distance This repository contains the official PyTorch implementation of our paper: Density-aware Chamfer Distance as a Comprehe

93 Dec 15, 2022

Unofficial implementation of the paper: PonderNet: Learning to Ponder in TensorFlow

PonderNet-TensorFlow This is an Unofficial Implementation of the paper: PonderNet: Learning to Ponder in TensorFlow. Official PyTorch Implementation:

1 Oct 23, 2022

The Deep Learning with Julia book, using Flux.jl.

Deep Learning with Julia DL with Julia is a book about how to do various deep learning tasks using the Julia programming language and specifically the

67 Dec 25, 2022

Official PyTorch implementation of the paper "Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory (SB-FBSDE)"

Official PyTorch implementation of the paper "Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory (SB-FBSDE)" which introduces a new class of deep generative models that gene

43 Jan 03, 2023

Zero-Shot Text-to-Image Generation VQGAN+CLIP Dockerized

VQGAN-CLIP-Docker About Zero-Shot Text-to-Image Generation VQGAN+CLIP Dockerized This is a stripped and minimal dependency repository for running loca

73 Sep 11, 2022

Code for the paper "Learning-Augmented Algorithms for Online Steiner Tree"

Learning-Augmented Algorithms for Online Steiner Tree This is the code for the paper "Learning-Augmented Algorithms for Online Steiner Tree". Requirem

0 Dec 09, 2021

A series of convenience functions to make basic image processing operations such as translation, rotation, resizing, skeletonization, and displaying Matplotlib images easier with OpenCV and Python.

imutils A series of convenience functions to make basic image processing functions such as translation, rotation, resizing, skeletonization, and displ

4.3k Jan 08, 2023

A complete, self-contained example for training ImageNet at state-of-the-art speed with FFCV

ffcv ImageNet Training A minimal, single-file PyTorch ImageNet training script designed for hackability. Run train_imagenet.py to get... ...high accur

92 Dec 31, 2022

Implementation of our paper 'RESA: Recurrent Feature-Shift Aggregator for Lane Detection' in AAAI2021.

RESA PyTorch implementation of the paper "RESA: Recurrent Feature-Shift Aggregator for Lane Detection". Our paper has been accepted by AAAI2021. Intro

137 Jan 02, 2023

Clean and readable code for Decision Transformer: Reinforcement Learning via Sequence Modeling

Minimal implementation of Decision Transformer: Reinforcement Learning via Sequence Modeling in PyTorch for mujoco control tasks in OpenAI gym

104 Jan 06, 2023

Official implementation of the RAVE model: a Realtime Audio Variational autoEncoder

RAVE: Realtime Audio Variational autoEncoder Official implementation of RAVE: A variational autoencoder for fast and high-quality neural audio synthes

587 Jan 01, 2023

[NeurIPS 2021] COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining

COCO-LM This repository contains the scripts for fine-tuning COCO-LM pretrained models on GLUE and SQuAD 2.0 benchmarks. Paper: COCO-LM: Correcting an

106 Dec 12, 2022

The official implementation of ELSA: Enhanced Local Self-Attention for Vision Transformer

ELSA: Enhanced Local Self-Attention for Vision Transformer By Jingkai Zhou, Pich

87 Dec 19, 2022

Single Image Deraining Using Bilateral Recurrent Network (TIP 2020)

Single Image Deraining Using Bilateral Recurrent Network Introduction Single image deraining has received considerable progress based on deep convolut

23 Aug 10, 2022

Command-line tool for downloading and extending the RedCaps dataset.

RedCaps Downloader This repository provides the official command-line tool for downloading and extending the RedCaps dataset. Users can seamlessly dow

33 Dec 14, 2022

Anomaly detection related books, papers, videos, and toolboxes

Anomaly Detection Learning Resources Outlier Detection (also known as Anomaly Detection) is an exciting yet challenging field, which aims to identify

6.7k Dec 31, 2022

A tight inclusion function for continuous collision detection

Tight-Inclusion Continuous Collision Detection A conservative Continuous Collision Detection (CCD) method with support for minimum separation. You can

89 Jan 01, 2023

[CVPR 2022] PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervision (Oral)

PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervision Kehong Gong*, Bingbing Li*, Jianfeng Zhang*, Ta

256 Dec 28, 2022

CNNs for Sentence Classification in PyTorch

Introduction This is the implementation of Kim's Convolutional Neural Networks for Sentence Classification paper in PyTorch. Kim's implementation of t

956 Dec 19, 2022

A collection of Google research projects related to Federated Learning and Federated Analytics.

Federated Research Federated Research is a collection of research projects related to Federated Learning and Federated Analytics. Federated learning i

483 Jan 05, 2023

Latte: Cross-framework Python Package for Evaluation of Latent-based Generative Models

Related tags

Overview

Latte

Installation

Running tests locally

Example

Functional API

Modular API

TorchMetrics API

Keras Metric API

Documentation

Supported metrics

Bundled metric modules

Cite

Comments

Documentation: Metric Descriptions

Add Smoothness and Monotonicity support

Smoothness

Monotonicity

Add Modularity support

Add SAP support

Add DMIG, DLIG, XMIG support

DMIG

XMIG

DLIG

Add MIG support

Support issue for on-the-fly computation in TF2 graph mode