Toolbox to analyze temporal context invariance of deep neural networks

Last update: Oct 23, 2022

Overview

PyTCI

A toolbox that estimates the integration window of a sensory response using the "Temporal Context Invariance" paradigm (TCI).

The TCI method

Integration windows are defined as the time window within which stimuli alter a sensory response and outside of which stimuli have little effect. Integration windows provide a simple and general way to define the analysis timescale of a response. We estimate integration windows by presenting segments of natural stimuli in two different pseudorandom orders, such that the same segment occurs in two different contexts (is surrounded by different segments). We then estimate the smallest segment duration outside of which stimuli have little effect on the response. The TCI paradigm was initially developed to estimate integration windows for biological neural systems:

Multiscale integration organizes hierarchical computation in human auditory cortex

The method however can be applied to any sensory response, and we have recently used the method to understand how deep speech recognition systems learn to flexibly integrate across multiple timescales:

Understanding Adaptive, Multiscale Temporal Integration In Deep Speech Recognition Systems

This toolbox implements the analyses described in the above NeurIPS paper. We estimate context invariance using the "cross-context correlation" and then estimate the integration window by finding the smallest segment duration needed to achieve a given correlation threshold. Note that this approach is not robust to data noise and thus is not appropriate for biological neural systems (we will be releasing a different toolbox soon that addresses this limitation).

Usage

To demonstrate the application of the toolbox, two Jupyter notebooks have been provided in the "Examples" directory:

Example-Toy: Shows how to apply the TCI method to a toy model that integrates sound energy within a gamma-distributed window. Covers most of the functionality of the toolbox.
Example-DeepSpeech: Shows how to use the TCI method to estimate integration windows from the DeepSpeech2 model described in the paper, implemented in PyTorch. This notebook requires the pretrained model and speech audio clips in Examples/resources.tar to be extracted and placed in a directory named Examples/resources. It also has extra dependencies that need to be installed.

NOTE: These notebooks might not render as intended on GitHub. For correct rendering, open them locally in a Jupyter notebook or using nbviewer.

Installation

To install or update this package through pip, run the following command:

pip install git+https://github.com/naplab/PyTCI.git

Spontaneous Facial Micro Expression Recognition using 3D Spatio-Temporal Convolutional Neural Networks

Spontaneous Facial Micro Expression Recognition using 3D Spatio-Temporal Convolutional Neural Networks Abstract Facial expression recognition in video

103 Dec 29, 2022

Complex-Valued Neural Networks (CVNN)Complex-Valued Neural Networks (CVNN)

Complex-Valued Neural Networks (CVNN) Done by @NEGU93 - J. Agustin Barrachina Using this library, the only difference with a Tensorflow code is that y

1 Nov 12, 2021

A framework that constructs deep neural networks, autoencoders, logistic regressors, and linear networks

A framework that constructs deep neural networks, autoencoders, logistic regressors, and linear networks without the use of any outside machine learning libraries - all from scratch.

2 Nov 14, 2022

Bayesian-Torch is a library of neural network layers and utilities extending the core of PyTorch to enable the user to perform stochastic variational inference in Bayesian deep neural networks

Bayesian-Torch is a library of neural network layers and utilities extending the core of PyTorch to enable the user to perform stochastic variational inference in Bayesian deep neural networks. Bayesian-Torch is designed to be flexible and seamless in extending a deterministic deep neural network architecture to corresponding Bayesian form by simply replacing the deterministic layers with Bayesian layers.

210 Jan 4, 2023

An implementation demo of the ICLR 2021 paper Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks in PyTorch.

Neural Attention Distillation This is an implementation demo of the ICLR 2021 paper Neural Attention Distillation: Erasing Backdoor Triggers from Deep

84 Jan 4, 2023

DeepHyper: Scalable Asynchronous Neural Architecture and Hyperparameter Search for Deep Neural Networks

What is DeepHyper? DeepHyper is a software package that uses learning, optimization, and parallel computing to automate the design and development of

214 Jan 8, 2023

Unofficial implementation of Point-Unet: A Context-Aware Point-Based Neural Network for Volumetric Segmentation

Point-Unet This is an unofficial implementation of the MICCAI 2021 paper Point-Unet: A Context-Aware Point-Based Neural Network for Volumetric Segment

9 Dec 7, 2022

This repository contains notebook implementations of the following Neural Process variants: Conditional Neural Processes (CNPs), Neural Processes (NPs), Attentive Neural Processes (ANPs).

The Neural Process Family This repository contains notebook implementations of the following Neural Process variants: Conditional Neural Processes (CN

892 Dec 28, 2022

IDRLnet, a Python toolbox for modeling and solving problems through Physics-Informed Neural Network (PINN) systematically.

IDRLnet IDRLnet is a machine learning library on top of PyTorch. Use IDRLnet if you need a machine learning library that solves both forward and inver

105 Dec 17, 2022

Toolbox to analyze temporal context invariance of deep neural networks

Related tags

Overview

PyTCI

The TCI method

Usage

Installation

You might also like...

Spontaneous Facial Micro Expression Recognition using 3D Spatio-Temporal Convolutional Neural Networks

Complex-Valued Neural Networks (CVNN)Complex-Valued Neural Networks (CVNN)

A framework that constructs deep neural networks, autoencoders, logistic regressors, and linear networks

Bayesian-Torch is a library of neural network layers and utilities extending the core of PyTorch to enable the user to perform stochastic variational inference in Bayesian deep neural networks

An implementation demo of the ICLR 2021 paper Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks in PyTorch.

DeepHyper: Scalable Asynchronous Neural Architecture and Hyperparameter Search for Deep Neural Networks

Unofficial implementation of Point-Unet: A Context-Aware Point-Based Neural Network for Volumetric Segmentation

This repository contains notebook implementations of the following Neural Process variants: Conditional Neural Processes (CNPs), Neural Processes (NPs), Attentive Neural Processes (ANPs).

IDRLnet, a Python toolbox for modeling and solving problems through Physics-Informed Neural Network (PINN) systematically.

Releases(v0.2)

v0.2(Nov 18, 2021)

Owner

Subdivision-based Mesh Convolutional Networks

Vis2Mesh: Efficient Mesh Reconstruction from Unstructured Point Clouds of Large Scenes with Learned Virtual View Visibility ICCV2021

yolov5 deepsort 行人车辆跟踪检测计数

The final project of "Applying AI to 2D Medical Imaging Data" of "AI for Healthcare" nanodegree - Udacity.

Large-scale open domain KNOwledge grounded conVERsation system based on PaddlePaddle

An AI made using artificial intelligence (AI) and machine learning algorithms (ML) .

The official code repo of "HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection"

The Malware Open-source Threat Intelligence Family dataset contains 3,095 disarmed PE malware samples from 454 families

Pytorch implementation of Integrating Tree Path in Transformer for Code Representation

Autotype on websites that have copy-paste disabled like Moodle, HackerEarth contest etc.

Code for LIGA-Stereo Detector, ICCV'21

GeoMol: Torsional Geometric Generation of Molecular 3D Conformer Ensembles

A PyTorch implementation of "From Two to One: A New Scene Text Recognizer with Visual Language Modeling Network" (ICCV2021)

Numenta Platform for Intelligent Computing is an implementation of Hierarchical Temporal Memory (HTM), a theory of intelligence based strictly on the neuroscience of the neocortex.

Code for the paper Hybrid Spectrogram and Waveform Source Separation

Offical code for the paper: "Growing 3D Artefacts and Functional Machines with Neural Cellular Automata" https://arxiv.org/abs/2103.08737

StyleGAN2 Webtoon / Anime Style Toonify

[ACL-IJCNLP 2021] Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning

Bringing Characters to Life with Computer Brains in Unity

MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Research

Toolbox to analyze temporal context invariance of deep neural networks

Related tags

Overview

PyTCI

The TCI method

Usage

Installation

You might also like...

Spontaneous Facial Micro Expression Recognition using 3D Spatio-Temporal Convolutional Neural Networks

Complex-Valued Neural Networks (CVNN)Complex-Valued Neural Networks (CVNN)

A framework that constructs deep neural networks, autoencoders, logistic regressors, and linear networks

Bayesian-Torch is a library of neural network layers and utilities extending the core of PyTorch to enable the user to perform stochastic variational inference in Bayesian deep neural networks

An implementation demo of the ICLR 2021 paper Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks in PyTorch.

DeepHyper: Scalable Asynchronous Neural Architecture and Hyperparameter Search for Deep Neural Networks

Unofficial implementation of Point-Unet: A Context-Aware Point-Based Neural Network for Volumetric Segmentation

This repository contains notebook implementations of the following Neural Process variants: Conditional Neural Processes (CNPs), Neural Processes (NPs), Attentive Neural Processes (ANPs).

IDRLnet, a Python toolbox for modeling and solving problems through Physics-Informed Neural Network (PINN) systematically.

Releases(v0.2)

v0.2(Nov 18, 2021)

Owner

Subdivision-based Mesh Convolutional Networks

Vis2Mesh: Efficient Mesh Reconstruction from Unstructured Point Clouds of Large Scenes with Learned Virtual View Visibility ICCV2021

yolov5 deepsort 行人 车辆 跟踪 检测 计数

The final project of "Applying AI to 2D Medical Imaging Data" of "AI for Healthcare" nanodegree - Udacity.

Large-scale open domain KNOwledge grounded conVERsation system based on PaddlePaddle

An AI made using artificial intelligence (AI) and machine learning algorithms (ML) .

The official code repo of "HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection"

The Malware Open-source Threat Intelligence Family dataset contains 3,095 disarmed PE malware samples from 454 families

Pytorch implementation of Integrating Tree Path in Transformer for Code Representation

Autotype on websites that have copy-paste disabled like Moodle, HackerEarth contest etc.

Code for LIGA-Stereo Detector, ICCV'21

GeoMol: Torsional Geometric Generation of Molecular 3D Conformer Ensembles

A PyTorch implementation of "From Two to One: A New Scene Text Recognizer with Visual Language Modeling Network" (ICCV2021)

Numenta Platform for Intelligent Computing is an implementation of Hierarchical Temporal Memory (HTM), a theory of intelligence based strictly on the neuroscience of the neocortex.

Code for the paper Hybrid Spectrogram and Waveform Source Separation

Offical code for the paper: "Growing 3D Artefacts and Functional Machines with Neural Cellular Automata" https://arxiv.org/abs/2103.08737

StyleGAN2 Webtoon / Anime Style Toonify

[ACL-IJCNLP 2021] Improving Named Entity Recognition by External Context Retrieving and Cooperative Learning

Bringing Characters to Life with Computer Brains in Unity

MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Research

yolov5 deepsort 行人车辆跟踪检测计数