Local Attention - Flax module for Jax

Last update: Jun 16, 2022

Overview

Local Attention - Flax

Autoregressive Local Attention - Flax module for Jax

Install

$ pip install local-attention-flax

Usage

from jax import random
from local_attention_flax import LocalAttention

attn = LocalAttention(
    dim = 256,
    dim_head = 64,
    heads = 8,
    window_size = 128
)

key = random.PRNGKey(0)
x = random.normal(key, (512, 256))

params = attn.init(key, x)
out = attn.apply(params, x)  # (512, 256)

CLOOB training (JAX) and inference (JAX and PyTorch)

cloob-training Pretrained models There are two pretrained CLOOB models in this repo at the moment, a 16 epoch and a 32 epoch ViT-B/16 checkpoint train

64 Nov 27, 2022

Implementation of the 😇 Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones

HaloNet - Pytorch Implementation of the Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones. This re

189 Nov 22, 2022

Locally Enhanced Self-Attention: Rethinking Self-Attention as Local and Context Terms

LESA Introduction This repository contains the official implementation of Locally Enhanced Self-Attention: Rethinking Self-Attention as Local and Cont

20 Dec 31, 2021

计算机视觉中用到的注意力模块和其他即插即用模块PyTorch Implementation Collection of Attention Module and Plug&Play Module

PyTorch实现多种计算机视觉中网络设计中用到的Attention机制，还收集了一些即插即用模块。由于能力有限精力有限，可能很多模块并没有包括进来，有任何的建议或者改进，可以提交issue或者进行PR。

599 Dec 23, 2022

Implementation of Invariant Point Attention, used for coordinate refinement in the structure module of Alphafold2, as a standalone Pytorch module

Invariant Point Attention - Pytorch Implementation of Invariant Point Attention as a standalone module, which was used in the structure module of Alph

113 Jan 5, 2023

RoBERTa Marathi Language model trained from scratch during huggingface 🤗 x flax community week

RoBERTa base model for Marathi Language (मराठी भाषा) Pretrained model on Marathi language using a masked language modeling (MLM) objective. RoBERTa wa

23 Oct 19, 2022

Memory Efficient Attention (O(sqrt(n)) for Jax and PyTorch

Memory Efficient Attention This is unofficial implementation of Self-attention Does Not Need O(n^2) Memory for Jax and PyTorch. Implementation is almo

126 Dec 27, 2022

A Pytree Module system for Deep Learning in JAX

Treex A Pytree-based Module system for Deep Learning in JAX Intuitive: Modules are simple Python objects that respect Object-Oriented semantics and sh

216 Dec 20, 2022

Graph neural network message passing reframed as a Transformer with local attention

Adjacent Attention Network An implementation of a simple transformer that is equivalent to graph neural network where the message passing is done with

49 Dec 28, 2022

Releases(0.0.2)

0.0.2(May 26, 2021)

Source code(tar.gz)
Source code(zip)
0.0.1(May 26, 2021)

Source code(tar.gz)
Source code(zip)

Owner

Phil Wang

Working with Attention

GitHub Repository

Transformers provides thousands of pretrained models to perform tasks on different modalities such as text, vision, and audio.

English | 简体中文 | 繁體中文 | 한국어 State-of-the-art Machine Learning for JAX, PyTorch and TensorFlow 🤗 Transformers provides thousands of pretrained models

50 Nov 12, 2022

Official implementation of NeurIPS 2021 paper "One Loss for All: Deep Hashing with a Single Cosine Similarity based Learning Objective"

71 Dec 22, 2022

This is an implementation for the CVPR2020 paper "Learning Invariant Representation for Unsupervised Image Restoration"

Learning Invariant Representation for Unsupervised Image Restoration (CVPR 2020) Introduction This is an implementation for the paper "Learning Invari

88 Nov 07, 2022

An implementation of Video Frame Interpolation via Adaptive Separable Convolution using PyTorch

This work has now been superseded by: https://github.com/sniklaus/revisiting-sepconv sepconv-slomo This is a reference implementation of Video Frame I

984 Dec 16, 2022

This is a work in progress reimplementation of Instant Neural Graphics Primitives

Neural Hash Encoding This is a work in progress reimplementation of Instant Neural Graphics Primitives Currently this can train an implicit representa

79 Sep 01, 2022

eXPeditious Data Transfer

xpdt: eXPeditious Data Transfer About xpdt is (yet another) language for defining data-types and generating code for serializing and deserializing the

3 Jan 06, 2022

This repository contains the code for the paper "Hierarchical Motion Understanding via Motion Programs"

Hierarchical Motion Understanding via Motion Programs (CVPR 2021) This repository contains the official implementation of: Hierarchical Motion Underst

40 Dec 05, 2022

ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure

ViViT is a collection of numerical tricks to efficiently access curvature from the generalized Gauss-Newton (GGN) matrix based on its low-rank structure. Provided functionality includes computing

12 Dec 08, 2022

Deep Ensemble Learning with Jet-Like architecture

Ransomware analysis using DEL with jet-like architecture comprising two CNN wings, a sparse AE tail, a non-linear PCA to produce a diverse feature space, and an MLP nose

2 Feb 06, 2022

Automatic Image Background Subtraction

Automatic Image Background Subtraction This repo contains set of scripts for automatic one-shot image background subtraction task using the following

6 Dec 05, 2022

Single cell current best practices tutorial case study for the paper:Luecken and Theis, "Current best practices in single-cell RNA-seq analysis: a tutorial"

Scripts for "Current best-practices in single-cell RNA-seq: a tutorial" This repository is complementary to the publication: M.D. Luecken, F.J. Theis,

968 Dec 28, 2022

Local Attention - Flax module for Jax

Related tags

Overview

Local Attention - Flax

Install

Usage

You might also like...

CLOOB training (JAX) and inference (JAX and PyTorch)

Implementation of the 😇 Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones

Locally Enhanced Self-Attention: Rethinking Self-Attention as Local and Context Terms

计算机视觉中用到的注意力模块和其他即插即用模块PyTorch Implementation Collection of Attention Module and Plug&Play Module

Implementation of Invariant Point Attention, used for coordinate refinement in the structure module of Alphafold2, as a standalone Pytorch module

RoBERTa Marathi Language model trained from scratch during huggingface 🤗 x flax community week

Memory Efficient Attention (O(sqrt(n)) for Jax and PyTorch

A Pytree Module system for Deep Learning in JAX

Graph neural network message passing reframed as a Transformer with local attention

Releases(0.0.2)

0.0.2(May 26, 2021)

0.0.1(May 26, 2021)

Owner

Phil Wang

Transformers provides thousands of pretrained models to perform tasks on different modalities such as text, vision, and audio.

Official implementation of NeurIPS 2021 paper "One Loss for All: Deep Hashing with a Single Cosine Similarity based Learning Objective"

This is an implementation for the CVPR2020 paper "Learning Invariant Representation for Unsupervised Image Restoration"

An implementation of Video Frame Interpolation via Adaptive Separable Convolution using PyTorch

This is a work in progress reimplementation of Instant Neural Graphics Primitives

eXPeditious Data Transfer

This repository contains the code for the paper "Hierarchical Motion Understanding via Motion Programs"

ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure

Deep Ensemble Learning with Jet-Like architecture

Automatic Image Background Subtraction

Single cell current best practices tutorial case study for the paper:Luecken and Theis, "Current best practices in single-cell RNA-seq analysis: a tutorial"

YKKDetector For Python

Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration

Trains an agent with stochastic policy gradient ascent to solve the Lunar Lander challenge from OpenAI

Yolov5 + Deep Sort with PyTorch

CS50's Introduction to Artificial Intelligence Test Scripts

Attentive Implicit Representation Networks (AIR-Nets)

Kernel Point Convolutions

An air quality monitoring service with a Raspberry Pi and a SDS011 sensor.

Machine Unlearning with SISA