Implementation of the 😇 Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones

Last update: Nov 22, 2022

Overview

HaloNet - Pytorch

Implementation of the Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones. This repository will only house the attention layer and not much more.

Install

$ pip install halonet-pytorch

Usage

import torch
from halonet_pytorch import HaloAttention

attn = HaloAttention(
    dim = 512,         # dimension of feature map
    block_size = 8,    # neighborhood block size (feature map must be divisible by this)
    halo_size = 4,     # halo size (block receptive field)
    dim_head = 64,     # dimension of each head
    heads = 4          # number of attention heads
).cuda()

fmap = torch.randn(1, 512, 32, 32).cuda()
attn(fmap) # (1, 512, 32, 32)

Citations

@misc{vaswani2021scaling,
    title   = {Scaling Local Self-Attention For Parameter Efficient Visual Backbones}, 
    author  = {Ashish Vaswani and Prajit Ramachandran and Aravind Srinivas and Niki Parmar and Blake Hechtman and Jonathon Shlens},
    year    = {2021},
    eprint  = {2103.12731},
    archivePrefix = {arXiv},
    primaryClass = {cs.CV}
}

You might also like...

The official implementation of ELSA: Enhanced Local Self-Attention for Vision Transformer

ELSA: Enhanced Local Self-Attention for Vision Transformer By Jingkai Zhou, Pich

87 Dec 19, 2022

Unofficial PyTorch reimplementation of the paper Swin Transformer V2: Scaling Up Capacity and Resolution

PyTorch reimplementation of the paper Swin Transformer V2: Scaling Up Capacity and Resolution [arXiv 2021].

122 Dec 12, 2022

Official code for "Focal Self-attention for Local-Global Interactions in Vision Transformers"

Focal Transformer This is the official implementation of our Focal Transformer -- "Focal Self-attention for Local-Global Interactions in Vision Transf

486 Dec 20, 2022

A PyTorch implementation of " EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks."

EfficientNet A PyTorch implementation of EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. [arxiv] [Official TF Repo] Implemen

298 Dec 10, 2022

Official pytorch implementation of "Scaling-up Disentanglement for Image Translation", ICCV 2021.

41 Nov 29, 2022

Implementation of "Scaled-YOLOv4: Scaling Cross Stage Partial Network" using PyTorch framwork.

YOLOv4-large This is the implementation of "Scaled-YOLOv4: Scaling Cross Stage Partial Network" using PyTorch framwork. YOLOv4-CSP YOLOv4-tiny YOLOv4-

2k Jan 2, 2023

Official Pytorch Implementation of Relational Self-Attention: What's Missing in Attention for Video Understanding

Relational Self-Attention: What's Missing in Attention for Video Understanding This repository is the official implementation of "Relational Self-Atte

43 Dec 7, 2022

Image-Scaling Attacks and Defenses

Image-Scaling Attacks & Defenses This repository belongs to our publication: Erwin Quiring, David Klein, Daniel Arp, Martin Johns and Konrad Rieck. Ad

163 Nov 21, 2022

For auto aligning, cropping, and scaling HR and LR images for training image based neural networks

ImgAlign For auto aligning, cropping, and scaling HR and LR images for training image based neural networks Usage Make sure OpenCV is installed, 'pip

15 Dec 4, 2022

Implementation of the 😇 Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones

Related tags

Overview

HaloNet - Pytorch

Install

Usage

Citations

You might also like...

The official implementation of ELSA: Enhanced Local Self-Attention for Vision Transformer

Unofficial PyTorch reimplementation of the paper Swin Transformer V2: Scaling Up Capacity and Resolution

Official code for "Focal Self-attention for Local-Global Interactions in Vision Transformers"

A PyTorch implementation of " EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks."

Official pytorch implementation of "Scaling-up Disentanglement for Image Translation", ICCV 2021.

Implementation of "Scaled-YOLOv4: Scaling Cross Stage Partial Network" using PyTorch framwork.

Official Pytorch Implementation of Relational Self-Attention: What's Missing in Attention for Video Understanding

Image-Scaling Attacks and Defenses

For auto aligning, cropping, and scaling HR and LR images for training image based neural networks

Releases(0.0.4)

0.0.4(Mar 24, 2021)

0.0.3(Mar 24, 2021)

0.0.2(Mar 24, 2021)

0.0.1(Mar 24, 2021)

Owner

Phil Wang

Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding

deep learning for image processing including classification and object-detection etc.

DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting

A Closer Look at Structured Pruning for Neural Network Compression

Tensorflow 2 implementation of the paper: Learning and Evaluating Representations for Deep One-class Classification published at ICLR 2021

NumQMBasic - A mini-course offered to Undergrad physics students

Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.

Official codebase for Decision Transformer: Reinforcement Learning via Sequence Modeling.

A keras-based real-time model for medical image segmentation (CFPNet-M)

Inferred Model-based Fuzzer

Pythonic particle-based (super-droplet) warm-rain/aqueous-chemistry cloud microphysics package with box, parcel & 1D/2D prescribed-flow examples in Python, Julia and Matlab

Tightness-aware Evaluation Protocol for Scene Text Detection

Style transfer, deep learning, feature transform

IGCN : Image-to-graph convolutional network

A collection of models for image<->text generation in ACM MM 2021.

PyTorch implementation of Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy

Parallel Latent Tree-Induction for Faster Sequence Encoding

Histology images query (unsupervised)

Implementation of the paper Recurrent Glimpse-based Decoder for Detection with Transformer.

Voice of Pajlada with model and weights.