Deformable Attention
Implementation of Deformable Attention from this paper in Pytorch, which appears to be an improvement to what was proposed in DETR. The relative positional embedding has also been modified for better extrapolation, using the Continuous Positional Embedding proposed in SwinV2.
Install
$ pip install deformable-attention
Usage
import torch
from deformable_attention import DeformableAttention
attn = DeformableAttention(
dim = 512, # feature dimensions
dim_head = 64, # dimension per head
heads = 8, # attention heads
dropout = 0., # dropout
downsample_factor = 4, # downsample factor (r in paper)
offset_scale = 4, # scale of offset, maximum offset
offset_groups = None, # number of offset groups, should be multiple of heads
offset_kernel_size = 6, # offset kernel size
)
x = torch.randn(1, 512, 64, 64)
attn(x) # (1, 512, 64, 64)
3d deformable attention
import torch
from deformable_attention import DeformableAttention3D
attn = DeformableAttention3D(
dim = 512, # feature dimensions
dim_head = 64, # dimension per head
heads = 8, # attention heads
dropout = 0., # dropout
downsample_factor = (2, 8, 8), # downsample factor (r in paper)
offset_scale = (2, 8, 8), # scale of offset, maximum offset
offset_kernel_size = (4, 10, 10), # offset kernel size
)
x = torch.randn(1, 512, 10, 32, 32) # (batch, dimension, frames, height, width)
attn(x) # (1, 512, 10, 32, 32)
1d deformable attention for good measure
import torch
from deformable_attention import DeformableAttention1D
attn = DeformableAttention1D(
dim = 128,
downsample_factor = 4,
offset_scale = 2,
offset_kernel_size = 6
)
x = torch.randn(1, 128, 512)
attn(x) # (1, 128, 512)
Citation
@misc{xia2022vision,
title = {Vision Transformer with Deformable Attention},
author = {Zhuofan Xia and Xuran Pan and Shiji Song and Li Erran Li and Gao Huang},
year = {2022},
eprint = {2201.00520},
archivePrefix = {arXiv},
primaryClass = {cs.CV}
}
@misc{liu2021swin,
title = {Swin Transformer V2: Scaling Up Capacity and Resolution},
author = {Ze Liu and Han Hu and Yutong Lin and Zhuliang Yao and Zhenda Xie and Yixuan Wei and Jia Ning and Yue Cao and Zheng Zhang and Li Dong and Furu Wei and Baining Guo},
year = {2021},
eprint = {2111.09883},
archivePrefix = {arXiv},
primaryClass = {cs.CV}
}
5 Nov 21, 2022
28 Dec 15, 2022
1 Jan 7, 2022
189 Nov 22, 2022
43 Dec 7, 2022
1k Dec 31, 2022
84 Jan 4, 2023
45 Dec 26, 2022
176 Dec 30, 2022
64 Dec 27, 2022
64 Jan 07, 2023
1 Nov 12, 2021
167 Dec 29, 2022
133 Dec 21, 2022
44 Nov 20, 2022
16 Sep 11, 2021
2 Dec 01, 2022
28 Dec 25, 2022
319 Dec 30, 2022
16 Nov 21, 2022
15 Nov 08, 2022
108 Jan 05, 2023
4 Feb 14, 2022
153 Jan 04, 2023
12 Jan 05, 2023
1 Sep 18, 2022
119 Dec 04, 2022