Awesome Transformer Architecture Search:

To keep track of the large number of recent papers that look at the intersection of Transformers and Neural Architecture Search (NAS), we have created this awesome list of curated papers and resources, inspired by awesome-autodl, awesome-architecture-search, and awesome-computer-vision. Papers are divided into the following categories:

General Transformer search
Domain Specific, applied Transformer search (divided into NLP, Vision, ASR)
Insights on Transformer components or searchable parameters
Transformer Surveys

This repository is maintained by the AutoML Group Freiburg. Please feel free to pull requests or open an issue to add papers.

General Transformer Search

Title	Venue	Group
UniNet: Unified Architecture Search with Convolutions, Transformer and MLP	arxiv [Oct'21]	SenseTime
Analyzing and Mitigating Interference in Neural Architecture Search	arxiv [Aug'21]	Tsinghua, MSR
BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search	ICCV'21	Sun Yat-sen University
Memory-Efficient Differentiable Transformer Architecture Search	ACL-IJCNLP'21	MSR, Peking University
Finding Fast Transformers: One-Shot Neural Architecture Search by Component Composition	arxiv [Aug'20]	Google Research
AutoTrans: Automating Transformer Design via Reinforced Architecture Search	arxiv [Sep'20]	Fudan University
NAT: Neural Architecture Transformer for Accurate and Compact Architectures	NeurIPS'19	Tencent AI
The Evolved Transformer	ICML'19	Google Brain

Domain Specific Transformer Search

Vision

Title	Venue	Group
AutoFormer: Searching Transformers for Visual Recognition	ICCV'21	MSR
GLiT: Neural Architecture Search for Global and Local Image Transformer	ICCV'21	University of Sydney
Searching for Efficient Multi-Stage Vision Transformers	ICCV'21 workshop	MIT
HR-NAS: Searching Efficient High-Resolution Neural Architectures with Lightweight Transformers	CVPR'21	Bytedance Inc.
Vision Transformer Architecture Search	arxiv [June'21]	SenseTime, Tsingua University

Natural Language Processing

Title	Venue	Group
AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models	ACL'21	MIT
NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search	KDD'21	MSR, Tsinghua University
AutoBERT-Zero: Evolving the BERT backbone from scratch	arxiv [July'21]	Huawei Noah’s Ark Lab
HAT: Hardware-Aware Transformers for Efficient Natural Language Processing	ACL'20	MIT

Automatic Speech Recognition

Title	Venue	Group
LightSpeech: Lightweight and Fast Text to Speech with Neural Architecture Search	ICASSP'21	MSR
Darts-Conformer: Towards Efficient Gradient-Based Neural Architecture Search For End-to-End ASR	arxiv [Aug'21]	NPU, Xi'an
Improved Conformer-based End-to-End Speech Recognition Using Neural Architecture Search	arxiv [April'21]	Chinese Academy of Sciences
Evolved Speech-Transformer: Applying Neural Architecture Search to End-to-End Automatic Speech Recognition	INTERSPEECH'20	VUNO Inc.

Insights on Transformer components and interesting papers

Title	Venue	Group
Patches are All You Need ?	ICLR'22 under review	-
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows	ICCV'21 best paper	MSR
Rethinking Spatial Dimensions of Vision Transformers	ICCV'21	NAVER AI
What makes for hierarchical vision transformers	arxiv [Sept'21]	HUST
AutoAttend: Automated Attention Representation Search	ICML'21	Tsinghua University
Rethinking Attention with Performers	ICLR'21 Oral	Google
LambdaNetworks: Modeling long-range Interactions without Attention	ICLR'21	Google Research
HyperGrid Transformers	ICLR'21	Google Research
LocalViT: Bringing Locality to Vision Transformers	arxiv [April'21]	ETH Zurich
NASABN: A Neural Architecture Search Framework for Attention-Based Networks	IJCNN'20	Chinese Academy of Sciences
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned	ACL'19	Yandex

Transformer Surveys

Title	Venue	Group
Transformers in Vision: A Survey	arxiv [Oct'21]	MBZ University of AI
Efficient Transformers: A Survey	arxiv [Sept'21]	Google Research

Misc resources

Awesome Visual Transformer

A curated list of awesome resources combining Transformers with Neural Architecture Search

Related tags

Overview

Awesome Transformer Architecture Search:

General Transformer Search

Domain Specific Transformer Search

Vision

Natural Language Processing

Automatic Speech Recognition

Insights on Transformer components and interesting papers

Transformer Surveys

Misc resources

Owner

Yash Mehta

Exploring the Dual-task Correlation for Pose Guided Person Image Generation

An easy-to-use app to visualise attentions of various VQA models.

Space Ship Simulator using python

This repo contains the official implementations of EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis

Tensors and Dynamic neural networks in Python with strong GPU acceleration

Multi Camera Calibration

A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.

Notebooks for my "Deep Learning with TensorFlow 2 and Keras" course

Deep and online learning with spiking neural networks in Python

Code Repository for The Kaggle Book, Published by Packt Publishing

Data Consistency for Magnetic Resonance Imaging

GNN4Traffic - This is the repository for the collection of Graph Neural Network for Traffic Forecasting

Region-aware Contrastive Learning for Semantic Segmentation, ICCV 2021

Adaptable tools to make reinforcement learning and evolutionary computation algorithms.

[ICRA 2022] An opensource framework for cooperative detection. Official implementation for OPV2V.

DAT4 - General Assembly's Data Science course in Washington, DC

Temporal-Relational CrossTransformers

An open source Python package for plasma science that is under development

Evaluation toolkit of the informative tracking benchmark comprising 9 scenarios, 180 diverse videos, and new challenges.

Awesome Transformers in Medical Imaging