Implementation of Hourglass Transformer, in Pytorch, from Google and OpenAI

Last update: Dec 25, 2022

Overview

Hourglass Transformer - Pytorch (wip)

Implementation of Hourglass Transformer, in Pytorch. It will also contain some of my own ideas about how to make it work better.

Citations

@misc{nawrot2021hierarchical,
    title   = {Hierarchical Transformers Are More Efficient Language Models}, 
    author  = {Piotr Nawrot and Szymon Tworkowski and Michał Tyrolski and Łukasz Kaiser and Yuhuai Wu and Christian Szegedy and Henryk Michalewski},
    year    = {2021},
    eprint  = {2110.13711},
    archivePrefix = {arXiv},
    primaryClass = {cs.LG}
}

You might also like...

A large dataset of 100k Google Satellite and matching Map images, resembling pix2pix's Google Maps dataset.

Larger Google Sat2Map dataset This dataset extends the aerial ⟷ Maps dataset used in pix2pix (Isola et al., CVPR17). The provide script download_sat2m

34 Dec 28, 2022

Red Team tool for exfiltrating files from a target's Google Drive that you have access to, via Google's API.

GD-Thief Red Team tool for exfiltrating files from a target's Google Drive that you(the attacker) has access to, via the Google Drive API. This includ

39 Dec 27, 2022

Google-drive-to-sqlite - Create a SQLite database containing metadata from Google Drive

google-drive-to-sqlite Create a SQLite database containing metadata from Google

140 Dec 4, 2022

BasicRL: easy and fundamental codes for deep reinforcement learning。It is an improvement on rainbow-is-all-you-need and OpenAI Spinning Up.

BasicRL: easy and fundamental codes for deep reinforcement learning BasicRL is an improvement on rainbow-is-all-you-need and OpenAI Spinning Up. It is

12 Apr 28, 2022

This is an official implementation for "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" on Object Detection and Instance Segmentation.

Swin Transformer for Object Detection This repo contains the supported code and configuration files to reproduce object detection results of Swin Tran

1.4k Dec 30, 2022

Third party Pytorch implement of Image Processing Transformer (Pre-Trained Image Processing Transformer arXiv:2012.00364v2)

ImageProcessingTransformer Third party Pytorch implement of Image Processing Transformer (Pre-Trained Image Processing Transformer arXiv:2012.00364v2)

61 Jan 1, 2023

Transformer - Transformer in PyTorch

Transformer 完成进度 Embeddings and PositionalEncoding with example. MultiHeadAttent

1 Jan 6, 2022

Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network)

Deep Daze mist over green hills shattered plates on the grass cosmic love and attention a time traveler in the crowd life during the plague meditative

4.4k Jan 3, 2023

Plug-n-Play Reinforcement Learning in Python with OpenAI Gym and JAX

coax is built on top of JAX, but it doesn't have an explicit dependence on the jax python package. The reason is that your version of jaxlib will depend on your CUDA version.

128 Dec 27, 2022

Comments

No option for rotary positional embeddings

For the ImageNet64 generation experiments, the original authors used rotary positional embeddings. I think it would be worth adding this feature to the repository.

I can prepare a pull request that solves this~

opened by vvvm23 4
Issue with an example

@lucidrains Just Awesome to see this super fast launch !

The third example under "For Images...." throws the following error .,

Then , when I gave the inputs with num_tokens = 20000, max_seq_len = 1024, I got the following runtime error

opened by nsankar 1

Releases(0.0.6)

0.0.6(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.5(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.4(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.3(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.2(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.1(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)

Owner

Phil Wang

Working with Attention. It's all we need

GitHub Repository

For holding anime-related object classification and detection models

Animesion An end-to-end framework for anime-related object classification, detection, segmentation, and other models. Update: 01/22/2020. Due to time-

72 Nov 30, 2022

用opencv的dnn模块做yolov5目标检测，包含C++和Python两个版本的程序

yolov5-dnn-cpp-py yolov5s,yolov5l,yolov5m,yolov5x的onnx文件在百度云盘下载，链接：https://pan.baidu.com/s/1d67LUlOoPFQy0MV39gpJiw 提取码：bayj python版本的主程序是main_yolov5.

365 Jan 04, 2023

Large dataset storage format for Pytorch

H5Record Large dataset ( 100G, = 1T) storage format for Pytorch (wip) Support python 3 pip install h5record Why? Writing large dataset is still a

43 Oct 22, 2022

Learning Lightweight Low-Light Enhancement Network using Pseudo Well-Exposed Images

Learning Lightweight Low-Light Enhancement Network using Pseudo Well-Exposed Images This repository contains the implementation of the following paper

9 Jul 30, 2022

GalaXC: Graph Neural Networks with Labelwise Attention for Extreme Classification

GalaXC GalaXC: Graph Neural Networks with Labelwise Attention for Extreme Classification @InProceedings{Saini21, author = {Saini, D. and Jain,

28 Dec 05, 2022

Official implementation of the NeurIPS'21 paper 'Conditional Generation Using Polynomial Expansions'.

Conditional Generation Using Polynomial Expansions Official implementation of the conditional image generation experiments as described on the NeurIPS

4 Aug 07, 2022

Python implementation of Wu et al (2018)'s registration fusion

reg-fusion Projection of a central sulcus probability map using the RF-ANTs approach (right hemisphere shown). This is a Python implementation of Wu e

26 Nov 12, 2021

Python version of the amazing Reaction Mechanism Generator (RMG).

Reaction Mechanism Generator (RMG) Description This repository contains the Python version of Reaction Mechanism Generator (RMG), a tool for automatic

284 Dec 27, 2022

Code for "Unsupervised Source Separation via Bayesian inference in the latent domain"

LQVAE-separation Code for "Unsupervised Source Separation via Bayesian inference in the latent domain" Paper Samples GT Compressed Separated Drums GT

30 Oct 25, 2022

ShuttleNet: Position-aware Fusion of Rally Progress and Player Styles for Stroke Forecasting in Badminton (AAAI'22)

ShuttleNet: Position-aware Rally Progress and Player Styles Fusion for Stroke Forecasting in Badminton (AAAI 2022) Official code of the paper ShuttleN

11 Nov 30, 2022

🔎 Monitor deep learning model training and hardware usage from your mobile phone 📱

Monitor deep learning model training and hardware usage from mobile. 🔥 Features Monitor running experiments from mobile phone (or laptop) Monitor har

1.2k Dec 25, 2022

Generative Models as a Data Source for Multiview Representation Learning

GenRep Project Page | Paper Generative Models as a Data Source for Multiview Representation Learning Ali Jahanian, Xavier Puig, Yonglong Tian, Phillip

81 Dec 03, 2022

Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more. Runs on single machine, Hadoop, Spark, Dask, Flink and DataFlow

eXtreme Gradient Boosting Community | Documentation | Resources | Contributors | Release Notes XGBoost is an optimized distributed gradient boosting l

23.6k Dec 31, 2022

Implementation of Hourglass Transformer, in Pytorch, from Google and OpenAI

Related tags

Overview

Hourglass Transformer - Pytorch (wip)

Citations

You might also like...

A large dataset of 100k Google Satellite and matching Map images, resembling pix2pix's Google Maps dataset.

Red Team tool for exfiltrating files from a target's Google Drive that you have access to, via Google's API.

Google-drive-to-sqlite - Create a SQLite database containing metadata from Google Drive

BasicRL: easy and fundamental codes for deep reinforcement learning。It is an improvement on rainbow-is-all-you-need and OpenAI Spinning Up.

This is an official implementation for "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" on Object Detection and Instance Segmentation.

Third party Pytorch implement of Image Processing Transformer (Pre-Trained Image Processing Transformer arXiv:2012.00364v2)

Transformer - Transformer in PyTorch

Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network)

Plug-n-Play Reinforcement Learning in Python with OpenAI Gym and JAX

Comments

No option for rotary positional embeddings

Issue with an example

Releases(0.0.6)

0.0.6(Nov 9, 2021)

0.0.5(Nov 9, 2021)

0.0.4(Nov 9, 2021)

0.0.3(Nov 9, 2021)

0.0.2(Nov 9, 2021)

0.0.1(Nov 9, 2021)

Owner

Phil Wang

For holding anime-related object classification and detection models

用opencv的dnn模块做yolov5目标检测，包含C++和Python两个版本的程序

Large dataset storage format for Pytorch

Learning Lightweight Low-Light Enhancement Network using Pseudo Well-Exposed Images

GalaXC: Graph Neural Networks with Labelwise Attention for Extreme Classification

Official implementation of the NeurIPS'21 paper 'Conditional Generation Using Polynomial Expansions'.

Python implementation of Wu et al (2018)'s registration fusion

Python version of the amazing Reaction Mechanism Generator (RMG).

Code for "Unsupervised Source Separation via Bayesian inference in the latent domain"

ShuttleNet: Position-aware Fusion of Rally Progress and Player Styles for Stroke Forecasting in Badminton (AAAI'22)

🔎 Monitor deep learning model training and hardware usage from your mobile phone 📱

Generative Models as a Data Source for Multiview Representation Learning

Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more. Runs on single machine, Hadoop, Spark, Dask, Flink and DataFlow

An implementation of quantum convolutional neural network with MindQuantum. Huawei, classifying MNIST dataset

Large-scale language modeling tutorials with PyTorch

Object recognition using Azure Custom Vision AI and Azure Functions

Vignette is a face tracking software for characters using osu!framework.

Approaches to modeling terrain and maps in python

Instant-nerf-pytorch - NeRF trained SUPER FAST in pytorch

An open source object detection toolbox based on PyTorch