Implementation of Hourglass Transformer, in Pytorch, from Google and OpenAI

Last update: Dec 25, 2022

Overview

Hourglass Transformer - Pytorch (wip)

Implementation of Hourglass Transformer, in Pytorch. It will also contain some of my own ideas about how to make it work better.

Citations

@misc{nawrot2021hierarchical,
    title   = {Hierarchical Transformers Are More Efficient Language Models}, 
    author  = {Piotr Nawrot and Szymon Tworkowski and Michał Tyrolski and Łukasz Kaiser and Yuhuai Wu and Christian Szegedy and Henryk Michalewski},
    year    = {2021},
    eprint  = {2110.13711},
    archivePrefix = {arXiv},
    primaryClass = {cs.LG}
}

You might also like...

A large dataset of 100k Google Satellite and matching Map images, resembling pix2pix's Google Maps dataset.

Larger Google Sat2Map dataset This dataset extends the aerial ⟷ Maps dataset used in pix2pix (Isola et al., CVPR17). The provide script download_sat2m

34 Dec 28, 2022

Red Team tool for exfiltrating files from a target's Google Drive that you have access to, via Google's API.

GD-Thief Red Team tool for exfiltrating files from a target's Google Drive that you(the attacker) has access to, via the Google Drive API. This includ

39 Dec 27, 2022

Google-drive-to-sqlite - Create a SQLite database containing metadata from Google Drive

google-drive-to-sqlite Create a SQLite database containing metadata from Google

140 Dec 4, 2022

BasicRL: easy and fundamental codes for deep reinforcement learning。It is an improvement on rainbow-is-all-you-need and OpenAI Spinning Up.

BasicRL: easy and fundamental codes for deep reinforcement learning BasicRL is an improvement on rainbow-is-all-you-need and OpenAI Spinning Up. It is

12 Apr 28, 2022

This is an official implementation for "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" on Object Detection and Instance Segmentation.

Swin Transformer for Object Detection This repo contains the supported code and configuration files to reproduce object detection results of Swin Tran

1.4k Dec 30, 2022

Third party Pytorch implement of Image Processing Transformer (Pre-Trained Image Processing Transformer arXiv:2012.00364v2)

ImageProcessingTransformer Third party Pytorch implement of Image Processing Transformer (Pre-Trained Image Processing Transformer arXiv:2012.00364v2)

61 Jan 1, 2023

Transformer - Transformer in PyTorch

Transformer 完成进度 Embeddings and PositionalEncoding with example. MultiHeadAttent

1 Jan 6, 2022

Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network)

Deep Daze mist over green hills shattered plates on the grass cosmic love and attention a time traveler in the crowd life during the plague meditative

4.4k Jan 3, 2023

Plug-n-Play Reinforcement Learning in Python with OpenAI Gym and JAX

coax is built on top of JAX, but it doesn't have an explicit dependence on the jax python package. The reason is that your version of jaxlib will depend on your CUDA version.

128 Dec 27, 2022

Comments

No option for rotary positional embeddings

For the ImageNet64 generation experiments, the original authors used rotary positional embeddings. I think it would be worth adding this feature to the repository.

I can prepare a pull request that solves this~

opened by vvvm23 4
Issue with an example

@lucidrains Just Awesome to see this super fast launch !

The third example under "For Images...." throws the following error .,

Then , when I gave the inputs with num_tokens = 20000, max_seq_len = 1024, I got the following runtime error

opened by nsankar 1

Releases(0.0.6)

0.0.6(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.5(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.4(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.3(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.2(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.1(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)

Owner

Phil Wang

Working with Attention. It's all we need

GitHub Repository

Code to reproduce results from the paper "AmbientGAN: Generative models from lossy measurements"

AmbientGAN: Generative models from lossy measurements This repository provides code to reproduce results from the paper AmbientGAN: Generative models

87 Oct 19, 2022

WormMovementSimulation - 3D Simulation of Worm Body Movement with Neurons attached to its body

Generate 3D Locomotion Data This module is intended to create 2D video trajector

1 Aug 09, 2022

This repository contains the implementation of the paper: "Towards Frequency-Based Explanation for Robust CNN"

RobustFreqCNN About This repository contains the implementation of the paper "Towards Frequency-Based Explanation for Robust CNN" arxiv. It primarly d

2 Jan 23, 2022

[CVPR 2020] Interpreting the Latent Space of GANs for Semantic Face Editing

InterFaceGAN - Interpreting the Latent Space of GANs for Semantic Face Editing Figure: High-quality facial attributes editing results with InterFaceGA

1.3k Dec 29, 2022

PyTorch code accompanying our paper on Maximum Entropy Generators for Energy-Based Models

Maximum Entropy Generators for Energy-Based Models All experiments have tensorboard visualizations for samples / density / train curves etc. To run th

135 Oct 27, 2022

Source code for Zalo AI 2021 submission

zalo_ltr_2021 Source code for Zalo AI 2021 submission Solution: Pipeline We use the pipepline in the picture below: Our pipeline is combination of BM2

128 Dec 27, 2022

Official Pytorch implementation of the paper: "Locally Shifted Attention With Early Global Integration"

Locally-Shifted-Attention-With-Early-Global-Integration Pretrained models You can download all the models from here. Training Imagenet python -m torch

14 Apr 15, 2022

This is an official implementation for "Exploiting Temporal Contexts with Strided Transformer for 3D Human Pose Estimation".

Exploiting Temporal Contexts with Strided Transformer for 3D Human Pose Estimation This repo is the official implementation of Exploiting Temporal Con

241 Jan 07, 2023

Implementation of Hourglass Transformer, in Pytorch, from Google and OpenAI

Related tags

Overview

Hourglass Transformer - Pytorch (wip)

Citations

You might also like...

A large dataset of 100k Google Satellite and matching Map images, resembling pix2pix's Google Maps dataset.

Red Team tool for exfiltrating files from a target's Google Drive that you have access to, via Google's API.

Google-drive-to-sqlite - Create a SQLite database containing metadata from Google Drive

BasicRL: easy and fundamental codes for deep reinforcement learning。It is an improvement on rainbow-is-all-you-need and OpenAI Spinning Up.

This is an official implementation for "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" on Object Detection and Instance Segmentation.

Third party Pytorch implement of Image Processing Transformer (Pre-Trained Image Processing Transformer arXiv:2012.00364v2)

Transformer - Transformer in PyTorch

Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network)

Plug-n-Play Reinforcement Learning in Python with OpenAI Gym and JAX

Comments

No option for rotary positional embeddings

Issue with an example

Releases(0.0.6)

0.0.6(Nov 9, 2021)

0.0.5(Nov 9, 2021)

0.0.4(Nov 9, 2021)

0.0.3(Nov 9, 2021)

0.0.2(Nov 9, 2021)

0.0.1(Nov 9, 2021)

Owner

Phil Wang

Code to reproduce results from the paper "AmbientGAN: Generative models from lossy measurements"

WormMovementSimulation - 3D Simulation of Worm Body Movement with Neurons attached to its body

This repository contains the implementation of the paper: "Towards Frequency-Based Explanation for Robust CNN"

[CVPR 2020] Interpreting the Latent Space of GANs for Semantic Face Editing

PyTorch code accompanying our paper on Maximum Entropy Generators for Energy-Based Models

Source code for Zalo AI 2021 submission

Official Pytorch implementation of the paper: "Locally Shifted Attention With Early Global Integration"

This is an official implementation for "Exploiting Temporal Contexts with Strided Transformer for 3D Human Pose Estimation".

Reporting and Visualization for Hazardous Events

Simple Linear 2nd ODE Solver GUI - A 2nd constant coefficient linear ODE solver with simple GUI using euler's method

This thesis is mainly concerned with state-space methods for a class of deep Gaussian process (DGP) regression problems

A curated list of awesome Machine Learning frameworks, libraries and software.

Official PyTorch implementation of Data-free Knowledge Distillation for Object Detection, WACV 2021.

code for paper -- "Seamless Satellite-image Synthesis"

Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition - NeurIPS2021

Dynamic wallpaper generator.

Code and data for "TURL: Table Understanding through Representation Learning"

LightningFSL: Pytorch-Lightning implementations of Few-Shot Learning models.

Code For TDEER: An Efficient Translating Decoding Schema for Joint Extraction of Entities and Relations (EMNLP2021)

Pytorch Performace Tuning, WandB, AMP, Multi-GPU, TensorRT, Triton