Implementation of Hourglass Transformer, in Pytorch, from Google and OpenAI

Last update: Dec 25, 2022

Overview

Hourglass Transformer - Pytorch (wip)

Implementation of Hourglass Transformer, in Pytorch. It will also contain some of my own ideas about how to make it work better.

Citations

@misc{nawrot2021hierarchical,
    title   = {Hierarchical Transformers Are More Efficient Language Models}, 
    author  = {Piotr Nawrot and Szymon Tworkowski and Michał Tyrolski and Łukasz Kaiser and Yuhuai Wu and Christian Szegedy and Henryk Michalewski},
    year    = {2021},
    eprint  = {2110.13711},
    archivePrefix = {arXiv},
    primaryClass = {cs.LG}
}

You might also like...

A large dataset of 100k Google Satellite and matching Map images, resembling pix2pix's Google Maps dataset.

Larger Google Sat2Map dataset This dataset extends the aerial ⟷ Maps dataset used in pix2pix (Isola et al., CVPR17). The provide script download_sat2m

34 Dec 28, 2022

Red Team tool for exfiltrating files from a target's Google Drive that you have access to, via Google's API.

GD-Thief Red Team tool for exfiltrating files from a target's Google Drive that you(the attacker) has access to, via the Google Drive API. This includ

39 Dec 27, 2022

Google-drive-to-sqlite - Create a SQLite database containing metadata from Google Drive

google-drive-to-sqlite Create a SQLite database containing metadata from Google

140 Dec 4, 2022

BasicRL: easy and fundamental codes for deep reinforcement learning。It is an improvement on rainbow-is-all-you-need and OpenAI Spinning Up.

BasicRL: easy and fundamental codes for deep reinforcement learning BasicRL is an improvement on rainbow-is-all-you-need and OpenAI Spinning Up. It is

12 Apr 28, 2022

This is an official implementation for "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" on Object Detection and Instance Segmentation.

Swin Transformer for Object Detection This repo contains the supported code and configuration files to reproduce object detection results of Swin Tran

1.4k Dec 30, 2022

Third party Pytorch implement of Image Processing Transformer (Pre-Trained Image Processing Transformer arXiv:2012.00364v2)

ImageProcessingTransformer Third party Pytorch implement of Image Processing Transformer (Pre-Trained Image Processing Transformer arXiv:2012.00364v2)

61 Jan 1, 2023

Transformer - Transformer in PyTorch

Transformer 完成进度 Embeddings and PositionalEncoding with example. MultiHeadAttent

1 Jan 6, 2022

Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network)

Deep Daze mist over green hills shattered plates on the grass cosmic love and attention a time traveler in the crowd life during the plague meditative

4.4k Jan 3, 2023

Plug-n-Play Reinforcement Learning in Python with OpenAI Gym and JAX

coax is built on top of JAX, but it doesn't have an explicit dependence on the jax python package. The reason is that your version of jaxlib will depend on your CUDA version.

128 Dec 27, 2022

Comments

No option for rotary positional embeddings

For the ImageNet64 generation experiments, the original authors used rotary positional embeddings. I think it would be worth adding this feature to the repository.

I can prepare a pull request that solves this~

opened by vvvm23 4
Issue with an example

@lucidrains Just Awesome to see this super fast launch !

The third example under "For Images...." throws the following error .,

Then , when I gave the inputs with num_tokens = 20000, max_seq_len = 1024, I got the following runtime error

opened by nsankar 1

Releases(0.0.6)

0.0.6(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.5(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.4(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.3(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.2(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.1(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)

Owner

Phil Wang

Working with Attention. It's all we need

GitHub Repository

Official Repo of my work for SREC Nandyal Machine Learning Bootcamp

About the Bootcamp A 3-day Machine Learning Bootcamp organised by Department of Electronics and Communication Engineering, Santhiram Engineering Colle

1 Nov 29, 2021

Implementation of ICCV2021(Oral) paper - VMNet: Voxel-Mesh Network for Geodesic-aware 3D Semantic Segmentation

VMNet: Voxel-Mesh Network for Geodesic-Aware 3D Semantic Segmentation Created by Zeyu HU Introduction This work is based on our paper VMNet: Voxel-Mes

82 Dec 27, 2022

Quantify the difference between two arbitrary curves in space

similaritymeasures Quantify the difference between two arbitrary curves Curves in this case are: discretized by inidviudal data points ordered from a

175 Jan 08, 2023

Surrogate-Assisted Genetic Algorithm for Wrapper Feature Selection

SAGA Surrogate-Assisted Genetic Algorithm for Wrapper Feature Selection Please refer to the Jupyter notebook (Example.ipynb) for an example of using t

9 Dec 28, 2022

Multi-label classification of retinal disorders

Multi-label classification of retinal disorders This is a deep learning course project. The goal is to develop a solution, using computer vision techn

1 Jan 29, 2022

Complex Answer Generation For Conversational Search Systems.

Complex Answer Generation For Conversational Search Systems. Code for Does Structure Matter? Leveraging Data-to-Text Generation for Answering Complex

0 Dec 06, 2021

Jupyter notebooks showing best practices for using cx_Oracle, the Python DB API for Oracle Database

Python cx_Oracle Notebooks, 2022 The repository contains Jupyter notebooks showing best practices for using cx_Oracle, the Python DB API for Oracle Da

13 Dec 15, 2022

Open-sourcing the Slates Dataset for recommender systems research

FINN.no Recommender Systems Slate Dataset This repository accompany the paper "Dynamic Slate Recommendation with Gated Recurrent Units and Thompson Sa

48 Nov 28, 2022

Machine learning evaluation metrics, implemented in Python, R, Haskell, and MATLAB / Octave

Note: the current releases of this toolbox are a beta release, to test working with Haskell's, Python's, and R's code repositories. Metrics provides i

1.6k Dec 26, 2022

using STGCN to achieve egg classification task

EEG Classification The task requires us to classify electroencephalography(EEG) into six categories, including human body, human face, animal body,

4 Jun 13, 2022

An Abstract Cyber Security Simulation and Markov Game for OpenAI Gym

gym-idsgame An Abstract Cyber Security Simulation and Markov Game for OpenAI Gym gym-idsgame is a reinforcement learning environment for simulating at

29 Dec 03, 2022

KUIELAB-MDX-Net got the 2nd place on the Leaderboard A and the 3rd place on the Leaderboard B in the MDX-Challenge ISMIR 2021

74 Dec 28, 2022

Implementation of Hourglass Transformer, in Pytorch, from Google and OpenAI

Related tags

Overview

Hourglass Transformer - Pytorch (wip)

Citations

You might also like...

A large dataset of 100k Google Satellite and matching Map images, resembling pix2pix's Google Maps dataset.

Red Team tool for exfiltrating files from a target's Google Drive that you have access to, via Google's API.

Google-drive-to-sqlite - Create a SQLite database containing metadata from Google Drive

BasicRL: easy and fundamental codes for deep reinforcement learning。It is an improvement on rainbow-is-all-you-need and OpenAI Spinning Up.

This is an official implementation for "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" on Object Detection and Instance Segmentation.

Third party Pytorch implement of Image Processing Transformer (Pre-Trained Image Processing Transformer arXiv:2012.00364v2)

Transformer - Transformer in PyTorch

Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network)

Plug-n-Play Reinforcement Learning in Python with OpenAI Gym and JAX

Comments

No option for rotary positional embeddings

Issue with an example

Releases(0.0.6)

0.0.6(Nov 9, 2021)

0.0.5(Nov 9, 2021)

0.0.4(Nov 9, 2021)

0.0.3(Nov 9, 2021)

0.0.2(Nov 9, 2021)

0.0.1(Nov 9, 2021)

Owner

Phil Wang

Official Repo of my work for SREC Nandyal Machine Learning Bootcamp

Implementation of ICCV2021(Oral) paper - VMNet: Voxel-Mesh Network for Geodesic-aware 3D Semantic Segmentation

Quantify the difference between two arbitrary curves in space

Surrogate-Assisted Genetic Algorithm for Wrapper Feature Selection

Multi-label classification of retinal disorders

Complex Answer Generation For Conversational Search Systems.

Jupyter notebooks showing best practices for using cx_Oracle, the Python DB API for Oracle Database

Open-sourcing the Slates Dataset for recommender systems research

Machine learning evaluation metrics, implemented in Python, R, Haskell, and MATLAB / Octave

using STGCN to achieve egg classification task

An Abstract Cyber Security Simulation and Markov Game for OpenAI Gym

KUIELAB-MDX-Net got the 2nd place on the Leaderboard A and the 3rd place on the Leaderboard B in the MDX-Challenge ISMIR 2021

DPC: Unsupervised Deep Point Correspondence via Cross and Self Construction (3DV 2021)

A simple Rock-Paper-Scissors game using CV in python

MAterial del programa Misión TIC 2022

This project is the PyTorch implementation of our CVPR 2022 paper:

Simple Dynamic Batching Inference

blind SQLIpy sebuah alat injeksi sql yang menggunakan waktu sql untuk mendapatkan sebuah server database.

Learning Efficient Online 3D Bin Packing on Packing Configuration Trees

🕵 Artificial Intelligence for social control of public administration