GPT - gMLP
This repository will attempt to crack long context autoregressive language modeling (GPT) using variations of gMLPs. Specifically, it will contain a variant that does gMLP for local sliding windows. The hope is to be able to stretch a single GPU to be able to train context lengths of 4096 and above efficiently and well.
GPT is technically a misnomer now, since there will be no attention (transformer) at all contained in the architecture.
Install
$ pip install g-mlp-gpt
Usage
import torch
from g_mlp_gpt import gMLPGPT
model = gMLPGPT(
num_tokens = 20000,
dim = 512,
depth = 4,
seq_len = 1024,
window_size = (128, 256, 512, 1024) # window sizes for each depth
)
x = torch.randint(0, 20000, (1, 1000))
logits = model(x) # (1, 1000, 20000)
16k context length
import torch
from g_mlp_gpt import gMLPGPT
model = gMLPGPT(
num_tokens = 20000,
dim = 512,
seq_len = 16384,
depth = 8,
reversible = True,
window = (128, 128, 256, 512, 1024, 1024, 2048, 2048, 4096, 4096, 8192, 8192),
axial = (1, 1, 1, 1, 1, 1, 2, 2, 4, 4, 8, 8)
).cuda()
x = torch.randint(0, 20000, (1, 16384)).cuda()
logits = model(x) # (1, 16384, 20000)
Citations
@misc{liu2021pay,
title = {Pay Attention to MLPs},
author = {Hanxiao Liu and Zihang Dai and David R. So and Quoc V. Le},
year = {2021},
eprint = {2105.08050},
archivePrefix = {arXiv},
primaryClass = {cs.LG}
}
9 Nov 16, 2022
24 Oct 26, 2022
331 Jan 3, 2023
74 Dec 5, 2022
28 Feb 20, 2022
81 Nov 26, 2022
39 Nov 30, 2022
49 Dec 22, 2022
120 Dec 12, 2022
6 Dec 29, 2022
22 Aug 04, 2022
1 Feb 11, 2022
310 Dec 13, 2022
11 Oct 09, 2022
55 Jun 18, 2022
55 Dec 12, 2022
2 Dec 27, 2021
4 Feb 05, 2022
2 Oct 26, 2021
0 Dec 18, 2021
41 Dec 10, 2022
325 Jan 05, 2023
3 Dec 15, 2021
2 Nov 08, 2021
1.2k Dec 26, 2022
89 Aug 15, 2022
56 Dec 16, 2022
9 Nov 10, 2022