Linear Transformers Are Secretly Fast Weight Programmers
This repository contains the code accompanying the paper Linear Transformers Are Secretly Fast Weight Programmers which is published at ICML'21. It also contains the logs of all synthetic experiments.
Synthetic Experiments
Requirements
$ cat req.txt
jupyter==1.0.0
pandas==1.0.1
seaborn==0.10.0
torch==1.6.0
matplotlib==3.1.3
numpy==1.17.2
pip3 install -r req.txt
Rerun Experiments
Logs are provided in the synthetic/logs folder. The files in that folder are a result of running the following commands:
Setting 1 (capacity):
python3 main.py --begin=20 --end=600 --step=20 --attn_name=softmax --update_rule=sum
python3 main.py --begin=20 --end=600 --step=20 --attn_name=linear --update_rule=sum
python3 main.py --begin=20 --end=600 --step=20 --attn_name=dpfp --attn_arg=1 --update_rule=sum
python3 main.py --begin=20 --end=600 --step=20 --attn_name=dpfp --attn_arg=2 --update_rule=sum
python3 main.py --begin=20 --end=600 --step=20 --attn_name=dpfp --attn_arg=3 --update_rule=sum
python3 main.py --begin=20 --end=600 --step=20 --attn_name=favor --attn_arg=64 --update_rule=sum
python3 main.py --begin=20 --end=600 --step=20 --attn_name=favor --attn_arg=128 --update_rule=sum
python3 main.py --begin=20 --end=600 --step=20 --attn_name=favor --attn_arg=512 --update_rule=sum
Setting 2 (update rule):
python3 main.py --begin=20 --end=200 --step=20 --attn_name=dpfp --attn_arg=1 --update_rule=sum --replace
python3 main.py --begin=20 --end=200 --step=20 --attn_name=dpfp --attn_arg=1 --update_rule=ours --replace
python3 main.py --begin=20 --end=200 --step=20 --attn_name=tanh --update_rule=fwm --replace
python3 main.py --begin=20 --end=200 --step=20 --attn_name=dpfp --attn_arg=1 --update_rule=fwm --replace
python3 main.py --begin=20 --end=200 --step=20 --attn_name=dpfp --attn_arg=2 --update_rule=ours --replace
python3 main.py --begin=20 --end=200 --step=20 --attn_name=linear --update_rule=ours --replace
python3 main.py --begin=20 --end=200 --step=20 --attn_name=favor --attn_arg=64 --update_rule=ours --replace
python3 main.py --begin=20 --end=200 --step=20 --attn_name=favor --attn_arg=128 --update_rule=ours --replace
Generate figures from the logs using the following notebooks:
synthetic/setting1_generate_figure.ipynb
synthetic/setting2_generate_figure.ipynb
Language Modelling & Machine Translation
The toolkit and scripts for language modeling experiments can be found at IDSIA/lmtool-fwms.
For machine translation experiments, we ported the different attention functions implemented in the language modeling toolkit to the multi-head attention implementation in FAIRSEQ.
Citation
@inproceedings{schlag2021linear,
title={Linear Transformers Are Secretly Fast Weight Programmers},
author={Imanol Schlag and Kazuki Irie and J\"urgen Schmidhuber},
booktitle={Proc. Int. Conf. on Machine Learning (ICML)},
address = {Virtual only},
month = jul,
year={2021}
}
1 Apr 28, 2022
12 Apr 07, 2022
114 Dec 29, 2022
3 Apr 02, 2022
116 Nov 09, 2022
251 Jan 01, 2023
41 Dec 30, 2022
163 Dec 31, 2022
41 Dec 27, 2022
9 Oct 31, 2022
1.6k Jan 07, 2023
1.8k Dec 21, 2022
2 Apr 20, 2022
184 Dec 18, 2022
3 Dec 04, 2022
1 Feb 04, 2022
858 Dec 27, 2022
121 Dec 27, 2022
5.6k Jan 03, 2023
1 Aug 21, 2022