Linear Transformers Are Secretly Fast Weight Programmers
This repository contains the code accompanying the paper Linear Transformers Are Secretly Fast Weight Programmers which is published at ICML'21. It also contains the logs of all synthetic experiments.
Synthetic Experiments
Requirements
$ cat req.txt
jupyter==1.0.0
pandas==1.0.1
seaborn==0.10.0
torch==1.6.0
matplotlib==3.1.3
numpy==1.17.2
pip3 install -r req.txt
Rerun Experiments
Logs are provided in the synthetic/logs folder. The files in that folder are a result of running the following commands:
Setting 1 (capacity):
python3 main.py --begin=20 --end=600 --step=20 --attn_name=softmax --update_rule=sum
python3 main.py --begin=20 --end=600 --step=20 --attn_name=linear --update_rule=sum
python3 main.py --begin=20 --end=600 --step=20 --attn_name=dpfp --attn_arg=1 --update_rule=sum
python3 main.py --begin=20 --end=600 --step=20 --attn_name=dpfp --attn_arg=2 --update_rule=sum
python3 main.py --begin=20 --end=600 --step=20 --attn_name=dpfp --attn_arg=3 --update_rule=sum
python3 main.py --begin=20 --end=600 --step=20 --attn_name=favor --attn_arg=64 --update_rule=sum
python3 main.py --begin=20 --end=600 --step=20 --attn_name=favor --attn_arg=128 --update_rule=sum
python3 main.py --begin=20 --end=600 --step=20 --attn_name=favor --attn_arg=512 --update_rule=sum
Setting 2 (update rule):
python3 main.py --begin=20 --end=200 --step=20 --attn_name=dpfp --attn_arg=1 --update_rule=sum --replace
python3 main.py --begin=20 --end=200 --step=20 --attn_name=dpfp --attn_arg=1 --update_rule=ours --replace
python3 main.py --begin=20 --end=200 --step=20 --attn_name=tanh --update_rule=fwm --replace
python3 main.py --begin=20 --end=200 --step=20 --attn_name=dpfp --attn_arg=1 --update_rule=fwm --replace
python3 main.py --begin=20 --end=200 --step=20 --attn_name=dpfp --attn_arg=2 --update_rule=ours --replace
python3 main.py --begin=20 --end=200 --step=20 --attn_name=linear --update_rule=ours --replace
python3 main.py --begin=20 --end=200 --step=20 --attn_name=favor --attn_arg=64 --update_rule=ours --replace
python3 main.py --begin=20 --end=200 --step=20 --attn_name=favor --attn_arg=128 --update_rule=ours --replace
Generate figures from the logs using the following notebooks:
synthetic/setting1_generate_figure.ipynb
synthetic/setting2_generate_figure.ipynb
Language Modelling & Machine Translation
The toolkit and scripts for language modeling experiments can be found at IDSIA/lmtool-fwms.
For machine translation experiments, we ported the different attention functions implemented in the language modeling toolkit to the multi-head attention implementation in FAIRSEQ.
Citation
@inproceedings{schlag2021linear,
title={Linear Transformers Are Secretly Fast Weight Programmers},
author={Imanol Schlag and Kazuki Irie and J\"urgen Schmidhuber},
booktitle={Proc. Int. Conf. on Machine Learning (ICML)},
address = {Virtual only},
month = jul,
year={2021}
}
1 Jan 11, 2022
2 Jan 28, 2022
92 Dec 25, 2022
114 Nov 04, 2022
129 Jan 06, 2023
15 Aug 25, 2022
11.9k Jan 08, 2023
6 Jun 27, 2022
919 Jan 03, 2023
14 Dec 07, 2022
14 Dec 12, 2022
3 Mar 02, 2022
612 Jan 04, 2023
1 Dec 17, 2021
2 Sep 27, 2022
279 Dec 09, 2022
5.3k Jan 07, 2023
2 Aug 29, 2022
11.4k Jan 01, 2023
287 Dec 14, 2022