I've written a series of articles on the transformer architecture and language models on Medium.

This repository contains an implementation of the Transformer architecture presented in the paper Attention Is All You Need by Ashish Vaswani, et. al.

My goal is to write an implementation that is easy to understand and dig into nitty-gritty details where the devil is.

You can use any Python virtual environment like venv and conda.

For example, with venv:

python3 -m venv venv
source venv/bin/activate

pip install --upgrade pip
pip install -e.

To use Spacy's tokenizer, make sure to download required languages.

For example, English and Germany tokenizers can be downloaded as below:

python -m spacy download en_core_web_sm
python -m spacy download de_core_news_sm

This project uses text datasets from Torchtext.

from torchtext import datasets

The default configuration uses Multi30k dataset.

python train.py config_path

The default config path is config/train.small.yaml.

The train script creates tensorboard log files under runs, and saves config.yaml and model checkpoints there.

You can run tensorboard to see the training progress.

tensorboard --logdir=runs

It is possible to resume training from a checkpoint.

python train.py --checkpoint_path runs/20220108-164720-Multi30k-Transformer/checkpoint-010-2.3343.pt

python test.py checkpoint_path

Example,

python test.py runs/20220108-164720-Multi30k-Transformer/checkpoint-010-2.3343.pt

The test.py resumes with the checkpoint and config.yaml in the same directory.

You can specify translator configuration by:

python test.py checkpoint_path [--config_path config_path]

The default is config/translator.beam.yaml.

There are some unit tests in the tests folder.

pytest tests

• The Annotated Transformer by Harvard NLP
• How to code The Transformer in Pytorch by Samuel Lynn-Evans
• The Illustrated Transformer by Jay Alammar
• Transformer Architecture: The Positional Encoding by Amirhossein Kazemnejad
• Transformers without Tears: Improving the Normalization of Self-Attention by Toan Q. Nguyen & Julian Salazar
• Tensor2Tensor by TensorFlow
• PyTorch Transformer by PyTorch
• Language Modeling with nn.Transformer and Torchtext by PyTorch
• My Medium Articles