NLP-Models-Tensorflow, Gathers machine learning and tensorflow deep learning models for NLP problems, code simplify inside Jupyter Notebooks 100%.
Table of contents
- Abstractive Summarization
- Chatbot
- Dependency Parser
- Entity Tagging
- Extractive Summarization
- Generator
- Language Detection
- Neural Machine Translation
- OCR
- POS Tagging
- Question-Answers
- Sentence pairs
- Speech-to-Text
- Spelling correction
- SQUAD Question-Answers
- Stemming
- Text Augmentation
- Text Classification
- Text Similarity
- Text-to-Speech
- Topic Generator
- Topic Modeling
- Unsupervised Extractive Summarization
- Vectorizer
- Old-to-Young Vocoder
- Visualization
- Attention
Objective
Original implementations are quite complex and not really beginner friendly. So I tried to simplify most of it. Also, there are tons of not-yet release papers implementation. So feel free to use it for your own research!
I will attached github repositories for models that I not implemented from scratch, basically I copy, paste and fix those code for deprecated issues.
Tensorflow version
Tensorflow version 1.13 and above only, not included 2.X version. 1.13 < Tensorflow < 2.0
pip install -r requirements.txt
Contents
Abstractive Summarization
Trained on India news.
Accuracy based on 10 epochs only, calculated using word positions.
Complete list (12 notebooks)
- LSTM Seq2Seq using topic modelling, test accuracy 13.22%
- LSTM Seq2Seq + Luong Attention using topic modelling, test accuracy 12.39%
- LSTM Seq2Seq + Beam Decoder using topic modelling, test accuracy 10.67%
- LSTM Bidirectional + Luong Attention + Beam Decoder using topic modelling, test accuracy 8.29%
- Pointer-Generator + Bahdanau, https://github.com/xueyouluo/my_seq2seq, test accuracy 15.51%
- Copynet, test accuracy 11.15%
- Pointer-Generator + Luong, https://github.com/xueyouluo/my_seq2seq, test accuracy 16.51%
- Dilated Seq2Seq, test accuracy 10.88%
- Dilated Seq2Seq + Self Attention, test accuracy 11.54%
- BERT + Dilated CNN Seq2seq, test accuracy 13.5%
- self-attention + Pointer-Generator, test accuracy 4.34%
- Dilated-CNN Seq2seq + Pointer-Generator, test accuracy 5.57%
Chatbot
Trained on Cornell Movie Dialog corpus, accuracy table in chatbot.
Complete list (54 notebooks)
- Basic cell Seq2Seq-manual
- LSTM Seq2Seq-manual
- GRU Seq2Seq-manual
- Basic cell Seq2Seq-API Greedy
- LSTM Seq2Seq-API Greedy
- GRU Seq2Seq-API Greedy
- Basic cell Bidirectional Seq2Seq-manual
- LSTM Bidirectional Seq2Seq-manual
- GRU Bidirectional Seq2Seq-manual
- Basic cell Bidirectional Seq2Seq-API Greedy
- LSTM Bidirectional Seq2Seq-API Greedy
- GRU Bidirectional Seq2Seq-API Greedy
- Basic cell Seq2Seq-manual + Luong Attention
- LSTM Seq2Seq-manual + Luong Attention
- GRU Seq2Seq-manual + Luong Attention
- Basic cell Seq2Seq-manual + Bahdanau Attention
- LSTM Seq2Seq-manual + Bahdanau Attention
- GRU Seq2Seq-manual + Bahdanau Attention
- LSTM Bidirectional Seq2Seq-manual + Luong Attention
- GRU Bidirectional Seq2Seq-manual + Luong Attention
- LSTM Bidirectional Seq2Seq-manual + Bahdanau Attention
- GRU Bidirectional Seq2Seq-manual + Bahdanau Attention
- LSTM Bidirectional Seq2Seq-manual + backward Bahdanau + forward Luong
- GRU Bidirectional Seq2Seq-manual + backward Bahdanau + forward Luong
- LSTM Seq2Seq-API Greedy + Luong Attention
- GRU Seq2Seq-API Greedy + Luong Attention
- LSTM Seq2Seq-API Greedy + Bahdanau Attention
- GRU Seq2Seq-API Greedy + Bahdanau Attention
- LSTM Seq2Seq-API Beam Decoder
- GRU Seq2Seq-API Beam Decoder
- LSTM Bidirectional Seq2Seq-API + Luong Attention + Beam Decoder
- GRU Bidirectional Seq2Seq-API + Luong Attention + Beam Decoder
- LSTM Bidirectional Seq2Seq-API + backward Bahdanau + forward Luong + Stack Bahdanau Luong Attention + Beam Decoder
- GRU Bidirectional Seq2Seq-API + backward Bahdanau + forward Luong + Stack Bahdanau Luong Attention + Beam Decoder
- Bytenet
- LSTM Seq2Seq + tf.estimator
- Capsule layers + LSTM Seq2Seq-API Greedy
- Capsule layers + LSTM Seq2Seq-API + Luong Attention + Beam Decoder
- LSTM Bidirectional Seq2Seq-API + backward Bahdanau + forward Luong + Stack Bahdanau Luong Attention + Beam Decoder + Dropout + L2
- DNC Seq2Seq
- LSTM Bidirectional Seq2Seq-API + Luong Monotic Attention + Beam Decoder
- LSTM Bidirectional Seq2Seq-API + Bahdanau Monotic Attention + Beam Decoder
- End-to-End Memory Network + Basic cell
- End-to-End Memory Network + LSTM cell
- Attention is all you need
- Transformer-XL
- Attention is all you need + Beam Search
- Transformer-XL + LSTM
- GPT-2 + LSTM
- CNN Seq2seq
- Conv-Encoder + LSTM
- Tacotron + Greedy decoder
- Tacotron + Beam decoder
- Google NMT
Dependency-Parser
Trained on CONLL English Dependency. Train set to train, dev and test sets to test.
Stackpointer and Biaffine-attention originally from https://github.com/XuezheMax/NeuroNLP2 written in Pytorch.
Accuracy based on arc, types and root accuracies after 15 epochs only.
Complete list (8 notebooks)
- Bidirectional RNN + CRF + Biaffine, arc accuracy 70.48%, types accuracy 65.18%, root accuracy 66.4%
- Bidirectional RNN + Bahdanau + CRF + Biaffine, arc accuracy 70.82%, types accuracy 65.33%, root accuracy 66.77%
- Bidirectional RNN + Luong + CRF + Biaffine, arc accuracy 71.22%, types accuracy 65.73%, root accuracy 67.23%
- BERT Base + CRF + Biaffine, arc accuracy 64.30%, types accuracy 62.89%, root accuracy 74.19%
- Bidirectional RNN + Biaffine Attention + Cross Entropy, arc accuracy 72.42%, types accuracy 63.53%, root accuracy 68.51%
- BERT Base + Biaffine Attention + Cross Entropy, arc accuracy 72.85%, types accuracy 67.11%, root accuracy 73.93%
- Bidirectional RNN + Stackpointer, arc accuracy 61.88%, types accuracy 48.20%, root accuracy 89.39%
- XLNET Base + Biaffine Attention + Cross Entropy, arc accuracy 74.41%, types accuracy 71.37%, root accuracy 73.17%
Entity-Tagging
Trained on CONLL NER.
Complete list (9 notebooks)
- Bidirectional RNN + CRF, test accuracy 96%
- Bidirectional RNN + Luong Attention + CRF, test accuracy 93%
- Bidirectional RNN + Bahdanau Attention + CRF, test accuracy 95%
- Char Ngrams + Bidirectional RNN + Bahdanau Attention + CRF, test accuracy 96%
- Char Ngrams + Bidirectional RNN + Bahdanau Attention + CRF, test accuracy 96%
- Char Ngrams + Residual Network + Bahdanau Attention + CRF, test accuracy 69%
- Char Ngrams + Attention is you all Need + CRF, test accuracy 90%
- BERT, test accuracy 99%
- XLNET-Base, test accuracy 99%
Extractive Summarization
Trained on CNN News dataset.
Accuracy based on ROUGE-2.
Complete list (4 notebooks)
- LSTM RNN, test accuracy 16.13%
- Dilated-CNN, test accuracy 15.54%
- Multihead Attention, test accuracy 26.33%
- BERT-Base
Generator
Trained on Shakespeare dataset.
Complete list (15 notebooks)
- Character-wise RNN + LSTM
- Character-wise RNN + Beam search
- Character-wise RNN + LSTM + Embedding
- Word-wise RNN + LSTM
- Word-wise RNN + LSTM + Embedding
- Character-wise + Seq2Seq + GRU
- Word-wise + Seq2Seq + GRU
- Character-wise RNN + LSTM + Bahdanau Attention
- Character-wise RNN + LSTM + Luong Attention
- Word-wise + Seq2Seq + GRU + Beam
- Character-wise + Seq2Seq + GRU + Bahdanau Attention
- Word-wise + Seq2Seq + GRU + Bahdanau Attention
- Character-wise Dilated CNN + Beam search
- Transformer + Beam search
- Transformer XL + Beam search
Language-detection
Trained on Tatoeba dataset.
Complete list (1 notebooks)
- Fast-text Char N-Grams
Neural Machine Translation
Trained on English-French, accuracy table in neural-machine-translation.
Complete list (53 notebooks)
1.basic-seq2seq 2.lstm-seq2seq 3.gru-seq2seq 4.basic-seq2seq-contrib-greedy 5.lstm-seq2seq-contrib-greedy 6.gru-seq2seq-contrib-greedy 7.basic-birnn-seq2seq 8.lstm-birnn-seq2seq 9.gru-birnn-seq2seq 10.basic-birnn-seq2seq-contrib-greedy 11.lstm-birnn-seq2seq-contrib-greedy 12.gru-birnn-seq2seq-contrib-greedy 13.basic-seq2seq-luong 14.lstm-seq2seq-luong 15.gru-seq2seq-luong 16.basic-seq2seq-bahdanau 17.lstm-seq2seq-bahdanau 18.gru-seq2seq-bahdanau 19.basic-birnn-seq2seq-bahdanau 20.lstm-birnn-seq2seq-bahdanau 21.gru-birnn-seq2seq-bahdanau 22.basic-birnn-seq2seq-luong 23.lstm-birnn-seq2seq-luong 24.gru-birnn-seq2seq-luong 25.lstm-seq2seq-contrib-greedy-luong 26.gru-seq2seq-contrib-greedy-luong 27.lstm-seq2seq-contrib-greedy-bahdanau 28.gru-seq2seq-contrib-greedy-bahdanau 29.lstm-seq2seq-contrib-beam-luong 30.gru-seq2seq-contrib-beam-luong 31.lstm-seq2seq-contrib-beam-bahdanau 32.gru-seq2seq-contrib-beam-bahdanau 33.lstm-birnn-seq2seq-contrib-beam-bahdanau 34.lstm-birnn-seq2seq-contrib-beam-luong 35.gru-birnn-seq2seq-contrib-beam-bahdanau 36.gru-birnn-seq2seq-contrib-beam-luong 37.lstm-birnn-seq2seq-contrib-beam-luongmonotonic 38.gru-birnn-seq2seq-contrib-beam-luongmonotic 39.lstm-birnn-seq2seq-contrib-beam-bahdanaumonotonic 40.gru-birnn-seq2seq-contrib-beam-bahdanaumonotic 41.residual-lstm-seq2seq-greedy-luong 42.residual-gru-seq2seq-greedy-luong 43.residual-lstm-seq2seq-greedy-bahdanau 44.residual-gru-seq2seq-greedy-bahdanau 45.memory-network-lstm-decoder-greedy 46.google-nmt 47.transformer-encoder-transformer-decoder 48.transformer-encoder-lstm-decoder-greedy 49.bertmultilanguage-encoder-bertmultilanguage-decoder 50.bertmultilanguage-encoder-lstm-decoder 51.bertmultilanguage-encoder-transformer-decoder 52.bertenglish-encoder-transformer-decoder 53.transformer-t2t-2gpu
OCR (optical character recognition)
Complete list (2 notebooks)
- CNN + LSTM RNN, test accuracy 100%
- Im2Latex, test accuracy 100%
POS-Tagging
Trained on CONLL POS.
Complete list (8 notebooks)
- Bidirectional RNN + CRF, test accuracy 92%
- Bidirectional RNN + Luong Attention + CRF, test accuracy 91%
- Bidirectional RNN + Bahdanau Attention + CRF, test accuracy 91%
- Char Ngrams + Bidirectional RNN + Bahdanau Attention + CRF, test accuracy 91%
- Char Ngrams + Bidirectional RNN + Bahdanau Attention + CRF, test accuracy 91%
- Char Ngrams + Residual Network + Bahdanau Attention + CRF, test accuracy 3%
- Char Ngrams + Attention is you all Need + CRF, test accuracy 89%
- BERT, test accuracy 99%
Question-Answers
Trained on bAbI Dataset.
Complete list (4 notebooks)
- End-to-End Memory Network + Basic cell
- End-to-End Memory Network + GRU cell
- End-to-End Memory Network + LSTM cell
- Dynamic Memory
Sentence-pair
Trained on Cornell Movie--Dialogs Corpus
Complete list (1 notebooks)
- BERT
Speech to Text
Trained on Toronto speech dataset.
Complete list (11 notebooks)
- Tacotron, https://github.com/Kyubyong/tacotron_asr, test accuracy 77.09%
- BiRNN LSTM, test accuracy 84.66%
- BiRNN Seq2Seq + Luong Attention + Cross Entropy, test accuracy 87.86%
- BiRNN Seq2Seq + Bahdanau Attention + Cross Entropy, test accuracy 89.28%
- BiRNN Seq2Seq + Bahdanau Attention + CTC, test accuracy 86.35%
- BiRNN Seq2Seq + Luong Attention + CTC, test accuracy 80.30%
- CNN RNN + Bahdanau Attention, test accuracy 80.23%
- Dilated CNN RNN, test accuracy 31.60%
- Wavenet, test accuracy 75.11%
- Deep Speech 2, test accuracy 81.40%
- Wav2Vec Transfer learning BiRNN LSTM, test accuracy 83.24%
Spelling correction
Complete list (4 notebooks)
- BERT-Base
- XLNET-Base
- BERT-Base Fast
- BERT-Base accurate
SQUAD Question-Answers
Trained on SQUAD Dataset.
Complete list (1 notebooks)
- BERT,
{"exact_match": 77.57805108798486, "f1": 86.18327335287402}
Stemming
Trained on English Lemmatization.
Complete list (6 notebooks)
- LSTM + Seq2Seq + Beam
- GRU + Seq2Seq + Beam
- LSTM + BiRNN + Seq2Seq + Beam
- GRU + BiRNN + Seq2Seq + Beam
- DNC + Seq2Seq + Greedy
- BiRNN + Bahdanau + Copynet
Text Augmentation
Complete list (8 notebooks)
- Pretrained Glove
- GRU VAE-seq2seq-beam TF-probability
- LSTM VAE-seq2seq-beam TF-probability
- GRU VAE-seq2seq-beam + Bahdanau Attention TF-probability
- VAE + Deterministic Bahdanau Attention, https://github.com/HareeshBahuleyan/tf-var-attention
- VAE + VAE Bahdanau Attention, https://github.com/HareeshBahuleyan/tf-var-attention
- BERT-Base + Nucleus Sampling
- XLNET-Base + Nucleus Sampling
Text classification
Trained on English sentiment dataset, accuracy table in text-classification.
Complete list (79 notebooks)
- Basic cell RNN
- Basic cell RNN + Hinge
- Basic cell RNN + Huber
- Basic cell Bidirectional RNN
- Basic cell Bidirectional RNN + Hinge
- Basic cell Bidirectional RNN + Huber
- LSTM cell RNN
- LSTM cell RNN + Hinge
- LSTM cell RNN + Huber
- LSTM cell Bidirectional RNN
- LSTM cell Bidirectional RNN + Huber
- LSTM cell RNN + Dropout + L2
- GRU cell RNN
- GRU cell RNN + Hinge
- GRU cell RNN + Huber
- GRU cell Bidirectional RNN
- GRU cell Bidirectional RNN + Hinge
- GRU cell Bidirectional RNN + Huber
- LSTM RNN + Conv2D
- K-max Conv1d
- LSTM RNN + Conv1D + Highway
- LSTM RNN + Basic Attention
- LSTM Dilated RNN
- Layer-Norm LSTM cell RNN
- Only Attention Neural Network
- Multihead-Attention Neural Network
- Neural Turing Machine
- LSTM Seq2Seq
- LSTM Seq2Seq + Luong Attention
- LSTM Seq2Seq + Bahdanau Attention
- LSTM Seq2Seq + Beam Decoder
- LSTM Bidirectional Seq2Seq
- Pointer Net
- LSTM cell RNN + Bahdanau Attention
- LSTM cell RNN + Luong Attention
- LSTM cell RNN + Stack Bahdanau Luong Attention
- LSTM cell Bidirectional RNN + backward Bahdanau + forward Luong
- Bytenet
- Fast-slow LSTM
- Siamese Network
- LSTM Seq2Seq + tf.estimator
- Capsule layers + RNN LSTM
- Capsule layers + LSTM Seq2Seq
- Capsule layers + LSTM Bidirectional Seq2Seq
- Nested LSTM
- LSTM Seq2Seq + Highway
- Triplet loss + LSTM
- DNC (Differentiable Neural Computer)
- ConvLSTM
- Temporal Convd Net
- Batch-all Triplet-loss + LSTM
- Fast-text
- Gated Convolution Network
- Simple Recurrent Unit
- LSTM Hierarchical Attention Network
- Bidirectional Transformers
- Dynamic Memory Network
- Entity Network
- End-to-End Memory Network
- BOW-Chars Deep sparse Network
- Residual Network using Atrous CNN
- Residual Network using Atrous CNN + Bahdanau Attention
- Deep pyramid CNN
- Transformer-XL
- Transfer learning GPT-2 345M
- Quasi-RNN
- Tacotron
- Slice GRU
- Slice GRU + Bahdanau
- Wavenet
- Transfer learning BERT Base
- Transfer learning XL-net Large
- LSTM BiRNN global Max and average pooling
- Transfer learning BERT Base drop 6 layers
- Transfer learning BERT Large drop 12 layers
- Transfer learning XL-net Base
- Transfer learning ALBERT
- Transfer learning ELECTRA Base
- Transfer learning ELECTRA Large
Text Similarity
Trained on MNLI.
Complete list (10 notebooks)
- BiRNN + Contrastive loss, test accuracy 73.032%
- BiRNN + Cross entropy, test accuracy 74.265%
- BiRNN + Circle loss, test accuracy 75.857%
- BiRNN + Proxy loss, test accuracy 48.37%
- BERT Base + Cross entropy, test accuracy 91.123%
- BERT Base + Circle loss, test accuracy 89.903%
- ELECTRA Base + Cross entropy, test accuracy 96.317%
- ELECTRA Base + Circle loss, test accuracy 95.603%
- XLNET Base + Cross entropy, test accuracy 93.998%
- XLNET Base + Circle loss, test accuracy 94.033%
Text to Speech
Trained on Toronto speech dataset.
Complete list (8 notebooks)
- Tacotron, https://github.com/Kyubyong/tacotron
- CNN Seq2seq + Dilated CNN vocoder
- Seq2Seq + Bahdanau Attention
- Seq2Seq + Luong Attention
- Dilated CNN + Monothonic Attention + Dilated CNN vocoder
- Dilated CNN + Self Attention + Dilated CNN vocoder
- Deep CNN + Monothonic Attention + Dilated CNN vocoder
- Deep CNN + Self Attention + Dilated CNN vocoder
Topic Generator
Trained on Malaysia news.
Complete list (4 notebooks)
- TAT-LSTM
- TAV-LSTM
- MTA-LSTM
- Dilated CNN Seq2seq
Topic Modeling
Extracted from English sentiment dataset.
Complete list (3 notebooks)
- LDA2Vec
- BERT Attention
- XLNET Attention
Unsupervised Extractive Summarization
Trained on random books.
Complete list (3 notebooks)
- Skip-thought Vector
- Residual Network using Atrous CNN
- Residual Network using Atrous CNN + Bahdanau Attention
Vectorizer
Trained on English sentiment dataset.
Complete list (11 notebooks)
- Word Vector using CBOW sample softmax
- Word Vector using CBOW noise contrastive estimation
- Word Vector using skipgram sample softmax
- Word Vector using skipgram noise contrastive estimation
- Supervised Embedded
- Triplet-loss + LSTM
- LSTM Auto-Encoder
- Batch-All Triplet-loss LSTM
- Fast-text
- ELMO (biLM)
- Triplet-loss + BERT
Visualization
Complete list (4 notebooks)
- Attention heatmap on Bahdanau Attention
- Attention heatmap on Luong Attention
- BERT attention, https://github.com/hsm207/bert_attn_viz
- XLNET attention
Old-to-Young Vocoder
Trained on Toronto speech dataset.
Complete list (1 notebooks)
- Dilated CNN
Attention
Complete list (8 notebooks)
- Bahdanau
- Luong
- Hierarchical
- Additive
- Soft
- Attention-over-Attention
- Bahdanau API
- Luong API
Not-deep-learning
- Markov chatbot
- Decomposition summarization (3 notebooks)
21 Sep 25, 2022
606 Dec 28, 2022
452 Dec 30, 2022
4 Dec 09, 2021
44 Dec 30, 2022
5.8k Jan 04, 2023
1k Dec 26, 2022
172 Dec 13, 2022
67 Nov 14, 2022
3 Jan 14, 2022
1 Feb 05, 2022
203 Dec 20, 2022
94 Dec 27, 2022
2.9k Jan 02, 2023
3 Aug 10, 2022
276 Dec 31, 2022
11.9k Jan 08, 2023
13 Dec 13, 2022
244 Dec 30, 2022
0 Feb 13, 2022