An IPython Notebook tutorial on deep learning for natural language processing, including structure prediction.

Overview

Table of Contents:

  1. Introduction to Torch's Tensor Library
  2. Computation Graphs and Automatic Differentiation
  3. Deep Learning Building Blocks: Affine maps, non-linearities, and objectives
  4. Optimization and Training
  5. Creating Network Components in Pytorch
  • Example: Logistic Regression Bag-of-Words text classifier
  1. Word Embeddings: Encoding Lexical Semantics
  • Example: N-Gram Language Modeling
  • Exercise: Continuous Bag-of-Words for learning word embeddings
  1. Sequence modeling and Long-Short Term Memory Networks
  • Example: An LSTM for Part-of-Speech Tagging
  • Exercise: Augmenting the LSTM tagger with character-level features
  1. Advanced: Dynamic Toolkits, Dynamic Programming, and the BiLSTM-CRF
  • Example: Bi-LSTM Conditional Random Field for named-entity recognition
  • Exercise: A new loss function for discriminative tagging

What is this tutorial?

I am writing this tutorial because, although there are plenty of other tutorials out there, they all seem to have one of three problems:

  • They have a lot of content on computer vision and conv nets, which is irrelevant for most NLP (although conv nets have been applied in cool ways to NLP problems).
  • Pytorch is brand new, and so many deep learning for NLP tutorials are in older frameworks, and usually not in dynamic frameworks like Pytorch, which have a totally different flavor.
  • The examples don't move beyond RNN language models and show the awesome stuff you can do when trying to do lingusitic structure prediction. I think this is a problem, because Pytorch's dynamic graphs make structure prediction one of its biggest strengths.

Specifically, I am writing this tutorial for a Natural Language Processing class at Georgia Tech, to ease into a problem set I wrote for the class on deep transition parsing. The problem set uses some advanced techniques. The intention of this tutorial is to cover the basics, so that students can focus on the more challenging aspects of the problem set. The aim is to start with the basics and move up to linguistic structure prediction, which I feel is almost completely absent in other Pytorch tutorials. The general deep learning basics have short expositions. Topics more NLP-specific received more in-depth discussions, although I have referred to other sources when I felt a full description would be reinventing the wheel and take up too much space.

Dependency Parsing Problem Set

As mentioned above, here is the problem set that goes through implementing a high-performing dependency parser in Pytorch. I wanted to add a link here since it might be useful, provided you ignore the things that were specific to the class. A few notes:

  • There is a lot of code, so the beginning of the problem set was mainly to get people familiar with the way my code represented the relevant data, and the interfaces you need to use. The rest of the problem set is actually implementing components for the parser. Since we hadn't done deep learning in the class before, I tried to provide an enormous amount of comments and hints when writing it.
  • There is a unit test for every deliverable, which you can run with nosetests.
  • Since we use this problem set in the class, please don't publically post solutions.
  • The same repo has some notes that include a section on shift-reduce dependency parsing, if you are looking for a written source to complement the problem set.
  • The link above might not work if it is taken down at the start of a new semester.

References:

  • I learned a lot about deep structure prediction at EMNLP 2016 from this tutorial on Dynet, given by Chris Dyer and Graham Neubig of CMU and Yoav Goldberg of Bar Ilan University. Dynet is a great package, especially if you want to use C++ and avoid dynamic typing. The final BiLSTM CRF exercise and the character-level features exercise are things I learned from this tutorial.
  • A great book on structure prediction is Linguistic Structure Prediction by Noah Smith. It doesn't use deep learning, but that is ok.
  • The best deep learning book I am aware of is Deep Learning, which is by some major contributors to the field and very comprehensive, although there is not an NLP focus. It is free online, but worth having on your shelf.

Exercises:

There are a few exercises in the tutorial, which are either to implement a popular model (CBOW) or augment one of my models. The character-level features exercise especially is very non-trivial, but very useful (I can't quote the exact numbers, but I have run the experiment before and usually the character-level features increase accuracy 2-3%). Since they aren't simple exercises, I will soon implement them myself and add them to the repo.

Suggestions:

Please open a GitHub issue if you find any mistakes or think there is a particular model that would be useful to add.

Owner
Robert
Software engineer at Citadel LLC and Georgia Tech grad. Primarily interested in natural language processing, finance, and high-performance computing.
Robert
C++ Implementation of PyTorch Tutorials for Everyone

C++ Implementation of PyTorch Tutorials for Everyone OS (Compiler)\LibTorch 1.9.0 macOS (clang 10.0, 11.0, 12.0) Linux (gcc 8, 9, 10, 11) Windows (msv

Omkar Prabhu 1.5k Jan 04, 2023
Pytorch implementations of various Deep NLP models in cs-224n(Stanford Univ)

DeepNLP-models-Pytorch Pytorch implementations of various Deep NLP models in cs-224n(Stanford Univ: NLP with Deep Learning) This is not for Pytorch be

Kim SungDong 2.9k Dec 24, 2022
This is a gentle introductin on how to start using an awesome library called Weights and Biases.

🪄 W&B Minimal PyTorch Tutorial This tutorial is also accompanied with a PyTorch source code, it can be found in src folder. Furthermore, all plots an

Nauryzbay K 8 Aug 20, 2022
Open source guides/codes for mastering deep learning to deploying deep learning in production in PyTorch, Python, C++ and more.

Deep Learning Materials by Deep Learning Wizard Start Learning Now Please head to www.deeplearningwizard.com to start learning! It is mobile/tablet fr

Ritchie Ng 572 Dec 28, 2022
PyTorch tutorials and best practices.

Effective PyTorch Table of Contents Part I: PyTorch Fundamentals PyTorch basics Encapsulate your model with Modules Broadcasting the good and the ugly

Vahid Kazemi 1.5k Jan 04, 2023
An IPython Notebook tutorial on deep learning for natural language processing, including structure prediction.

Table of Contents: Introduction to Torch's Tensor Library Computation Graphs and Automatic Differentiation Deep Learning Building Blocks: Affine maps,

Robert 1.8k Jan 04, 2023
A set of examples around pytorch in Vision, Text, Reinforcement Learning, etc.

PyTorch Examples WARNING: if you fork this repo, github actions will run daily on it. To disable this, go to /examples/settings/actions and Disable Ac

19.4k Jan 01, 2023
A collection of various deep learning architectures, models, and tips

Deep Learning Models A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks. Traditiona

Sebastian Raschka 15.5k Jan 07, 2023
PyTorch Implementation of Fully Convolutional Networks. (Training code to reproduce the original result is available.)

pytorch-fcn PyTorch implementation of Fully Convolutional Networks. Requirements pytorch = 0.2.0 torchvision = 0.1.8 fcn = 6.1.5 Pillow scipy tqdm

Kentaro Wada 1.6k Jan 04, 2023
PyTorch Tutorial for Deep Learning Researchers

This repository provides tutorial code for deep learning researchers to learn PyTorch. In the tutorial, most of the models were implemented with less

Yunjey Choi 25.4k Jan 05, 2023
Some example scripts on pytorch

pytorch-practice Some example scripts on pytorch CONLL 2000 Chunking task Uses BiLSTM CRF loss with char CNN embeddings. To run use: cd data/conll2000

Shubhanshu Mishra 180 Dec 22, 2022
Deep Learning (with PyTorch)

Deep Learning (with PyTorch) This notebook repository now has a companion website, where all the course material can be found in video and textual for

Alfredo Canziani 6.2k Jan 02, 2023
Minimal tutorials for PyTorch

Minimal tutorials for PyTorch adapted from Alec Radford's Theano tutorials. Tensor multiplication Linear Regression Logistic Regression Neural Network

Vinh Khuc 321 Oct 25, 2022
simple generative adversarial network (GAN) using PyTorch

Generative Adversarial Networks (GANs) in PyTorch Running Run the sample code by typing: ./gan_pytorch.py ...and you'll train two nets to battle it o

vanguard_space 32 Jun 14, 2020
Interactive deep learning book with multi-framework code, math, and discussions. Adopted at 200 universities.

D2L.ai: Interactive Deep Learning Book with Multi-Framework Code, Math, and Discussions Book website | STAT 157 Course at UC Berkeley | Latest version

Dive into Deep Learning (D2L.ai) 16k Jan 03, 2023
ConvNet training using pytorch

Convolutional networks using PyTorch This is a complete training example for Deep Convolutional Networks on various datasets (ImageNet, Cifar10, Cifar

Elad Hoffer 336 Dec 30, 2022
Simple PyTorch Tutorials Zero to ALL!

PyTorchZeroToAll Quick 3~4 day lecture materials for HKUST students. Video Lectures: (RNN TBA) Youtube Bilibili Slides Lecture Slides @GoogleDrive If

Sung Kim 3.7k Dec 30, 2022
Torch Containers simplified in PyTorch

pytorch-containers This repository aims to help former Torchies more seamlessly transition to the "Containerless" world of PyTorch by providing a list

Max deGroot 88 Apr 25, 2022
Simple examples to introduce PyTorch

This repository introduces the fundamental concepts of PyTorch through self-contained examples. At its core, PyTorch provides two main features: An n-

Justin Johnson 4.4k Jan 07, 2023
Image captioning - Tensorflow implementation of Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Introduction This neural system for image captioning is roughly based on the paper "Show, Attend and Tell: Neural Image Caption Generation with Visual

Guoming Wang 749 Dec 28, 2022