Introduction

This repository contains the code for the paper Sentence Bottleneck Autoencoders from Transformer Language Models by Ivan Montero, Nikolaos Pappas, and Noah A. Smith published at EMNLP 2021.

This paper proposes an approach to learning sentence representations by applying an autoencoder on top of pretrained masked LMs.

Further documentation and code cleanup are under the works!

Installation

Coming soon!

This code was tested under Ubuntu 18.04, Python 3.7, and PyTorch 1.6.

To install, run the setup file.

bash setup.sh

We use modified versions of the following repositories:

transformers==3.3.1
fairseq==0.9.0

Example Usage

Coming soon!

Modified Files

Coming soon!