Meta Language-Specific Layers in Multilingual Language Models

This repo contains the source codes for our paper

On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment

Zirui Wang, Zachary C. Lipton, Yulia Tsvetkov

EMNLP 2020

Introduction

This repo contains code to train multilingual language models (XLM) that (1) contain language-specific layers, and (2) meta-learn these layers through gradient of gradient.

Language-specific layers are served as meta parameters, optimized using an iterative procedure. The goal is to remedy negative transfer in multilingual models through a meta training objective. Please see our paper for details.

Dependencies

Python 3
XLM
NumPy
PyTorch

Usage

The code is based on the official implementation of XLM. This repo only contains files that we modified from the original codebase. To train a model, please merge code with the source code of XLM, and then follow the standard preprocessing and training instructions there.

Meta Language-Specific Layers in Multilingual Language Models

Related tags

Overview

Meta Language-Specific Layers in Multilingual Language Models

Introduction

Dependencies

Usage

Owner

Zirui Wang

A library of scripts that interact with the PythonTurtle module to create games, drawings, and more

Code for paper "ASAP-Net: Attention and Structure Aware Point Cloud Sequence Segmentation"

Pomodoro timer that acknowledges the inexorable, infinite passage of time

This is the official Pytorch implementation of "Lung Segmentation from Chest X-rays using Variational Data Imputation", Raghavendra Selvan et al. 2020

Pytorch implementation for the paper: Contrastive Learning for Cold-start Recommendation

Angular & Electron desktop UI framework. Angular components for native looking and behaving macOS desktop UI (Electron/Web)

TensorFlow code for the neural network presented in the paper: "Structural Language Models of Code" (ICML'2020)

An interactive DNN Model deployed on web that predicts the chance of heart failure for a patient with an accuracy of 98%

[NeurIPS 2021] "Drawing Robust Scratch Tickets: Subnetworks with Inborn Robustness Are Found within Randomly Initialized Networks" by Yonggan Fu, Qixuan Yu, Yang Zhang, Shang Wu, Xu Ouyang, David Cox, Yingyan Lin

Deep Learning for Human Part Discovery in Images - Chainer implementation

A Model for Natural Language Attack on Text Classification and Inference

Sparse Physics-based and Interpretable Neural Networks

Code for the published paper : Learning to recognize rare traffic sign

Semantically Contrastive Learning for Low-light Image Enhancement

In this project, we'll be making our own screen recorder in Python using some libraries.

TGS Salt Identification Challenge

Recommendation algorithms for large graphs

Only works with the dashboard version / branch of jesse

Keras like implementation of Deep Learning architectures from scratch using numpy.

State-of-the-art language models can match human performance on many tasks