Meta Language-Specific Layers in Multilingual Language Models

This repo contains the source codes for our paper

On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment

Zirui Wang, Zachary C. Lipton, Yulia Tsvetkov

EMNLP 2020

Introduction

This repo contains code to train multilingual language models (XLM) that (1) contain language-specific layers, and (2) meta-learn these layers through gradient of gradient.

Language-specific layers are served as meta parameters, optimized using an iterative procedure. The goal is to remedy negative transfer in multilingual models through a meta training objective. Please see our paper for details.

Dependencies

Python 3
XLM
NumPy
PyTorch

Usage

The code is based on the official implementation of XLM. This repo only contains files that we modified from the original codebase. To train a model, please merge code with the source code of XLM, and then follow the standard preprocessing and training instructions there.

Meta Language-Specific Layers in Multilingual Language Models

Related tags

Overview

Meta Language-Specific Layers in Multilingual Language Models

Introduction

Dependencies

Usage

Owner

Zirui Wang

Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes

Implementation of Shape Generation and Completion Through Point-Voxel Diffusion

A very lightweight monitoring system for Raspberry Pi clusters running Kubernetes.

Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

Official codebase for Decision Transformer: Reinforcement Learning via Sequence Modeling.

The MLOps platform for innovators 🚀

[AAAI-2022] Official implementations of MCL: Mutual Contrastive Learning for Visual Representation Learning

CUAD

Let's create a tool to convert Thailand budget from PDF to CSV.

A python library to artfully visualize Factorio Blueprints and an interactive web demo for using it.

A Simulated Optimal Intrusion Response Game

Deep learning (neural network) based remote photoplethysmography: how to extract pulse signal from video using deep learning tools

PyTorch implementation for our paper "Deep Facial Synthesis: A New Challenge"

Learning Tracking Representations via Dual-Branch Fully Transformer Networks

A-SDF: Learning Disentangled Signed Distance Functions for Articulated Shape Representation (ICCV 2021)

Capstone-Project-2 - A game program written in the Python language

Code for paper "ASAP-Net: Attention and Structure Aware Point Cloud Sequence Segmentation"

Shallow Convolutional Neural Networks for Human Activity Recognition using Wearable Sensors

An implementation of a discriminant function over a normal distribution to help classify datasets.

Convert weight file.pth to weight file.blob