Meta Language-Specific Layers in Multilingual Language Models

This repo contains the source codes for our paper

On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment

Zirui Wang, Zachary C. Lipton, Yulia Tsvetkov

EMNLP 2020

Introduction

This repo contains code to train multilingual language models (XLM) that (1) contain language-specific layers, and (2) meta-learn these layers through gradient of gradient.

Language-specific layers are served as meta parameters, optimized using an iterative procedure. The goal is to remedy negative transfer in multilingual models through a meta training objective. Please see our paper for details.

Dependencies

Python 3
XLM
NumPy
PyTorch

Usage

The code is based on the official implementation of XLM. This repo only contains files that we modified from the original codebase. To train a model, please merge code with the source code of XLM, and then follow the standard preprocessing and training instructions there.

Meta Language-Specific Layers in Multilingual Language Models

Related tags

Overview

Meta Language-Specific Layers in Multilingual Language Models

Introduction

Dependencies

Usage

Owner

Zirui Wang

Unofficial pytorch implementation of the paper "Dynamic High-Pass Filtering and Multi-Spectral Attention for Image Super-Resolution"

This is the source code of the 1st place solution for segmentation task (with Dice 90.32%) in 2021 CCF BDCI challenge.

VoxHRNet - Whole Brain Segmentation with Full Volume Neural Network

Diabet Feature Engineering - Predict whether people have diabetes when their characteristics are specified

An unsupervised learning framework for depth and ego-motion estimation from monocular videos

Official implementation of the method ContIG, for self-supervised learning from medical imaging with genomics

[NeurIPS 2021] Introspective Distillation for Robust Question Answering

CoANet: Connectivity Attention Network for Road Extraction From Satellite Imagery

Unified Pre-training for Self-Supervised Learning and Supervised Learning for ASR

PyTorch META-DATASET (Few-shot classification benchmark)

Film review classification

Pytorch implementation of Cut-Thumbnail in the paper Cut-Thumbnail:A Novel Data Augmentation for Convolutional Neural Network.

ADB-IP-ROTATION - Use your mobile phone to gain a temporary IP address using ADB and data tethering

This is a repository for a Semantic Segmentation inference API using the Gluoncv CV toolkit

Addon and nodes for working with structural biology and molecular data in Blender.

Code for Environment Inference for Invariant Learning (ICML 2020 UDL Workshop Paper)

Poisson Surface Reconstruction for LiDAR Odometry and Mapping

Code for "FGR: Frustum-Aware Geometric Reasoning for Weakly Supervised 3D Vehicle Detection", ICRA 2021

Pytorch modules for paralel models with same architecture. Ideal for multi agent-based systems

Project to create an open-source 6 DoF input device