Implementation of the paper "Fine-Tuning Transformers: Vocabulary Transfer"

Last update: Nov 30, 2022

Related tags

Overview

Transformer-vocabulary-transfer

Implementation of the paper "Fine-Tuning Transformers: Vocabulary Transfer"

Description

step 1 - Create SentencePiece vocabulary for dataset
step 2 - Train the first level model (BertForMaskedLM) on English Wikipedia from scratch
step 3 - Match vocabulary (first level model dataset & downstream task dataset)
step 4 - Transfer dictionary using mapping. Сreate folders and raw models for experiments.
step 5 - Train 1 epoch BertForMaskedLM on downstream task
step 6 - Train final (BertForSequenceClassification) downstream model

Citation

I. Samenko, A. Tikhonov, B. Kozlovsky, I. P. Yamshchikov. Fine-Tuning Transformers: Vocabulary Transfer.

Contact

*Igor Samenko: [email protected]

Owner

LEYA

LEYA is a Natural Language Processing research laboratory co-founded by Yandex and Higher School of Economics in Saint-Petersburg.

GitHub Repository

Implementation of MeMOT - Multi-Object Tracking with Memory - in Pytorch

MeMOT - Pytorch (wip) Implementation of MeMOT - Multi-Object Tracking with Memory - in Pytorch. This paper is just one in a line of work, but importan

15 May 09, 2022

Deep Residual Networks with 1K Layers

Deep Residual Networks with 1K Layers By Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun. Microsoft Research Asia (MSRA). Table of Contents Introduc

856 Jan 06, 2023

Multi-agent reinforcement learning algorithm and environment

Multi-agent reinforcement learning algorithm and environment [en/cn] Pytorch implements multi-agent reinforcement learning algorithms including IQL, Q

7 Sep 20, 2022

This is an official implementation of CvT: Introducing Convolutions to Vision Transformers.

Introduction This is an official implementation of CvT: Introducing Convolutions to Vision Transformers. We present a new architecture, named Convolut

175 Jan 08, 2023

tensorflow code for inverse face rendering

InverseFaceRender This is tensorflow code for our project: Learning Inverse Rendering of Faces from Real-world Videos. (https://arxiv.org/abs/2003.120

18 Nov 16, 2022

Relaxed-machines - explorations in neuro-symbolic differentiable interpreters

Relaxed Machines Explorations in neuro-symbolic differentiable interpreters. Baby steps: inc_stop Libraries JAX Haiku Optax Resources Chapter 3 (∂4: A

6 Feb 02, 2022

The official implementation of the Interspeech 2021 paper WSRGlow: A Glow-based Waveform Generative Model for Audio Super-Resolution.

WSRGlow The official implementation of the Interspeech 2021 paper WSRGlow: A Glow-based Waveform Generative Model for Audio Super-Resolution. Audio sa

96 Jan 03, 2023

This is an early in-development version of training CLIP models with hivemind.

A transformer that does not hog your GPU memory This is an early in-development codebase: if you want a stable and documented hivemind codebase, look

[email protected]"> 4 Nov 06, 2022

Pytorch implementation of Value Iteration Networks (NIPS 2016 best paper)

VIN: Value Iteration Networks A quick thank you A few others have released amazing related work which helped inspire and improve my own implementation

297 Dec 26, 2022

This is the official implementation of Elaborative Rehearsal for Zero-shot Action Recognition (ICCV2021)

Elaborative Rehearsal for Zero-shot Action Recognition This is an official implementation of: Shizhe Chen and Dong Huang, Elaborative Rehearsal for Ze

26 Sep 24, 2022

Parasite: a tool allowing you to compress and decompress files, to reduce their size

🦠 Parasite 🦠 Parasite is a tool written in Python3 allowing you to "compress" any file, reducing its size. ⭐ Features ⭐ + Fast + Good optimization,

30 Nov 25, 2022

KAPAO is an efficient multi-person human pose estimation model that detects keypoints and poses as objects and fuses the detections to predict human poses.

KAPAO (Keypoints and Poses as Objects) KAPAO is an efficient single-stage multi-person human pose estimation model that models keypoints and poses as

664 Dec 30, 2022

Implementation of the paper "Fine-Tuning Transformers: Vocabulary Transfer"

Related tags

Overview

Transformer-vocabulary-transfer

Description

Citation

Contact

Owner

LEYA

Implementation of MeMOT - Multi-Object Tracking with Memory - in Pytorch

Deep Residual Networks with 1K Layers

Multi-agent reinforcement learning algorithm and environment

This is an official implementation of CvT: Introducing Convolutions to Vision Transformers.

tensorflow code for inverse face rendering

Relaxed-machines - explorations in neuro-symbolic differentiable interpreters

The official implementation of the Interspeech 2021 paper WSRGlow: A Glow-based Waveform Generative Model for Audio Super-Resolution.

This is an early in-development version of training CLIP models with hivemind.

Pytorch implementation of Value Iteration Networks (NIPS 2016 best paper)

This is the official implementation of Elaborative Rehearsal for Zero-shot Action Recognition (ICCV2021)

Parasite: a tool allowing you to compress and decompress files, to reduce their size

KAPAO is an efficient multi-person human pose estimation model that detects keypoints and poses as objects and fuses the detections to predict human poses.

2D Time independent Schrodinger equation solver for arbitrary shape of well

[SIGGRAPH Asia 2021] Pose with Style: Detail-Preserving Pose-Guided Image Synthesis with Conditional StyleGAN

Node for thenewboston digital currency network.

Augmented Traffic Control: A tool to simulate network conditions

Training Cifar-10 Classifier Using VGG16

Confident Semantic Ranking Loss for Part Parsing

Rate-limit-semaphore - Semaphore implementation with rate limit restriction for async-style (any core)

A robust camera and Lidar fusion based velocity estimator to undistort the pointcloud.