Reproduce results and replicate training fo T0 (Multitask Prompted Training Enables Zero-Shot Task Generalization)

Last update: Dec 27, 2022

Related tags

Overview

T-Zero

This repository serves primarily as codebase and instructions for training, evaluation and inference of T0.

T0 is the model developed in Multitask Prompted Training Enables Zero-Shot Task Generalization. In this paper, we demonstrate that massive multitask prompted fine-tuning is extremely effective to obtain task zero-shot generalization. T0 outperforms or matches GPT-3 while being 16x smaller.

While the codebase in this repository mainly reproduces and replicates the training and evaluation of T0, it will be useful for future research on massively multitask fine-tuning.

Training: reproducing (or replicating) the massively multitask fine-tuning
Evaluation: reproducing the main results reported in the paper
Inference: running inference with T0

Citation

If you find this resource useful, please cite the paper introducing T0:

@misc{sanh2021multitask,
      title={Multitask Prompted Training Enables Zero-Shot Task Generalization},
      author={Victor Sanh and Albert Webson and Colin Raffel and Stephen H. Bach and Lintang Sutawika and Zaid Alyafeai and Antoine Chaffin and Arnaud Stiegler and Teven Le Scao and Arun Raja and Manan Dey and M Saiful Bari and Canwen Xu and Urmish Thakker and Shanya Sharma Sharma and Eliza Szczechla and Taewoon Kim and Gunjan Chhablani and Nihal Nayak and Debajyoti Datta and Jonathan Chang and Mike Tian-Jian Jiang and Han Wang and Matteo Manica and Sheng Shen and Zheng Xin Yong and Harshit Pandey and Rachel Bawden and Thomas Wang and Trishala Neeraj and Jos Rozen and Abheesht Sharma and Andrea Santilli and Thibault Fevry and Jason Alan Fries and Ryan Teehan and Stella Biderman and Leo Gao and Tali Bers and Thomas Wolf and Alexander M. Rush},
      year={2021},
      eprint={2110.08207},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

Reproduce results and replicate training fo T0 (Multitask Prompted Training Enables Zero-Shot Task Generalization)

Related tags

Overview

T-Zero

Contents

Citation

Owner

BigScience Workshop

Pytorch implementation of the paper DocEnTr: An End-to-End Document Image Enhancement Transformer.

Recognize Handwritten Digits using Deep Learning on the browser itself.

Train Scene Graph Generation for Visual Genome and GQA in PyTorch >= 1.2 with improved zero and few-shot generalization.

Official code of paper: MovingFashion: a Benchmark for the Video-to-Shop Challenge

Machine-in-the-Loop Rewriting for Creative Image Captioning

Code for "Long-tailed Distribution Adaptation"

This repository contains a toolkit for collecting, labeling and tracking object keypoints

a Lightweight library for sequential learning agents, including reinforcement learning

Decentralized Reinforcment Learning: Global Decision-Making via Local Economic Transactions (ICML 2020)

Official PyTorch implementation of our AAAI22 paper: TransMEF: A Transformer-Based Multi-Exposure Image Fusion Framework via Self-Supervised Multi-Task Learning. Code will be available soon.

FastyAPI is a Stack boilerplate optimised for heavy loads.

Bag of Tricks for Natural Policy Gradient Reinforcement Learning

FedScale: Benchmarking Model and System Performance of Federated Learning

This repository contains source code for the Situated Interactive Language Grounding (SILG) benchmark

《LXMERT: Learning Cross-Modality Encoder Representations from Transformers》(EMNLP 2020)

🍅🍅🍅YOLOv5-Lite: lighter, faster and easier to deploy. Evolved from yolov5 and the size of model is only 1.7M (int8) and 3.3M (fp16). It can reach 10+ FPS on the Raspberry Pi 4B when the input size is 320×320~

Code for: https://berkeleyautomation.github.io/bags/

Implementation of SiameseXML (ICML 2021)

auto-tuning momentum SGD optimizer

Object-aware Contrastive Learning for Debiased Scene Representation