RITA is a family of autoregressive protein models, developed by LightOn in collaboration with the OATML group at Oxford and the Debora Marks Lab at Harvard.

Last update: Dec 22, 2022

Overview

RITA: a Study on Scaling Up Generative Protein Sequence Models

RITA is a family of autoregressive protein models, developed by a collaboration of Lighton, the OATML group at Oxford, and the Debbie Marks Lab at Harvard.

Model	#Params	d_model	layers	lm loss uniref-100
Small	85M	768	12	2.31
Medium	300M	1024	24	2.01
Large	680M	1536	24	1.82
XLarge	1.2B	2048	24	1.70

Results

For full results see our preprint: https://arxiv.org/abs/2205.05789

Usage

Instantiate a model like so:

from transformers import AutoModel, AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("lightonai/RITA_s, trust_remote_code=True")
tokenizer = AutoTokenizer.from_pretrained("lightonai/RITA_s")

for generation we support pipelines:

from transformers import pipeline
rita_gen = pipeline('text-generation', model=model, tokenizer=tokenizer)
sequences = rita_gen("MAB", max_length=20, do_sample=True, top_k=950, repetition_penalty=1.2, 
                     num_return_sequences=2, eos_token_id=2)
for seq in sequences:
    print(f"seq: {seq['generated_text'].replace(' ', '')}")

Or see example.py

How to cite

@article{hesslow2022rita,
  title={RITA: a Study on Scaling Up Generative Protein Sequence Models},
  author={Hesslow, Daniel and Zanichelli, Niccol{\'o} and Notin, Pascal and Poli, Iacopo and Marks, Debora},
  journal={arXiv preprint arXiv:2205.05789},
  year={2022}
}

RITA is a family of autoregressive protein models, developed by LightOn in collaboration with the OATML group at Oxford and the Debora Marks Lab at Harvard.

Related tags

Overview

RITA: a Study on Scaling Up Generative Protein Sequence Models

Results

Usage

How to cite

Owner

LightOn

Code for paper "Learning to Reweight Examples for Robust Deep Learning"

Cancer metastasis detection with neural conditional random field (NCRF)

Unofficial PyTorch implementation of MobileViT.

A basic implementation of Layer-wise Relevance Propagation (LRP) in PyTorch.

A code implementation of AC-GC: Activation Compression with Guaranteed Convergence, in NeurIPS 2021.

CoReD: Generalizing Fake Media Detection with Continual Representation using Distillation (ACMMM'21 Oral Paper)

Action Segmentation Evaluation

Picasso: A CUDA-based Library for Deep Learning over 3D Meshes

Answer a series of contextually-dependent questions like they may occur in natural human-to-human conversations.

natural image generation using ConvNets

Machine Learning toolbox for Humans

PyTorch implementation of Weak-shot Fine-grained Classification via Similarity Transfer

A Python wrapper for Google Tesseract

My coursework for Machine Learning (2021 Spring) at National Taiwan University (NTU)

TransGAN: Two Transformers Can Make One Strong GAN

An official implementation of "SFNet: Learning Object-aware Semantic Correspondence" (CVPR 2019, TPAMI 2020) in PyTorch.

Code for our EMNLP 2021 paper “Heterogeneous Graph Neural Networks for Keyphrase Generation”

This repository is for Competition for ML_data class

Pytorch implementation code for [Neural Architecture Search for Spiking Neural Networks]

Improving Compound Activity Classification via Deep Transfer and Representation Learning