RITA is a family of autoregressive protein models, developed by LightOn in collaboration with the OATML group at Oxford and the Debora Marks Lab at Harvard.

Last update: Dec 22, 2022

Overview

RITA: a Study on Scaling Up Generative Protein Sequence Models

RITA is a family of autoregressive protein models, developed by a collaboration of Lighton, the OATML group at Oxford, and the Debbie Marks Lab at Harvard.

Model	#Params	d_model	layers	lm loss uniref-100
Small	85M	768	12	2.31
Medium	300M	1024	24	2.01
Large	680M	1536	24	1.82
XLarge	1.2B	2048	24	1.70

Results

For full results see our preprint: https://arxiv.org/abs/2205.05789

Usage

Instantiate a model like so:

from transformers import AutoModel, AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("lightonai/RITA_s, trust_remote_code=True")
tokenizer = AutoTokenizer.from_pretrained("lightonai/RITA_s")

for generation we support pipelines:

from transformers import pipeline
rita_gen = pipeline('text-generation', model=model, tokenizer=tokenizer)
sequences = rita_gen("MAB", max_length=20, do_sample=True, top_k=950, repetition_penalty=1.2, 
                     num_return_sequences=2, eos_token_id=2)
for seq in sequences:
    print(f"seq: {seq['generated_text'].replace(' ', '')}")

Or see example.py

How to cite

@article{hesslow2022rita,
  title={RITA: a Study on Scaling Up Generative Protein Sequence Models},
  author={Hesslow, Daniel and Zanichelli, Niccol{\'o} and Notin, Pascal and Poli, Iacopo and Marks, Debora},
  journal={arXiv preprint arXiv:2205.05789},
  year={2022}
}

RITA is a family of autoregressive protein models, developed by LightOn in collaboration with the OATML group at Oxford and the Debora Marks Lab at Harvard.

Related tags

Overview

RITA: a Study on Scaling Up Generative Protein Sequence Models

Results

Usage

How to cite

Owner

LightOn

Stratified Transformer for 3D Point Cloud Segmentation (CVPR 2022)

A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch

Spatial Attentive Single-Image Deraining with a High Quality Real Rain Dataset (CVPR'19)

Cossim - Sharpened Cosine Distance implementation in PyTorch

StorSeismic: An approach to pre-train a neural network to store seismic data features

Hyper-parameter optimization for sklearn

PyTorch implementation code for the paper MixCo: Mix-up Contrastive Learning for Visual Representation

Code for MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks

Unofficial Tensorflow Implementation of ConvNeXt from A ConvNet for the 2020s

An attempt at the implementation of Glom, Geoffrey Hinton's new idea that integrates neural fields, predictive coding, top-down-bottom-up, and attention (consensus between columns)

Label Mask for Multi-label Classification

The implementation of the paper "A Deep Feature Aggregation Network for Accurate Indoor Camera Localization".

PyTorch implementation of "Simple and Deep Graph Convolutional Networks"

Copy Paste positive polyp using poisson image blending for medical image segmentation

The PyTorch implementation of paper REST: Debiased Social Recommendation via Reconstructing Exposure Strategies

Rainbow DQN implementation that outperforms the paper's results on 40% of games using 20x less data 🌈

A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization

PyTorch Implementation of CvT: Introducing Convolutions to Vision Transformers

TF Image Segmentation: Image Segmentation framework

[NeurIPS 2021] Galerkin Transformer: a linear attention without softmax