The full training script for Enformer (Tensorflow Sonnet) on TPU clusters

Last update: Oct 19, 2022

Overview

Enformer TPU training script (wip)

The full training script for Enformer (Tensorflow Sonnet) on TPU clusters, in an effort to migrate the model to pytorch.

This was pieced together from the Deepmind Enformer repository, the colab training notebook, as well as Basenji sequence augmentation code

It accounts for:

distributed TPU training
distributed datasets
distributed validation
gradient clipping
cross replica batchnorms
dataset augmentation

Training takes about 3 days on v3-64

Todo

fix script for differences in sequence length in basenji training data, which is ~130k vs ~190k bp as in paper

Citations

@article {Avsec2021.04.07.438649,
    author  = {Avsec, {\v Z}iga and Agarwal, Vikram and Visentin, Daniel and Ledsam, Joseph R. and Grabska-Barwinska, Agnieszka and Taylor, Kyle R. and Assael, Yannis and Jumper, John and Kohli, Pushmeet and Kelley, David R.},
    title   = {Effective gene expression prediction from sequence by integrating long-range interactions},
    elocation-id = {2021.04.07.438649},
    year    = {2021},
    doi     = {10.1101/2021.04.07.438649},
    publisher = {Cold Spring Harbor Laboratory},
    URL     = {https://www.biorxiv.org/content/early/2021/04/08/2021.04.07.438649},
    eprint  = {https://www.biorxiv.org/content/early/2021/04/08/2021.04.07.438649.full.pdf},
    journal = {bioRxiv}
}

The full training script for Enformer (Tensorflow Sonnet) on TPU clusters

Related tags

Overview

Enformer TPU training script (wip)

Todo

Citations

Owner

Phil Wang

A 1.3B text-to-image generation model trained on 14 million image-text pairs

[AAAI-2022] Official implementations of MCL: Mutual Contrastive Learning for Visual Representation Learning

Official PyTorch Implementation of HELP: Hardware-adaptive Efficient Latency Prediction for NAS via Meta-Learning (NeurIPS 2021 Spotlight)

Codebase for the self-supervised goal reaching benchmark introduced in the LEXA paper

[ICCV21] Code for RetrievalFuse: Neural 3D Scene Reconstruction with a Database

Bu repo SAHI uygulamasını mantığını öğreniyoruz.

Diffgram - Supervised Learning Data Platform

Extremely easy multi instancing software for minecraft speedrunning.

Detecting drunk people through thermal images using Deep Learning (CNN)

Code for paper "Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs"

BASH - Biomechanical Animated Skinned Human

Transfer style api - An API to use with Tranfer Style App, where you can use two image and transfer the style

Official code for our ICCV paper: "From Continuity to Editability: Inverting GANs with Consecutive Images"

Pytorch implementation of OCNet series and SegFix.

A tensorflow=1.13 implementation of Deconvolutional Networks on Graph Data (NeurIPS 2021)

Split Variational AutoEncoder

This MVP data web app uses the Streamlit framework and Facebook's Prophet forecasting package to generate a dynamic forecast from your own data.

Implementation of Deep Deterministic Policy Gradiet Algorithm in Tensorflow

GraphLily: A Graph Linear Algebra Overlay on HBM-Equipped FPGAs

[CVPR2021] Look before you leap: learning landmark features for one-stage visual grounding.