An end-to-end implementation of intent prediction with Metaflow and other cool tools

Overview

You Don't Need a Bigger Boat

An end-to-end (Metaflow-based) implementation of an intent prediction flow for kids who can't MLOps good and wanna learn to do other stuff good too.

This is a WIP - check back often for updates.

Philosophical Motivations

There is plenty of tutorials and blog posts around the Internet on data pipelines and tooling. However:

  • they (for good pedagogical reasons) tend to focus on one tool / step at a time, leaving us to wonder how the rest of the pipeline works;
  • they (for good pedagogical reasons) tend to work in a toy-world fashion, leaving us to wonder what would happen when a real dataset and a real-world problem enter the scene.

This repository (and soon-to-be-drafted written tutorial) aims to fill these gaps. In particular:

  • we provide open-source working code that glues together what we believe are some of the best tools in the ecosystem, going all the way from raw data to a deployed endpoint serving predictions;
  • we run the pipeline under a realistic load for companies at "reasonable scale", leveraging a huge open dataset we released in 2021; moreover, we train a model for a real-world use case, and show how to monitor it after deployment.

The repo may also be seen as a (very opinionated) introduction to modern, PaaS-like pipelines; while there is obviously room for disagreement over tool X or tool Y, we believe the general principles to be sound for companies at "reasonable scale": in-between bare-bone infrastructure for Tech Giants, and ready-made solutions for low-code/simple scenarios, there is a world of exciting machine learning at scale for sophisticated practitioners who don't want to waste their time managing cloud resources.

Overview

The repo shows how several (mostly open-source) tools can be effectively combined together to run data pipelines. The project current features:

The following picture from our Recsys paper (forthcoming) gives a quick overview of such a pipeline:

Recsys flow

We provide two versions of the pipeline, depending on the sophistication of the setup:

  • a Metaflow-only version, which runs from static data files (see below) to Sagemaker as a single Flow, and can be run from a Metaflow-enabled laptop without much additional setup;
  • a data warehouse version, which runs in a more realistic setup, reading data from Snowflake and using an external orchestrator to run the steps. In this setup, the downside is that a Snowflake and a Prefect Cloud accounts are required (nonetheless, both are veasy to get); the upside is that the pipeline reflects almost perfectly a real setup, and Metaflow can be used specifically for the ML part of the process.

The parallelism between the two scenarios should be pretty clear by looking at the two projects: if you are familiarizing with all the tools for the first time, we suggest you to start from the Metaflow version and then move to the full-scale one when all the pieces of the puzzle are well understood.

Relevant Material

If you want to know more, you can give a look at the following material:

TBC

Status Update

July 2021

End-2-end flow working for remote and local projects; started standardizing Prefect agents with Docker and adding other services (monitoring, feature store etc.).

TO-DOs:

  • dockerize the local flow;
  • write-up all of this as a blog post;
  • improve code / readability / docs, add potentially some more pics and some videos;
  • providing an orchestrator-free version, by using step functions to manage the steps;
  • finish feature store and gantry integration;
  • add Github Action flow;
  • continue improving the DAG card project.

Setup

General Prerequisites (do this first!)

Irrespectively of the flow you wish to run, some general tools need to be in place: Metaflow of course, as the heart of our ML practice, but also data and AWS users/roles. Please go through the general items below before tackling the flow-specific instructions.

After you finish the prerequisites below, you can run the flow you desire: each folder - remote and local - contains a specific README which should allow you to quickly run the project end-to-end: please refer to that documentation for flow-specific instructions (check back often for updates).

Dataset

The project leverages the open dataset from the 2021 Coveo Data Challenge: the dataset can be downloaded directly from here (refer to the full README for terms and conditions). Data is freely available under a research-friendly license - for background information on the dataset, the use cases and relevant work in the ML literature, please refer to the accompanying paper.

Once you download and unzip the dataset in a local folder of your choice (the zip contains 3 files, browsing_train.csv, search_train.csv, sku_to_content.csv), write down their location as an absolute path (e.g. /Users/jacopo/Documents/data/train/browsing_train.csv): both projects need to know where the dataset is.

AWS

Both projects - remote and local - use AWS services extensively - and by design: this ties back to our philosophy of PaaS-whenever-possible, and play nicely with our core adoption of Metaflow. While you can setup your users in many functionally equivalent ways, note that if you want to run the pipeline from ingestion to serving you need to be comfortable with the following AWS interactions:

  • Metaflow stack (see below): we assume you installed the Metaflow stack and can run it with an AWS profile of your choice;
  • Serverless stack (see below): we assume you can run serverless deploy in your AWS stack;
  • Sagemaker user: we assume you have an AWS user with permissions to manage Sagemaker endpoints (it may be totally distinct from any other Metaflow user).

TBC

Serverless

We wrap Sagemaker predictions in a serverless REST endpoint provided by AWS Lambda and API Gateway. To manage the lambda stack we use Serverless as a wrapper around AWS infrastructure.

TBC

Metaflow

Metaflow: Configuration

If you have an AWS profile configured with a metaflow-friendly user, and you created metaflow stack with CloudFormation, you can run the following command with the resources created by CloudFormation to set up metaflow on AWS:

metaflow configure aws --profile metaflow

Remember to use METAFLOW_PROFILE=metaflow to use this profile when running a flow. Once you completed the setup, you can run flow_playground.py to test the AWS setup is working as expected (in particular, GPU batch jobs can run correctly). To run the flow with the custom profile created, you should do:

METAFLOW_PROFILE=metaflow python flow_playground.py run

Metaflow: Tips & Tricks
  1. Parallelism Safe Guard
    • The flag --max-workers should be used to limit the maximum number of parallel steps
    • For example METAFLOW_PROFILE=metaflow python flow_playground.py run --max-workers 8 limits the maximum number of parallel tasks to 8
  2. Environment Variables in AWS Batch
    • The @environment decorator is used in conjunction with @batch to pass environment variables to AWS Batch, which will not directly have access to env variables on your local machine
    • In the local example, we use @environemnt to pass the Weights & Biases API Key (amongst other things)
  3. Resuming Flows
    • Resuming flows is useful during development to avoid re-running compute/time intensive steps such as data preparation
    • METAFLOW_PROFILE=metaflow python flow_playground.py resume <STEP_NAME> --origin-run-id <RUN_ID>
  4. Local-Only execution
    • It may sometimes be useful to debug locally (i.e to avoid Batch startup latency), we introduce a wrapper enable_decorator around the @batch decorator which enables or disables a decorator's functionality
    • We use this in conjunction with an environment variable EN_BATCH to toggle the functionality of all @batch decorators.

FAQ

  1. Both projects deal with data that has already been ingested/transmitted to the pipeline, but are silent on data collection. Any serverless option there as well?

    Yes. In e-commerce use cases, for example, pixel tracking is standard (e.g. Google Analytics), so a serverless /collect endpoint can be used to get front-end data and drop it in a pure PaaS pipeline with Firehose and Snowpipe, for example. While a bit out-dated for some details, we championed exactly this approach a while ago: if you want to know more, you can start from this Medium post and old code.

TBC

How to Cite our Work

If you find our principles, code or data useful, please cite our work:

Paper (forthcoming in RecSys2021)

@inproceedings{10.1145/3460231.3474604,
author = {Tagliabue, Jacopo},
title = {You Do Not Need a Bigger Boat: Recommendations at Reasonable Scale in a (Mostly) Serverless and Open Stack},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3460231.3474604},
doi = {10.1145/3460231.3474604},
series = {RecSys '21}
}

Data

@inproceedings{CoveoSIGIR2021,
author = {Tagliabue, Jacopo and Greco, Ciro and Roy, Jean-Francis and Bianchi, Federico and Cassani, Giovanni and Yu, Bingqing and Chia, Patrick John},
title = {SIGIR 2021 E-Commerce Workshop Data Challenge},
year = {2021},
booktitle = {SIGIR eCom 2021}
}
Owner
Jacopo Tagliabue
I failed the Turing Test once, but that was many friends ago.
Jacopo Tagliabue
Pytorch implementation for "Implicit Feature Alignment: Learn to Convert Text Recognizer to Text Spotter".

Implicit Feature Alignment: Learn to Convert Text Recognizer to Text Spotter This is a pytorch-based implementation for paper Implicit Feature Alignme

wangtianwei 61 Nov 12, 2022
RLDS stands for Reinforcement Learning Datasets

RLDS RLDS stands for Reinforcement Learning Datasets and it is an ecosystem of tools to store, retrieve and manipulate episodic data in the context of

Google Research 135 Jan 01, 2023
Code for the paper "Graph Attention Tracking". (CVPR2021)

SiamGAT 1. Environment setup This code has been tested on Ubuntu 16.04, Python 3.5, Pytorch 1.2.0, CUDA 9.0. Please install related libraries before r

122 Dec 24, 2022
PyTorch-based framework for Deep Hedging

PFHedge: Deep Hedging in PyTorch PFHedge is a PyTorch-based framework for Deep Hedging. PFHedge Documentation Neural Network Architecture for Efficien

139 Dec 30, 2022
Unofficial PyTorch implementation of the Adaptive Convolution architecture for image style transfer

AdaConv Unofficial PyTorch implementation of the Adaptive Convolution architecture for image style transfer from "Adaptive Convolutions for Structure-

65 Dec 22, 2022
Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators

Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators This is our Pytorch implementation for t

RUCAIBox 12 Jul 22, 2022
DeepMind's software stack for physics-based simulation and Reinforcement Learning environments, using MuJoCo.

dm_control: DeepMind Infrastructure for Physics-Based Simulation. DeepMind's software stack for physics-based simulation and Reinforcement Learning en

DeepMind 3k Dec 31, 2022
A lane detection integrated Real-time Instance Segmentation based on YOLACT (You Only Look At CoefficienTs)

Real-time Instance Segmentation and Lane Detection This is a lane detection integrated Real-time Instance Segmentation based on YOLACT (You Only Look

Jin 4 Dec 30, 2022
Near-Duplicate Video Retrieval with Deep Metric Learning

Near-Duplicate Video Retrieval with Deep Metric Learning This repository contains the Tensorflow implementation of the paper Near-Duplicate Video Retr

2 Jan 24, 2022
Motion planning algorithms commonly used on autonomous vehicles. (path planning + path tracking)

Overview This repository implemented some common motion planners used on autonomous vehicles, including Hybrid A* Planner Frenet Optimal Trajectory Hi

Huiming Zhou 1k Jan 09, 2023
Dynamic vae - Dynamic VAE algorithm is used for anomaly detection of battery data

Dynamic VAE frame Automatic feature extraction can be achieved by probability di

10 Oct 07, 2022
A list of Machine Learning Art Colabs

ML Visual Art Colabs A list of cool Colabs on Machine Learning Imagemaking or other artistic purposes 3D Ken Burns Effect Ken Burns Effect by Manuel R

Derrick Schultz (he/him) 789 Dec 12, 2022
A Pytorch Implementation of a continuously rate adjustable learned image compression framework.

GainedVAE A Pytorch Implementation of a continuously rate adjustable learned image compression framework, Gained Variational Autoencoder(GainedVAE). N

39 Dec 24, 2022
Code from Daniel Lemire, A Better Alternative to Piecewise Linear Time Series Segmentation

PiecewiseLinearTimeSeriesApproximation code from Daniel Lemire, A Better Alternative to Piecewise Linear Time Series Segmentation, SIAM Data Mining 20

Daniel Lemire 21 Oct 27, 2022
Tensorflow implementation of "BEGAN: Boundary Equilibrium Generative Adversarial Networks"

BEGAN in Tensorflow Tensorflow implementation of BEGAN: Boundary Equilibrium Generative Adversarial Networks. Requirements Python 2.7 or 3.x Pillow tq

Taehoon Kim 922 Dec 21, 2022
PyTorch implementation of our ICCV 2021 paper Intrinsic-Extrinsic Preserved GANs for Unsupervised 3D Pose Transfer.

Unsupervised_IEPGAN This is the PyTorch implementation of our ICCV 2021 paper Intrinsic-Extrinsic Preserved GANs for Unsupervised 3D Pose Transfer. Ha

25 Oct 26, 2022
Air Quality Prediction Using LSTM

AirQualityPredictionUsingLSTM In this Repo, i present to you the winning solution of smart gujarat hackathon 2019 where the task was to predict the qu

Deepak Nandwani 2 Dec 13, 2022
On the Adversarial Robustness of Visual Transformer

On the Adversarial Robustness of Visual Transformer Code for our paper "On the Adversarial Robustness of Visual Transformers"

Rulin Shao 35 Dec 14, 2022
Code for the CVPR2021 workshop paper "Noise Conditional Flow Model for Learning the Super-Resolution Space"

NCSR: Noise Conditional Flow Model for Learning the Super-Resolution Space Official NCSR training PyTorch Code for the CVPR2021 workshop paper "Noise

57 Oct 03, 2022
Locationinfo - A script helps the user to show network information such as ip address

Description This script helps the user to show network information such as ip ad

Roxcoder 1 Dec 30, 2021