metamaker
Simple command line tool to train and deploy your machine learning models with AWS SageMaker
Features
metamaker enables you to:
- Build a docker image for training and inference with poetry and FastAPI
- Train your own machine learning model with SageMaker
- Deploy inference endpoint with SageMaker
Usage
- Create poetry project and install metamaker
❯ poetry new your_module
❯ cd your_module
❯ poetry add metamaker
- Define scripts for traning and inference in
main.py
Output: # ... from your_module import Model, Input, Output app = MetaMaker[Model, Input, Output]() @app.trainer def train( dataset_path: Path, artifact_path: Path, hyperparameters: Dict[str, Any], ) -> None: model = Model(**hyperparameters) model.train(dataset_path / "train.csv") model.save(artifact_path / "model.tar.gz") @app.loader def load(artifact_path: Path) -> Model: return Model.load(artifact_path / "model.tar.gz") @app.predictor def predict(model: Model, data: Input) -> Output: return model.predict(data) ">
from pathlib import Path from typing import Any, Dict from metamaker import MetaMaker # Import your model, and input/output data classs: # # Model ... machine learning model class you want to use # Input ... input data class for inference # Output ... ouput data class for inference # # Note that the Input and Output are used as type hints to # create API endpoint with FastAPI like below: # # @fastapi_app.post("/invocations") # def predict(data: Input) -> Output: # ... from your_module import Model, Input, Output app = MetaMaker[Model, Input, Output]() @app.trainer def train( dataset_path: Path, artifact_path: Path, hyperparameters: Dict[str, Any], ) -> None: model = Model(**hyperparameters) model.train(dataset_path / "train.csv") model.save(artifact_path / "model.tar.gz") @app.loader def load(artifact_path: Path) -> Model: return Model.load(artifact_path / "model.tar.gz") @app.predictor def predict(model: Model, data: Input) -> Output: return model.predict(data)
- Write metamaker configs in
metamaker.yaml
# Specify metamaker handler like: `path.to.module:app_name`
handler: main:app
# dataset_path and artifact_path should be directories and end with '/'
dataset_path: s3://your-bucket/path/to/dataset/
artifact_path: s3://your-bucket/path/to/artifacts/
hyperparameter_path: ./hparams.yaml
image:
name: metamaker
includes:
- your_module/
- main.py
excludes:
- __pycache__/
- '*.py[cod]'
training:
execution_role: arn:aws:iam::xxxxxxxxxxxx:role/SageMakerExecutionRole
instance:
type: ml.m5.large
count: 1
inference:
endpoint_name: your_endpoint
instance:
type: ml.t2.meduim
count: 1
- Build docker image and push to ECR
metamaker build --deploy .
- Train your model with SageMaker and deploy endpoint
metamker sagemaker train --deploy
30 Nov 08, 2022
3 Oct 12, 2022
41 Jul 27, 2022
6 Nov 13, 2022
7 Oct 18, 2022
2 Apr 01, 2022
2 Apr 21, 2022
3 May 24, 2022
11 Nov 15, 2022
5 Sep 30, 2022
102 Dec 14, 2022
23 Jul 04, 2022
19 Jan 21, 2022
4.5k Jan 01, 2023
1 Dec 29, 2021
10 Jun 26, 2022
143 Jan 02, 2023
10 Apr 18, 2022
42 Dec 18, 2022
1 May 25, 2022