Kubernetes-native workflow automation platform for complex, mission-critical data and ML processes at scale. It has been battle-tested at Lyft, Spotify, Freenome, and others and is truly open-source.

Last update: Jan 01, 2023

Overview

Flyte

Flyte is a workflow automation platform for complex, mission-critical data, and ML processes at scale

Home Page · Quick Start · Documentation · Live Roadmap · RFC's & Proposals · Features · Community & Resources · Changelogs · Components

💥 Introduction

Flyte is a structured programming and distributed processing platform that enables highly concurrent, scalable, and maintainable workflows for Machine Learning and Data Processing. It is a fabric that connects disparate computation backends using a type-safe data dependency graph. It records all changes to a pipeline, making it possible to rewind time. It also stores a history of all executions and provides an intuitive UI, CLI, and REST/gRPC API to interact with the computation.

Flyte is more than a workflow engine -- it uses workflow as a core concept, and task (a single unit of execution) as a top-level concept. Multiple tasks arranged in a data producer-consumer order creates a workflow.

Workflows and Tasks can be written in any language, with out-of-the-box support for Python, Java and Scala. Flyte was designed to manage complexity that arises in Data and ML teams and ensures them to keep up their high velocity of delivering business impacting features. One way to it achieves this is by separating the control-plane from the user-plane. Thus, every organization can offer Flyte like a service to their end users and the service is managed by folks who are more infrastructure focused, while the users use the intuitive interface of flytekit.

⏳ Five Reasons to Use Flyte

Kubernetes-Native Workflow Automation Platform
Ergonomic SDK's in Python, Java & Scala
Versioned, Auditable & Reproducible Pipelines
Data Aware and Strongly typed
Resource aware and deployed to scale with your organization

🚀 Quick Start

With Docker installed and Flytectl installed, run the following command:

  flytectl sandbox start

This creates a local Flyte sandbox. Once the sandbox is ready, you should see the following message: Flyte is ready! Flyte UI is available at http://localhost:30081/console.

Visit http://localhost:30081/console to view the Flyte dashboard.

Here's a quick visual tour of the console:

To dig deeper into Flyte, refer to the Documentation.

⭐️ Current Deployments & Contributors

🛣️ Live Roadmap

Live roadmap for the project can be found @Github Live Roadmap.

🔥 Features

Used at Scale in production by 500+ on one deployment. Used in production at multiple firms. Proved to scale to more than 1 million executions, and 40+ million containers
Data Aware and Resource aware (Allows organizations to separate concerns - users can use the API, platforms/infra teams can manage the deployments and scaling)
Enables collaboration across your organization by:
- Executing distributed data pipelines/workflows
- Making it easy to stitch together workflows from different teams and domain experts and share them across teams
- Comparing results of training workflows over time and across pipelines
- Simplifying the complexity of multi-step, multi-owner workflows
Get Started quickly -- start locally and scale to the cloud instantly
gRPC / REST interface to define and execute tasks and workflows
Type safe construction of pipelines -- each task has an interface characterized by its input and output, so illegal construction of pipelines fails during declaration, rather than at runtime
Supports multiple data types for machine learning and data processing pipelines, such as Blobs (images, arbitrary files), Directories, Schema (columnar structured data), collections, maps, etc.
Memoization and Lineage tracking
Provides logging and observability
Workflow features:
- Start with one task, convert to a pipeline, attach multiple schedules, trigger using a programmatic API, or on-demand
- Parallel step execution
- Extensible backend to add customized plugin experience (with simplified user experience)
- Branching
- Workflow of workflows - subworkflows (a workflow can be embedded within one node of the top-level workflow)
- Distributed remote child workflows (a remote workflow can be triggered and statically verified at compile time)
- Array Tasks (map a function over a large dataset -- ensures controlled execution of thousands of containers)
- Dynamic workflows creation and execution with runtime type safety
- flytekit plugins with first-class support in Python
- Arbitrary flytekit-less containers tasks (RawContainer)
Guaranteed reproducibility of pipelines via:
Multi-cloud support (AWS, GCP, and others)
No single point of failure, and is resilient by design
Automated notifications to Slack, Email, and Pagerduty
Multi K8s cluster support
Out of the box support to run Spark jobs on K8s, Hive queries, etc.
Snappy Console & Golang CLI (flytectl)
Written in Golang and optimized for jobs that run for a long period of time.
Grafana templates (user/system observability)
Deploy with Helm and kustomize

🔌 Available Plugins

Containers
K8s Pods
AWS Batch ArrayJobs
K8s Pod Arrays
K8s Spark (native Pyspark and Java/Scala)
AWS Athena
Qubole Hive
Presto Queries
Distributed Pytorch (K8s Native) -- Pytorch Operator
Sagemaker (builtin algorithms & custom models)
Distributed Tensorflow (K8s Native)
Papermill notebook execution (Python and Spark)
Type safe and data checking for Pandas dataframe using Pandera
Versioned datastores using DoltHub and Dolt
Use SQLAlchemy to query any relational database
Build your own plugins that use library containers
Snowflake queries

📦 Component Repos

Repo	Language	Purpose	Status
flyte	Kustomize,RST	deployment, documentation, issues	Production-grade
flyteidl	Protobuf	gRPC/REST API, Workflow language spec	Production-grade
flytepropeller	Go	execution engine	Production-grade
flyteadmin	Go	control plane	Production-grade
flytekit	Python	python SDK and tools	Production-grade
flyteconsole	Typescript	Flyte UI	Production-grade
datacatalog	Go	manage input & output artifacts	Production-grade
flyteplugins	Go	Flyte Backend plugins	Production-grade
flytecopilot	Go	Sidecar to manage input/output for sdk-less	Production-grade
flytestdlib	Go	standard library	Production-grade
flytesnacks	Python	examples, tips, and tricks	Maintained
flytekit-java	Java/Scala	Java & scala SDK for authoring Flyte workflows	Incubating
flytectl	Go	A standalone Flyte CLI	Production-grade
homebrew-tap	Ruby	Tap for downloadable flyte tools (cli etc)	Production-grade
bazel-rules	skylark/py	Use Bazel to build Flyte workflows and tasks	Incubating

Functional Tests Matrix

We run a suite of tests (defined in https://github.com/flyteorg/flytesnacks/blob/master/cookbook/flyte_tests_manifest.json) to ensure that basic functionality, and a subset of the integrations work across a variety of release versions. Those tests are run in a cluster where specific versions of the flyte components, such as console, flyteadmin, datacatalog, and flytepropeller, are installed. The table below has different release versions as the columns and the result of each test suite as rows.

workflow group	nightly
core
integrations-hive
integrations-k8s-spark
integrations-kfpytorch
integrations-pod
integrations-pandera_examples
integrations-papermilltasks
integrations-greatexpectations
integrations-sagemaker-pytorch
integrations-sagemaker-training

🛣️ RFC's (Request for Commments) & Proposals

Flyte is a Community Driven and Community Owned Software. It is managed using a steering committee and encourages collaboration. The community has a long roadmap for Flyte, but they know that, there might be some other interesting ideas, extensions or additions that you may want to propose. This is done usually starting with a

Github Issue - We maintain issues for all repos in the main flyte repo.
Writing down your proposal using a Documented RFC process
For small changes RFCs are not required, but for larger changes, RFC's are encouraged. Ofcourse drop into the Slack channel and talk to the community, if you want to test the waters, before proposing.

🤝 Community & Resources

Here are some resources to help you learn more about Flyte.

Communication Channels

Biweekly Community Sync

📣 Flyte OSS Community Sync Every other Tuesday, 9am-10am PDT. Check out the calendar, and register to stay up-to-date with our meeting times. Or join us on Zoom.
Upcoming meeting agenda, previous meeting notes, and a backlog of topics are captured in this document.
If you'd like to revisit any previous community sync meetings, you can access the video recordings on Flyte's YouTube channel.

Blog Posts

Flyte blog site

Flyte Monthly

Conference Talks & Pod Casts

Conference

Kubecon 2019 - Flyte: Cloud Native Machine Learning and Data Processing Platform video | deck
Kubecon 2019 - Running LargeScale Stateful workloads on Kubernetes at Lyft video
re:invent 2019 - Implementing ML workflows with Kubernetes and Amazon Sagemaker video
Cloud-native machine learning at Lyft with AWS Batch and Amazon EKS video
OSS + ELC NA 2020 splash
Datacouncil video | splash
FB [email protected] Making MLOps & DataOps a reality
GAIC 2020
OSPOCon 2021:
- Building and Growing an Open Source Community for an Incubating Project video
- Enforcing Data Quality in Data Processing and ML Pipelines with Flyte and Pandera video
- Self-serve Feature Engineering Platform Using Flyte and Feast video
- Efficient Data Parallel Distributed Training with Flyte, Spark & Horovod video
KubeCon+CloudNativeCon North America 2021 - How Spotify Leverages Flyte To Coordinate Financial Analytics Company-Wide session
PyData Global 2021 - Robust, End-to-end Online Machine Learning Applications with Flytekit, Pandera and Streamlit session
ODSC West Reconnect - Deep Dive Into Flyte workshop

Podcasts

TWIML&AI - Scalable and Maintainable ML Workflows at Lyft - Flyte
Software Engineering Daily - Flyte: Lyft Data Processing Platform
MLOps Coffee session - Flyte: an open-source tool for scalable, extensible, and portable workflows
Open Data Science - West Warm Up session with Ketan Umare - Creator of Flyte

💖 All Contributors

A big thank you to the community for making Flyte possible!

Comments

[Docs] Flytectl docs cleanup
Describe the documentation update

Fix typos, grammar in flytectl docs

Format code and content to properly render the docs

Add any missing command docs and also add some more examples if there's a possibility (optional)

Make sure flags' descriptions respect capitalized case

Getting the docs consistent to have flytectl command instead of bin/flytectl which is used in some places

Providing output for the example commands

On the whole, improve flytectl user experience through docs!

GitHub repo(s) flytectl
documentation flytectl hacktoberfest
opened by samhita-alla 27

[Flytectl Feature] Add field selector for all get commands

Describe the feature/command for flytectl Currently the get list commands for various flyte resources including project, tasks, workflows, launchplans, executions etc return the entire result set or are bounded through Limit query inorder to not fetch the entire dataset for that object type.

The ask of this feature is to implement a field selector interface for flytectl which allows to narrow down the result set matching the field selector criteria.

This is similar to kubectl field selector. https://kubernetes.io/docs/concepts/overview/working-with-objects/field-selectors/

Provide a possible output or ux example Once implemented, user should be able to fetch project, tasks, workflows,launchplans,executions based on the field selector. Invalid field selector should throw an error or provide details on whats fields are available for selection.

eg: This one returns all the executions in the domain and project.

{
    "executions": [
        {
            "id": {
                "project": "flytesnacks",
                "domain": "development",
                "name": "er7xaa2vxf"
            },
            "spec": {
                "launch_plan": {
                    "resource_type": "LAUNCH_PLAN",
                    "project": "flytesnacks",
                    "domain": "development",
                    "name": "recipes.core.basic.lp.go_greet",
                    "version": "v103"
                },
                "metadata": {
                    "principal": "flyteconsole",
                    "system_metadata": {}
                }
            },
            "closure": {
                "outputs": {
                    "uri": "s3://my-s3-bucket/metadata/propeller/flytesnacks-development-er7xaa2vxf/end-node/data/0/outputs.pb"
                },
                "phase": "SUCCEEDED",
                "started_at": "2021-02-17T07:10:09.271487Z",
                "duration": "43.817508900s",
                "created_at": "2021-02-17T07:10:09.234425400Z",
                "updated_at": "2021-02-17T07:10:53.088995900Z",
                "workflow_id": {
                    "resource_type": "WORKFLOW",
                    "project": "flytesnacks",
                    "domain": "development",
                    "name": "recipes.core.basic.lp.go_greet",
                    "version": "v103"
                }
            }
        },
        {
            "id": {
                "project": "flytesnacks",
                "domain": "development",
                "name": "oeh94k9r2r"
            },
            "spec": {
                "launch_plan": {
                    "resource_type": "LAUNCH_PLAN",
                    "project": "flytesnacks",
                    "domain": "development",
                    "name": "recipes.core.07_juptyer_notebooks.whole_square_workflow.whole_square_wf",
                    "version": "v58"
                },
                "metadata": {
                    "principal": "flyteconsole",
                    "system_metadata": {}
                }
            },
            "closure": {
                "abort_metadata": {},
                "phase": "FAILED",
                "started_at": "2021-02-05T14:50:00.598630500Z",
                "duration": "27.059172300s",
                "created_at": "2021-02-05T14:50:00.470665800Z",
                "updated_at": "2021-02-05T14:50:27.657802300Z",
                "workflow_id": {
                    "resource_type": "WORKFLOW",
                    "project": "flytesnacks",
                    "domain": "development",
                    "name": "recipes.core.07_juptyer_notebooks.whole_square_workflow.whole_square_wf",
                    "version": "v58"
                }
            }
        }
    ]
}

Now with field selector i should be able to specify any field in the json and be able to filter based on it. for eg on the phase field. this should result in returning just one matching the phase criterial w.r.t above data bin/flytectl get executions --field-selector=phase=SUCCEEDED

{
    "executions": [
        {
            "id": {
                "project": "flytesnacks",
                "domain": "development",
                "name": "er7xaa2vxf"
            },
            "spec": {
                "launch_plan": {
                    "resource_type": "LAUNCH_PLAN",
                    "project": "flytesnacks",
                    "domain": "development",
                    "name": "recipes.core.basic.lp.go_greet",
                    "version": "v103"
                },
                "metadata": {
                    "principal": "flyteconsole",
                    "system_metadata": {}
                }
            },
            "closure": {
                "outputs": {
                    "uri": "s3://my-s3-bucket/metadata/propeller/flytesnacks-development-er7xaa2vxf/end-node/data/0/outputs.pb"
                },
                "phase": "SUCCEEDED",
                "started_at": "2021-02-17T07:10:09.271487Z",
                "duration": "43.817508900s",
                "created_at": "2021-02-17T07:10:09.234425400Z",
                "updated_at": "2021-02-17T07:10:53.088995900Z",
                "workflow_id": {
                    "resource_type": "WORKFLOW",
                    "project": "flytesnacks",
                    "domain": "development",
                    "name": "recipes.core.basic.lp.go_greet",
                    "version": "v103"
                }
            }
        }
    ]
}

Following doc will help in finding how the filters work in admin service which flytectl communicates with for performing its operations. https://flyte.readthedocs.io/en/latest/dive_deep/admin_service.html#using-the-admin-service

enhancement flytectl

opened by pmahindrakar-oss 27

[Core Feature] Wait on ongoing task when cache is enabled

Motivation: Why do you think this is important? This would allow all concurrent cached Flyte tasks with the same input to wait on the first one to complete instead of causing a race condition when they execute at the same time. This would also help save resources by preventing repeated work.

Goal: What should the final outcome look like, ideally? When multiple cached tasks with the same inputs are running concurrently, only one should be executing, the rest should be waiting on that one to finish.

Describe alternatives you've considered Writing our own cache system by hashing inputs, storing them in s3, and checking them before executing the main logic.

[Optional] Propose: Link/Inline OR Additional context Cached tasks already exist today, but it only works for already completed tasks. The proposal is to enable it for ongoing tasks as well.
enhancement

opened by michaels-lyft 26
Migrate to go module
Why

everything of flyte is on go-1.13 and go module has become mature to use in production

it seems to be the future

one fewer tool (dep) to install and manage

no more vendor folder

I tried the migration and it was smooth https://github.com/lyft/datacatalog/pull/21

good first issue
opened by honnix 24

[BUG] Propeller Panic On K8s Array Task (datacatalog implicated)

Describe the bug

We ran a map task over a Python task that takes in a FlyteFile as input, and returns an int:

@task(
    requests=Resources(cpu="12", mem="25Gi", ephemeral_storage="325Gi"),
    limits=Resources(cpu="14", mem="25Gi"),
    retries=3,
    cache=True,
    cache_version="1.0",
)
def rf_germline_mappable(rf_germline_input_file: FlyteFile) -> int:

We have no special overrides on the map task, just:

    map_task(
        rf_germline_mappable, concurrency=constants.FLYTE_MAP_TASK_DEFAULT_CONCURRENCY
    )(rf_germline_input_file=comparison_sets).with_overrides(
        requests=Resources(cpu="12", mem="25Gi", ephemeral_storage="325Gi"),
        limits=Resources(cpu="14", mem="25Gi"),
    )

When this runs (the first time), it fails (no pods are spun up, it fails all in flyte propeller before the map task launches at all.

It fails when we resume as well.

None of the map tasks in the workflow before this seem to have this issue. The task that prepares the List[FlyteFiles] before this one seems to complete fine. We checked manually, and the FlyteFiles in question appear to all correctly exist in the S3 bucket that Flyte is using.

When this happens, we see a bunch of warnings in data catalog pod logs that look like: Dataset does not exist key: {Project:relative-finder Name:flyte_task-relative_finder.workflows.rf_germline.mapper_rf_germline_mappable_2 Domain:development Version:1.0-85909EZ4-PmK8qF6C UUID:}, err missing entity of type Dataset with identifier project:"relative-finder" name:"flyte_task-relative_finder.workflows.rf_germline.mapper_rf_germline_mappable_2" domain:"development" version:"1.0-85909EZ4-PmK8qF6C"

The stack trace is:

Workflow[relative-finder:development:relative_finder.workflows.relative_finder.relative_finder_wf] failed. RuntimeExecutionError: max number of system retry attempts [51/50] exhausted. Last known status message: failed at Node[n1]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [k8s-array]: panic when executing a plugin [k8s-array]. Stack: [goroutine 760 [running]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:24 +0x65
github.com/flyteorg/flytepropeller/pkg/controller/nodes/task.Handler.invokePlugin.func1.1()
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/handler.go:375 +0xfe
panic({0x1f3c580, 0x3952540})
	/usr/local/go/src/runtime/panic.go:838 +0x207
github.com/flyteorg/flytestdlib/bitarray.(*BitSet).IsSet(...)
	/go/pkg/mod/github.com/flyteorg/[email protected]/bitarray/bitset.go:33
github.com/flyteorg/flyteplugins/go/tasks/plugins/array/core.InitializeExternalResources({0x2796db0, 0xc01870b290}, {0x27a2700?, 0xc00116c4d0?}, 0xc002130240, 0x23cf0a8)
	/go/pkg/mod/github.com/flyteorg/[email protected]/go/tasks/plugins/array/core/metadata.go:33 +0x1e1
github.com/flyteorg/flyteplugins/go/tasks/plugins/array/k8s.Executor.Handle({{0x7fe260ad1090, 0xc00098afc0}, {{0x2789d50, 0xc0018ca0b0}}, {{0x2789d50, 0xc0018ca160}}}, {0x2796db0, 0xc01870b290}, {0x27a2700, 0xc00116c4d0})
	/go/pkg/mod/github.com/flyteorg/[email protected]/go/tasks/plugins/array/k8s/executor.go:94 +0x225
github.com/flyteorg/flytepropeller/pkg/controller/nodes/task.Handler.invokePlugin.func1(0x0?, {0x2796db0, 0xc01870b050}, {0x2799298?, 0xc0007daf00?}, 0x0?)
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/handler.go:382 +0x178
github.com/flyteorg/flytepropeller/pkg/controller/nodes/task.Handler.invokePlugin({{0x27970f8, 0xc0011919b0}, {0x27848f0, 0xc000d37da0}, 0xc00143ccc0, 0xc00143ccf0, 0xc00143cd20, {0x2798818, 0xc001710000}, 0xc0018522c0, ...}, ...)
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/handler.go:384 +0x9a
github.com/flyteorg/flytepropeller/pkg/controller/nodes/task.Handler.Handle({{0x27970f8, 0xc0011919b0}, {0x27848f0, 0xc000d37da0}, 0xc00143ccc0, 0xc00143ccf0, 0xc00143cd20, {0x2798818, 0xc001710000}, 0xc0018522c0, ...}, ...)
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/handler.go:617 +0x182b
github.com/flyteorg/flytepropeller/pkg/controller/nodes/dynamic.dynamicNodeTaskNodeHandler.handleParentNode({{0x279a148, 0xc000a5cdd0}, {{0xc000b258c0, {{...}, 0x0}, {0xc0009e8440, 0x4, 0x4}}, {0xc000b258e0, {{...}, ...}, ...}, ...}, ...}, ...)
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/dynamic/handler.go:70 +0xd8
github.com/flyteorg/flytepropeller/pkg/controller/nodes/dynamic.dynamicNodeTaskNodeHandler.Handle({{0x279a148, 0xc000a5cdd0}, {{0xc000b258c0, {{...}, 0x0}, {0xc0009e8440, 0x4, 0x4}}, {0xc000b258e0, {{...}, ...}, ...}, ...}, ...}, ...)
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/dynamic/handler.go:220 +0x9d0
github.com/flyteorg/flytepropeller/pkg/controller/nodes.(*nodeExecutor).execute(0xc00131e240, {0x2796db0, 0xc01870aba0}, {0x2798698, 0xc00143e000}, 0xc001082600, {0x27ab1b8?, 0xc001e981a0?})
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/executor.go:382 +0x157
github.com/flyteorg/flytepropeller/pkg/controller/nodes.(*nodeExecutor).handleQueuedOrRunningNode(0xc00131e240, {0x2796db0, 0xc01870aba0}, 0xc001082600, {0x2798698?, 0xc00143e000?})
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/executor.go:512 +0x227
github.com/flyteorg/flytepropeller/pkg/controller/nodes.(*nodeExecutor).handleNode(0xc00131e240, {0x2796db0, 0xc01870aba0}, {0x7fe2606e34d0, 0xc0050f5490}, 0xc001082600, {0x2798698?, 0xc00143e000})
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/executor.go:736 +0x3c5
github.com/flyteorg/flytepropeller/pkg/controller/nodes.(*nodeExecutor).RecursiveNodeHandler(0xc00131e240, {0x2796db0, 0xc01870a420}, {0x27a63e8, 0xc01ac29400}, {0x7fe2606e34d0, 0xc0050f5490}, {0x2784a30?, 0xc018156b60?}, {0x27a3810, ...})
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/executor.go:934 +0x705
github.com/flyteorg/flytepropeller/pkg/controller/nodes.(*nodeExecutor).handleDownstream(0x22f4f2d?, {0x2796db0, 0xc01870a420}, {0x27a63e8, 0xc01ac29400}, {0x7fe2606e34d0, 0xc0050f5490?}, {0x2784a30?, 0xc018156b60}, {0x27a3810, ...})
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/executor.go:774 +0x3c5
github.com/flyteorg/flytepropeller/pkg/controller/nodes.(*nodeExecutor).RecursiveNodeHandler(0xc00131e240, {0x2796db0, 0xc01870a420}, {0x27a63e8, 0xc01ac29400}, {0x7fe2606e34d0, 0xc0050f5490}, {0x2784a30?, 0xc018156b60?}, {0x27a3810, ...})
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/executor.go:941 +0x935
github.com/flyteorg/flytepropeller/pkg/controller/nodes.(*nodeExecutor).handleDownstream(0x22f4f2d?, {0x2796db0, 0xc01870a420}, {0x27a63e8, 0xc01ac29400}, {0x7fe2606e34d0, 0xc0050f5490?}, {0x2784a30?, 0xc018156b60}, {0x27a3810, ...})
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/executor.go:774 +0x3c5
github.com/flyteorg/flytepropeller/pkg/controller/nodes.(*nodeExecutor).RecursiveNodeHandler(0xc00131e240, {0x2796db0, 0xc01870a420}, {0x27a63e8, 0xc01ac29400}, {0x7fe2606e34d0, 0xc0050f5490}, {0x2784a30?, 0xc018156b60?}, {0x27a3810, ...})
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/executor.go:941 +0x935
github.com/flyteorg/flytepropeller/pkg/controller/nodes/subworkflow.(*subworkflowHandler).handleSubWorkflow(0xc000490c68, {0x2796db0, 0xc01870a420}, {0x27a4130, 0xc001082540}, {0x27a1b40, 0xc0050f5490}, {0x2784a30, 0xc018156b60})
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/subworkflow/subworkflow.go:74 +0x334
github.com/flyteorg/flytepropeller/pkg/controller/nodes/subworkflow.(*subworkflowHandler).CheckSubWorkflowStatus(0xc00af11c50?, {0x2796db0, 0xc01870a420}, {0x27a4130?, 0xc001082540?})
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/subworkflow/subworkflow.go:226 +0x3f1
github.com/flyteorg/flytepropeller/pkg/controller/nodes/subworkflow.(*workflowNodeHandler).Handle(0xc000490c40, {0x2796db0, 0xc01870a420}, {0x27a4130?, 0xc001082540?})
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/subworkflow/handler.go:91 +0x1690
github.com/flyteorg/flytepropeller/pkg/controller/nodes.(*nodeExecutor).execute(0xc00131e240, {0x2796db0, 0xc01870a420}, {0x2798758, 0xc000490c40}, 0xc001082540, {0x27ab1b8?, 0xc01bb53d40?})
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/executor.go:382 +0x157
github.com/flyteorg/flytepropeller/pkg/controller/nodes.(*nodeExecutor).handleQueuedOrRunningNode(0xc00131e240, {0x2796db0, 0xc01870a420}, 0xc001082540, {0x2798758?, 0xc000490c40?})
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/executor.go:512 +0x227
github.com/flyteorg/flytepropeller/pkg/controller/nodes.(*nodeExecutor).handleNode(0xc00131e240, {0x2796db0, 0xc01870a420}, {0x277d258, 0xc002cae500}, 0xc001082540, {0x2798758?, 0xc000490c40})
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/executor.go:736 +0x3c5
github.com/flyteorg/flytepropeller/pkg/controller/nodes.(*nodeExecutor).RecursiveNodeHandler(0xc00131e240, {0x2796db0, 0xc01870a0c0}, {0x27a63e8, 0xc01ac29360}, {0x277d258, 0xc002cae500}, {0x277d280?, 0xc002cae500?}, {0x27a3810, ...})
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/executor.go:934 +0x705
github.com/flyteorg/flytepropeller/pkg/controller/nodes.(*nodeExecutor).handleDownstream(0x22f4f2d?, {0x2796db0, 0xc01870a0c0}, {0x27a63e8, 0xc01ac29360}, {0x277d258, 0xc002cae500?}, {0x277d280?, 0xc002cae500}, {0x27a3810, ...})
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/executor.go:774 +0x3c5
github.com/flyteorg/flytepropeller/pkg/controller/nodes.(*nodeExecutor).RecursiveNodeHandler(0xc00131e240, {0x2796db0, 0xc01870a0c0}, {0x27a63e8, 0xc01ac29360}, {0x277d258, 0xc002cae500}, {0x277d280?, 0xc002cae500?}, {0x27a3810, ...})
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/nodes/executor.go:941 +0x935
github.com/flyteorg/flytepropeller/pkg/controller/workflow.(*workflowExecutor).handleRunningWorkflow(0xc000491e30, {0x2796db0, 0xc01870a0c0}, 0xc002cae500)
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/workflow/executor.go:147 +0x1b3
github.com/flyteorg/flytepropeller/pkg/controller/workflow.(*workflowExecutor).HandleFlyteWorkflow(0xc000491e30, {0x2796db0, 0xc01870a0c0}, 0xc002cae500)
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/workflow/executor.go:393 +0x40f
github.com/flyteorg/flytepropeller/pkg/controller.(*Propeller).TryMutateWorkflow.func2(0xc00145f0e0, {0x2796db0, 0xc01870a0c0}, 0xc010bf3848, 0x1e51040?)
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/handler.go:130 +0x18e
github.com/flyteorg/flytepropeller/pkg/controller.(*Propeller).TryMutateWorkflow(0xc00145f0e0, {0x2796db0, 0xc0186b5230}, 0xc002156f00)
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/handler.go:131 +0x459
github.com/flyteorg/flytepropeller/pkg/controller.(*Propeller).Handle(0xc00145f0e0, {0x2796db0, 0xc0186b5230}, {0xc004980330, 0x19}, {0xc00498034a, 0x8})
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/handler.go:205 +0x86d
github.com/flyteorg/flytepropeller/pkg/controller.(*WorkerPool).processNextWorkItem.func1(0xc00189ac60, 0xc010bf3f28, {0x1e51040?, 0xc003db8440})
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/workers.go:88 +0x510
github.com/flyteorg/flytepropeller/pkg/controller.(*WorkerPool).processNextWorkItem(0xc00189ac60, {0x2796db0, 0xc0186b5230})
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/workers.go:99 +0xf1
github.com/flyteorg/flytepropeller/pkg/controller.(*WorkerPool).runWorker(0x2796db0?, {0x2796db0, 0xc0007cda10})
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/workers.go:115 +0xbd
github.com/flyteorg/flytepropeller/pkg/controller.(*WorkerPool).Run.func1()
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/workers.go:150 +0x59
created by github.com/flyteorg/flytepropeller/pkg/controller.(*WorkerPool).Run
	/go/src/github.com/flyteorg/flytepropeller/pkg/controller/workers.go:147 +0x285
]

Expected behavior

No panci, code recognizes that the cache is empty and begins the task, caching the results.

Additional context to reproduce

We are on EKS, using managed node groups.

K8s version v1.22.10-eks-84b4fe6

Datacatalog cr.flyte.org/flyteorg/datacatalog-release:v1.1.0
Flyte propeller cr.flyte.org/flyteorg/flytepropeller:v1.1.24 (we have this up a bit higher for some more recent bugfixes that aren't on a proper release yet) Flyte admin cr.flyte.org/flyteorg/flyteadmin-release:v1.1.0

Screenshots

Unusual data catalog warning:

Are you sure this issue hasn't been raised already?

[X] Yes

Have you read the Code of Conduct?

[X] Yes

bug

opened by CalvinLeather 20

[Docs] `make` is resulting in an error when trying to build the flytesnacks docs
Description

There shouldn't be any errors. Will have to fix indentations, broken links, and RST styles.

Are you sure this issue hasn't been raised already?

[X] Yes

Have you read the Code of Conduct?

[X] Yes

documentation hacktoberfest
opened by samhita-alla 16
RFC for Eviction of cached task outputs

https://hackmd.io/qOztkaj4Rb6ypodvGEowAg?view

Comments already present on the HackMD doc are from our internal team and have been left in for clarification/further discussion.

Initial discussion on Slack

opened by MorpheusXAUT 16
[Feature] [FlyteKit] intra-task checkpointing and resumable task execution
Motivation: Why do you think this is important? ML training processes are usually iterative and long-running. The ML community has developed the checkpointing mechanism to preserve partial progress of a training process. The benefit of this mechanism is two-fold. Firstly, it allows users to jump-start the next time in the case of crashing or things going unexpectedly. Secondly, some of those intermediate checkpoints might actually be higher-quality models. It is not uncommon to see ML practitioners keep track of the best checkpoints (e.g., in terms of performance on validation dataset) in addition to the final model.

Checkpointing not only helps ML practitioners make the training more robust and efficient but also has an implication on cost optimization.

Today's cloud providers usually supply some types of interruptible compute instances as a way to maximize their own resource utilization. The incentive these providers give to the users is that leveraging these interruptible instances, such as AWS Spot Instances, gains the users some steep discounts on the hourly rates and therefore has the potential to save significantly.

Flyte's current way of leveraging spot instances works as follows: when a Flyte task is marked as interruptible, Flyte would first try to execute it on a spot instance and hope the execution to finish before getting interrupted. If the execution receives an interruption from the Spot Service, it is aborted and will be restarted from the beginning on the next instance it got scheduled onto. If the execution got interrupted too many times, to the extent that is beyond a predetermined level of tolerance, Flyte will then schedule it onto a normal, uninterruptible EC2 instance instead to ensure its completion.

This approach, however, has its limitations. When an execution runs on a Spot instance and gets interrupted, the progress is lost completely no matter how close it is to its completion. This lack of "resumability" incurs a significant amount of compute overhead, which naturally translates into extra dollars.

Checkpointing, luckily, provides the much-needed resumability. It allows users to persist some intermediate artifacts and data which allows the interrupted execution to resume from the partial progress made earlier without wasting time and money on computing what's already computed previously.

Goal: What should the final outcome look like, ideally? Flyte workflows are designed to be a way to checkpoint as they provide users a way to code a state machine with isolated failure domains. DataCatalog then helps in the recovery in alternate runs. With these two components working together, Flyte provides the checkpointing and resumption ability at task boundaries.

In this proposal, on the other hand, we aim to formulate the idea of explicit intra-task checkpointing. Our definition of intra-task checkpointing means a mechanism that (a) persists some intermediate artifacts that execution of user code produces on-demand, and (b) exposes them later to the same or a different execution of the same piece of user code so that users can "resume"

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Flyte component

[ ] Overall

[ ] Flyte Setup and Installation scripts

[ ] Flyte Documentation

[ ] Flyte communication (slack/email etc)

[x] FlytePropeller

[] FlyteIDL (Flyte specification language)

[x] Flytekit (Python SDK)

[ ] FlyteAdmin (Control Plane service)

[x] FlytePlugins

[x] DataCatalog

[ ] FlyteStdlib (common libraries)

[x] FlyteConsole (UI)

[ ] Other

[Optional] Propose: Link/Inline Link to the design doc: https://docs.google.com/document/d/1Ds8TlZpiHjDV48yWNITqrUkXNZNv6tA7VwcYEOYonf4/edit?ts=5fb7050a#

Additional context Add any other context or screenshots about the feature request here.

Is this a blocker for you to adopt Flyte Please let us know if this makes it impossible to adopt Flyte
enhancement flytekit
opened by bnsblue 16
Support run-to-completion flag

Right now, if any of the tasks in a workflow fails, Flyte will fail the entire workflow. We would like to have a flag to let the Flyte continue to run whatever tasks that are not affected by the failed tasks.

In the following example, if task 3 fails, task 5 will be marked won't run. While task 1, 2, 4 should continue to run.

enhancement

opened by datability-io 16
Updated the Existing Issue Templates to Issue forms
Updated all the existing issue templates to issue forms for ease of traversing through the multitude of issues and relevance of content to avoid spam. If there are any changes and or edits,please do let me know!

Issue: https://github.com/flyteorg/flyte/issues/1555

Here is how each of the template looks now:

Bug report:

Documentation:

Core Features:

Flyctectl feature:

Housekeeping:

UI feature:

hacktoberfest-accepted
opened by noobkid2411 15
[BUG] Task fails with error: `containers with unready status: [main]|failed to reserve container name`
Describe the bug We had a rare failure where a task (and consequently the workflow) failed with the following error:

containers with unready status: [main]|failed to reserve container name "main_f76be7b00ec9b4348871-n1-0_pineapple-development_987b2aa9-b537-4038-9884-8d2bc413762a_0": name "main_f76be7b00ec9b4348871-n1-0_pineapple-development_987b2aa9-b537-4038-9884-8d2bc413762a_0" is reserved for "6678ab271c041304d4a91e8e219dca896388098ec8a258dc49f4f9942296f669"

Expected behavior Flytepropeller handles these likely transient issues gracefully:

Relevant issue: https://github.com/containerd/containerd/issues/4604 See this comment:

Eventually, kubelet is able to resolve this issue without manual intervention, however, it is significantly slowing the deployment of new images during the release (extra 2-3 minutes to resolve name conflicts).

It is possible that Flyte might be catching and failing on the error too fast before kubelet can resolve this issue on its own. Flyte might need to wait and let this issue resolve, possibly with some reasonable timeout.

[Optional] Additional context To Reproduce Steps to reproduce the behavior: 1. 2.

Screenshots If applicable, add screenshots to help explain your problem.
bug
opened by jeevb 15
[Core feature] Convert List[Any] to a single pickle file
Motivation: Why do you think this is important?

Currently, flyte create N (size of list) pickle files if output type is List[Any]. This slows down serialization. it takes more than 15 mins to upload the pickles to s3 if the size of list is 1000.

People don't care about how we serialize List[Any]. We can just convert entire list into a single pickle file, which reduces the time required for serialization.

Goal: What should the final outcome look like, ideally?

it will make serialization faster

Describe alternatives you've considered

Raise an error when using large list

Add a detailed warning

Propose: Link/Inline OR Additional context

Slack Thread

Are you sure this issue hasn't been raised already?

[X] Yes

Have you read the Code of Conduct?

[X] Yes

enhancement flytekit
opened by pingsutw 0
[Core feature] [flytekit] data plugins update
Motivation: Why do you think this is important?

Currently we have our own datapersistence layer. This is because we wanted to support aws cli and gsutil. I think we should drop support for aws cli and gsutil and completely migrate to using datafusion / arrow file system or fsspec

On reading I see that the datafusion project has matured and has a fantastic rust object_Store crate which works great for Flytekit https://arrow.apache.org/docs/python/filesystems.html

Goal: What should the final outcome look like, ideally?

Flyte is enriched with the amazing integrations of arrow and the great growing ecosystem.

Describe alternatives you've considered

NA

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

[X] Yes

Have you read the Code of Conduct?

[X] Yes

enhancement flytekit
opened by kumare3 0
[Core feature] [Pod plugin] UX and example improvements
Motivation: Why do you think this is important?

Example for using Pod features is overly complicated.

Also the pod plugin does not make it simply possible to add pod-spec related features like node-selector etc.

it is possible to do this

@task( task_config=Pod( pod_spec=V1PodSpec( containers=[], node_selector={"node_group": "memory"}, ), ), requests=Resources( mem="1G", ), ) def my_pod_task(): ...

Goal: What should the final outcome look like, ideally?

Improve the example and point to the new pod plugin

Describe alternatives you've considered

NA

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

[X] Yes

Have you read the Code of Conduct?

[X] Yes

enhancement flytekit
opened by kumare3 0
[Core feature] Add support Hex plugin
Motivation: Why do you think this is important?

Hex is a modern Data Workspace. It makes it easy to connect to data, analyze it in collaborative SQL and Python-powered notebooks, and share work as interactive data apps and stories.

Hex creates a chart cell that allows editors to interactively explore and aggregate a dataframe, creating rich visualizations to share with app users.

We can add a Hex plugin to run the notebook in the Hex workspace and display the interactive chart on Flyte Deck.

https://user-images.githubusercontent.com/37936015/209454302-500e6861-488a-4edb-9186-54fb306170d7.mp4

Goal: What should the final outcome look like, ideally?

Have an interactive chart on Flyte Deck

Submit a job to Hex to run the notebook in Hex workspace.

Describe alternatives you've considered

NA

Propose: Link/Inline OR Additional context

https://learn.hex.tech/docs/logic-cell-types/display-cells/chart-cells

https://github.com/hex-inc/airflow-provider-hex

Are you sure this issue hasn't been raised already?

[X] Yes

Have you read the Code of Conduct?

[X] Yes

enhancement flytekit
opened by pingsutw 0
[BUG] Structured Dataset compatibility between plugins
Describe the bug

I was running some tasks in a notebooks where I was passing the results of a Spark task as a StructuredDataset and then trying to load them into a polars dataframe and a hugging face dataset.

It resulted in the following error for both plugins No such file or directory: /var/folders/wq/3hjh3ms916b6dj56zx0f_x000000gq/T/flyte-69d2tww2/sandbox/local_flytekit/95bac8efeb64a8d10d34c73b66df7051/00000. However, it did work for pandas.

It seems like polars and huggingface add in 00000 to the path in the transformers and spark does not.

polars: https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-polars/flytekitplugins/polars/sd_transformers.py#L43

spark: https://github.com/flyteorg/flytekit/blob/master/plugins/flytekit-spark/flytekitplugins/spark/sd_transformers.py#L29

Expected behavior

I would expect to be able to use a StructuredDataset from spark with dataframe libraries from all plugins.

Additional context to reproduce

from flytekit import task, StructuredDataset from flytekitplugins.spark.task import Spark from datasets import Dataset import polars as pl import datasets import pandas as pd

@task( task_config=Spark() ) def spark_task(path: str) -> StructuredDataset: sess = flytekit.current_context().spark_session df = sess.read.parquet(path) return StructuredDataset(dataframe=df)

df = spark_task(path="./ratings_100k.parquet")

try: df.open(pl.DataFrame).all().head() except Exception as e: print(e)

try: df.open(datasets.Dataset).all().head() except Exception as e: print(e)

df.open(pd.DataFrame).all().head()

Screenshots

Are you sure this issue hasn't been raised already?

[X] Yes

Have you read the Code of Conduct?

[X] Yes

bug untriaged
opened by esadler-hbo 1

Releases(v1.3.0-b8)

v1.3.0-b8(Dec 30, 2022)

v1.3.0-b8

Empty release for testing CI.
Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(353.43 KB)
flyte_gcp_manifest.yaml(352.63 KB)
flyte_sandbox_manifest.yaml(160.89 KB)
flyte_test_manifest.yaml(25.76 KB)
v1.3.0-b7(Dec 29, 2022)

v1.3.0-b7

Empty release for testing CI.
Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(353.43 KB)
flyte_gcp_manifest.yaml(352.63 KB)
flyte_sandbox_manifest.yaml(160.89 KB)
flyte_test_manifest.yaml(25.76 KB)
v1.3.0-b6(Dec 29, 2022)

Flyte v1.3.0-b6 Changelog

Pull in Doc hub changes.
Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(353.43 KB)
flyte_gcp_manifest.yaml(352.63 KB)
flyte_sandbox_manifest.yaml(160.89 KB)
flyte_test_manifest.yaml(25.76 KB)
v1.3.0-b5(Dec 20, 2022)

Flyte v1.3.0-b5 Changelog

Databricks and dbx plugin changes.
Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(353.23 KB)
flyte_gcp_manifest.yaml(352.44 KB)
flyte_sandbox_manifest.yaml(160.70 KB)
flyte_test_manifest.yaml(25.76 KB)
v1.3.0-b4(Dec 15, 2022)

Flyte v1.3.0-b4 Changelog

Pull in first batch of signaling changes.
Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(353.23 KB)
flyte_gcp_manifest.yaml(352.44 KB)
flyte_sandbox_manifest.yaml(160.70 KB)
flyte_test_manifest.yaml(25.76 KB)
v1.3.0-b3(Dec 7, 2022)

Flyte v1.3.0-b3 Changelog

Use checksums to apply cluster resource changes from flyteadmin
Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(353.23 KB)
flyte_gcp_manifest.yaml(352.44 KB)
flyte_sandbox_manifest.yaml(160.70 KB)
flyte_test_manifest.yaml(25.76 KB)
v1.2.1(Nov 17, 2022)
Flyte v1.2.1 Release

Improved Local Experience

This change introduces a new version of the flytectl demo start experience. With the v0.6.24 version of flytectl and later, when you start the demo environment, you'll get

Faster start up times after image downloaded as dependent images (like minio and postgres) now come pre-bundled.

A local Docker registry to use. Build custom task images tagged with localhost:30000 and push to make them available inside Flyte.

This release merges in all the changes here. https://github.com/flyteorg/flyte/compare/v1.2.0...819457850b634e626cb233b500e7855b65fc9719
Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(353.23 KB)
flyte_gcp_manifest.yaml(352.44 KB)
flyte_sandbox_manifest.yaml(160.70 KB)
flyte_test_manifest.yaml(25.76 KB)
v1.2.1-b1(Nov 16, 2022)

Flyte 1.2.1-b1 Release

Merged in all changes from master into this release-1.2 branch. https://github.com/flyteorg/flyte/compare/v1.2.0...819457850b634e626cb233b500e7855b65fc9719
Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(353.23 KB)
flyte_gcp_manifest.yaml(352.44 KB)
flyte_sandbox_manifest.yaml(160.70 KB)
flyte_test_manifest.yaml(25.76 KB)
v1.3.0-b2(Nov 15, 2022)

Flyte v1.3.0-b2 Changelog

New demo sandbox environment New Helm charts
Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(353.23 KB)
flyte_gcp_manifest.yaml(352.44 KB)
flyte_sandbox_manifest.yaml(160.70 KB)
flyte_test_manifest.yaml(25.76 KB)
v1.3.0-b1(Oct 14, 2022)

Flyte v1.3.0-b1 Changelog

Fix race in sandbox installation Updates to flyteadmin auth headers
Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(353.23 KB)
flyte_gcp_manifest.yaml(352.44 KB)
flyte_sandbox_manifest.yaml(160.69 KB)
flyte_test_manifest.yaml(25.75 KB)
v1.2.0(Oct 4, 2022)
Flyte 1.2 Release

Platform

Support for Ray (https://github.com/flyteorg/flyte/issues/2641) - Also see the blog post.

Execution names can be longer now, up to 63 characters (https://github.com/flyteorg/flyteadmin/pull/466)

Offloading FlyteWorkflow CRD static workflow spec (https://github.com/flyteorg/flyte/issues/2705)

Enabled FlytePropeller subqueue - this means that everytime a pod is updated in the workflow it reevals for faster downstream scheduling

Add container configuration to default pod template (https://github.com/flyteorg/flyte/issues/2703)

Fixed issues with blobstore writes - GCS had duplicate writes and subworkflow inputs were rewritten on every evaluation, this meant slower evaluations

Support external deletion of non-terminal map task subtasks (as a result of https://github.com/flyteorg/flyte/issues/2701)

Better truncation of flytepropeller logs (https://github.com/flyteorg/flyte/issues/2818)

Bug: Recovering subworkflows is now supported (https://github.com/flyteorg/flyte/issues/2840)

Fix snowflake plugin (https://github.com/flyteorg/flyteplugins/pull/286)

Flytekit

Hugging Face plugin (https://github.com/flyteorg/flytekit/pull/1116)

dbt plugin (https://github.com/flyteorg/flyte/issues/2202)

cache overriding behavior is now open to all types (https://github.com/flyteorg/flyte/issues/2912)

Bug: Fallback to pickling in the case of unknown types used Unions (https://github.com/flyteorg/flyte/issues/2823)

pyflyte run now supports imperative workflows

Newlines are now stripped from client secrets (https://github.com/flyteorg/flytekit/pull/1163)

Ensure repeatability in the generation of cache keys in the case of dictionaries (https://github.com/flyteorg/flytekit/pull/1126)

Support for multiple images in the yaml config file (https://github.com/flyteorg/flytekit/pull/1106)

And more. See the full changelog in https://github.com/flyteorg/flytekit/releases/tag/v1.2.0

Flyteconsole

fix: Make sure groups used in graph aren't undefined #545

fix: Graph Center on initial render #541

fix: Graph edge overlaps nodes #542

Fix searchbar X button #564

fix: Update timeline view to show dynamic wf internals on first render #562

fix: Webmanifest missing crossorigin attribute #566

fix: console showing subworkflows as unknown #570

fix: Dict value loses 1 trailing character on UI Launch. #561

fix: launchform validation #557

fix: integrate timeline and graph tabs wrappers under one component #572

added none type in union type #577

minor: inputHelpers InputProps #579

fix: fix test of launchform #581

Pruning some unused packages #583

fix: floor seconds to int in the edge case moment returns it as float #582

fix: add BASE_URL to dev startup, open deeply nested urls #589

fix: add default disabled state for only mine filter #585

fixed graph/timeline support for launchplanRef #601

fix: enable deeplinks in development #602

Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(353.23 KB)
flyte_gcp_manifest.yaml(352.44 KB)
flyte_sandbox_manifest.yaml(160.69 KB)
flyte_test_manifest.yaml(25.75 KB)
v1.2.0-b5(Oct 4, 2022)

1.2.0-b5 Test Release

Updated notifications to text
Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(353.23 KB)
flyte_gcp_manifest.yaml(352.44 KB)
flyte_sandbox_manifest.yaml(160.69 KB)
flyte_test_manifest.yaml(25.75 KB)
v1.2.0-b3(Sep 1, 2022)

Flyte v1.2.0-b3 Changelog

Allow adding sidecar containers to flyteadmin
Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(353.23 KB)
flyte_gcp_manifest.yaml(352.44 KB)
flyte_sandbox_manifest.yaml(160.69 KB)
flyte_test_manifest.yaml(25.75 KB)
v1.2.0-b2(Aug 17, 2022)

Flyte v1.2.0-b2 Changelog

Add support Ray task

Others will be filled in later
Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(353.23 KB)
flyte_gcp_manifest.yaml(352.44 KB)
flyte_sandbox_manifest.yaml(160.69 KB)
flyte_test_manifest.yaml(25.75 KB)
v1.2.0-b1(Jul 12, 2022)

Flyte v1.2.0-b1 Changelog

Fix indentation in flyte-core Helm chart
Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(353.27 KB)
flyte_gcp_manifest.yaml(352.48 KB)
flyte_sandbox_manifest.yaml(160.74 KB)
flyte_test_manifest.yaml(25.79 KB)
v1.1.0(Jul 6, 2022)
Flyte 1.1 Changelog

Platform

User Improvements

Support for Optional types. With the inclusion of Union types in flytekit, we can now support optional types.

Flyte Deck is now available. Please take a look at the documentation and also the OSS presentation that was done a few weeks back.

Backend Improvements

Allow different cookie settings in Admin.

Pluggable middleware in Admin authentication.

Server-side compiler strips Type Metadata.

Flytectl now uses latest launch plans for scheduled workflows

Allow spot instances at workflow start time

Bug Fixes

Propeller calling finalize rather than abort

Propeller correctly identify error when requesting a launch plan that does not exist.

Better handle execution CRDs that don't exist in Admin.

Fix panic when creating additional label options.

Check validity of notifications.

Revert Spark tasks to use the spark role if unspecified by the user.

...and more!

Flytekit

Work for Flyte Decks and Optional types.

pyflyte run now supports executing tasks.

pyflyte register combines the UX of run with the functionality of package. Please see the video posted along with the forthcoming documentation.

Bug Fixes

Pyflyte fixes - dot separated python packages fix, pyflyte run to respect the --config flag and read packages from environment variables

Set authorization key in the case of external command

UI

fix: support mapped tasks #494

feat: support flyte decks #504

feat: launch plans list & detail page #507

fix(bug-508): executions can not be filtered by start time #509

feat: navbar navigation dropdown #511

chore: support internal/external navigsation better #513

chore: Update Contributing.md #515

chore: update navigationDropdown usage #517

fix: cache icon for map task #519

fix: Relaunch form does not persist security context values when changed #527

fix: release process #529

fix: fix semantic-release config #532

test: fix time sensitive test #533

feat: Rename upgrade idl workflow #534

fix(491): remove favicon package + use favicon.svg by default #537

The lists above contains changes from the 1.0.x release as well.
Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(353.27 KB)
flyte_gcp_manifest.yaml(352.48 KB)
flyte_sandbox_manifest.yaml(160.74 KB)
flyte_test_manifest.yaml(25.79 KB)
v1.1.0-b1(Jun 30, 2022)

Flyte v1.1.0-b1 Changelog

will be filled in later
Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(353.23 KB)
flyte_gcp_manifest.yaml(352.44 KB)
flyte_sandbox_manifest.yaml(160.69 KB)
flyte_test_manifest.yaml(25.75 KB)
v1.0.2(Jun 7, 2022)
Flyte v1.0.2 Changelog

General

Housekeeping Update single binary component in releases and bump version of contour helm chart

Platform

Bugfix fix flyte-deps helm chart version

Bugfix Re-order clusterresourcesync annotations in helm chart

Feature Server-side compiler should strip Type Metadata

Bugfix With GRPC v1.46.0 non-ascii chars are not permitted in grpc metadata

Housekeeping Configure grpc_health_prob in admin

Feature In Flytectl use Launchplan with latest version for scheduled workflows

Bugfix Pods started before InjectFinalizer is disabled are never deleted

Housekeeping Checksum grpc_health_probe

Feature Allow to choose Spot Instances at workflow start time

Feature Use the same pod annotation formatting in syncresources cronjob

Housekeeping Migrate flyte sandbox docker image to use flyte-dep & flyte-core

Feature Buf integration for proto release

Flyteconsole

Refactor Move to monorepo structure to allow separate FlyteConsole into plugins system

Feature Add support for StructuredDataSet Input/Output type

Feature Updated Task details page to be able to browse different task versions

Feature Updated Input/Output and TaskDetails representation - now you can collapse parts of it.

Feature Show new map tasks in the GraphView with information of which subtasks are in which phase(running, erroring, succeeded)

Feature Add interruptible override to launch forms

Bugfix ensure that setups with different domains for console and admin API properly works

Bugfix Fixes one of the crashes in Graph component for Viel All Workflow section

Bugfix ensure that admin version is fully shown FC#465

[Bugfix] Small cosmetic updates: https://github.com/flyteorg/flyteconsole/issues/451, https://github.com/flyteorg/flyteconsole/issues/386, and https://github.com/flyteorg/flyteconsole/issues/398

Feature Map Tasks - allow to check statuses of all retries per child task

Feature Allow to rerun single task in execution page

[Plugins]: Adds flyte-api plugin package. You can check basic how to info and consume package from https://www.npmjs.com/package/@flyteconsole/flyte-api . It will allow you to authorize you flytectl queries and perform api requests from your app. At this point only request without included data in body is allowed. More work is planned in future updates.

Flytekit

Feature pyflyte run should support executing tasks

Bugfix Dot separated python packages does not work for pyflyte

Bugfix Pyflyte run doesn't respect the --config flag

Bugfix Read packages from environment variables

Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(353.23 KB)
flyte_gcp_manifest.yaml(352.43 KB)
flyte_sandbox_manifest.yaml(160.69 KB)
flyte_test_manifest.yaml(25.75 KB)
v1.0.2-b2(May 26, 2022)
Flyte 1.0.2-b2

Platform

Bugfix fix flyte-deps helm chart version

Bugfix Re-order clusterresourcesync annotations in helm chart

Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(353.23 KB)
flyte_gcp_manifest.yaml(352.43 KB)
flyte_sandbox_manifest.yaml(160.69 KB)
flyte_test_manifest.yaml(25.75 KB)
v1.0.2-b1(May 25, 2022)
Flyte v1.0.2-b1 Changelog

Platform

Feature Server-side compiler should strip Type Metadata

Bugfix With GRPC v1.46.0 non-ascii chars are not permitted in grpc metadata

Housekeeping Configure grpc_health_prob in admin

Feature In Flytectl use Launchplan with latest version for scheduled workflows

Bugfix Pods started before InjectFinalizer is disabled are never deleted

Housekeeping Checksum grpc_health_probe

Feature Allow to choose Spot Instances at workflow start time

Feature Use the same pod annotation formatting in syncresources cronjob

Flytekit

Feature pyflyte run should support executing tasks

Bugfix Dot separated python packages does not work for pyflyte

Bugfix Pyflyte run doesn't respect the --config flag

Bugfix Read packages from environment variables

Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(353.23 KB)
flyte_gcp_manifest.yaml(352.43 KB)
flyte_sandbox_manifest.yaml(160.69 KB)
flyte_test_manifest.yaml(25.75 KB)
v1.0.1(May 6, 2022)
v1.0.1 Release ChangeLog

System

Bugfix Addresses the regression to allow empty default complex values for workflow parameters

Bugfix for flyteadmin not respecting fallback substitution for execution spec parameters (e.g. labels, annotations, security context, raw output data config)

Bugfix Bump stow version to fix issue with GoogleAccessId

Flytekit

Bugfix Task chaining without the use of the create_node utility function

Bugfix Improved validation of image config objects

Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(353.22 KB)
flyte_gcp_manifest.yaml(352.43 KB)
flyte_sandbox_manifest.yaml(160.68 KB)
flyte_test_manifest.yaml(25.75 KB)
v1.0.0(Apr 26, 2022)
Flyte v1.0.0

Flyte 1.0 represents the first major release of the platform. Flyte APIs, and tools can now be considered [stable]. While it's impossible to assert so, any workflow/task written against flytekit v1.0.0 should be expected to run against all v1.x.x versions.

Quick Stats: | Stat | Change | |--------|----------------| | 4,125 | PRs Merged | | 18,983 | Comments Added | | 1,511 | Issues Created |

ChangeLog

FlyteKit

File ignore in fast register Many thanks to @bimtauer for adding this much requested feature! When packaging local files for fast-registration, flytekit will now respect .gitignore and .dockerignore. Please see the PR for more information.

pyflyte run The pyflyte run command has been slightly updated to not need a : when selecting the workflow.
$ pyflyte run --remote example.py wf --n 500 --mean 42 --sigma 2

Script mode: register and run workflows all in one command using a pre-defined base image

Flyte remote GA: register workflows and interact with Flyte execution artifacts programmmatically

Configuration overhaul: use the same config across flytekit and flytectl

Fast register without having AWS/GCP or other cloud credentials on your laptop, all you need is Flyte access

Core Platform

Improved Garbage Collector Garbage collection logic has been revamped to reduce load on KubeAPI and ensure terminated workflows are cleaned up in a timely fashion. [Action Required] Due to the change in the computation of how to clean up terminated workflows, users are advised to clean up old workflows once by running the following command per namespace:
kubectl delete fly -l termination-status=terminated --all-namespaces --cascade='background' --wait=false --force --grace-period=0

Single binary: deploy the entire Flyte back-end as a single binary. This speeds up sandbox and improves the local contributor experience. Coming soon: faster deployment for small-scale use-cases

Improved map task subtask handling

Cache status reporting

Individual log links mapped to subtasks

Interruptible failure handling for spot instances

Secret injection

Improved performance for fetching and rendering dynamic nodes

Set raw output data config at create execution time

Execution overrides at the project level for

Kubernetes service account

AssumableIAMRole

OutputLocationPrefix

Console

Project dashboard page with recent executions overview along with a config and other settings summary

Dynamic workflow rendering

Map task UI and UX improvements: see logs, retry attempts and more at the subtask level

Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(353.22 KB)
flyte_gcp_manifest.yaml(352.43 KB)
flyte_sandbox_manifest.yaml(160.68 KB)
flyte_test_manifest.yaml(25.75 KB)
v1.0.0-b1(Apr 21, 2022)

Flyte v1.0.0-b1 Changelog

Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(353.24 KB)
flyte_gcp_manifest.yaml(352.45 KB)
flyte_sandbox_manifest.yaml(160.71 KB)
flyte_test_manifest.yaml(25.76 KB)
v0.19.4(Apr 15, 2022)
Flyte v0.19.4 Changelog

Core Platform

Single binary: deploy the entire Flyte back-end as a single binary. This speeds up sandbox and improves the local contributor experience. Coming soon: faster deployment for small-scale use-cases

Improved map task subtask handling

Cache status reporting

Individual log links mapped to subtasks

Interruptible failure handling for spot instances

Secret injection

Improved performance for fetching and rendering dynamic nodes

Set raw output data config at create execution time

Execution overrides at the project level for

Kubernetes service account

AssumableIAMRole

OutputLocationPrefix

SDK

Script mode: register and run workflows all in one command using a pre-defined base image

Flyte remote GA: register workflows and interact with Flyte execution artifacts programmmatically

Configuration overhaul: use the same config across flytekit and flytectl

Fast register without having AWS or other cloud credentials on your laptop, all you need is Flyte access

Console

Project dashboard page with recent executions overview along with a config and other settings summary

Dynamic workflow rendering

Map task UI and UX improvements: see logs, retry attempts and more at the subtask level

Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(353.24 KB)
flyte_gcp_manifest.yaml(352.45 KB)
flyte_sandbox_manifest.yaml(160.71 KB)
flyte_test_manifest.yaml(25.76 KB)
v0.19.3(Mar 10, 2022)
Flyte v0.19.3 Changelog

Updates

Core Platform

Support for Type Annotations complete.

BigQuery plugin

Support for hash-based caching of schema and structured dataset offloaded types.

Configurable gRPC message size.

Cleanup of database connection handling in Admin.

Bug and performance fixes around

SDK

New current_task_metadata field in StructuredDatasetDecoder interface.

Authentication configs removed (auto-detected from Admin)

AWS Batch Task

Callable fetched entities

See the complete notes for flytekit v0.31.0. Keep in mind we'll be releasing v0.32.0 in a couple days. At that time the StructuredDataset type will be the default Flyte type for dataframes!

UI

Fix node IDs for certain nested scenarios where node IDs are shared across nested workflows.

Component Versions

Admin v0.6.92 to v0.6.112 Propeller v0.16.19 to v0.16.32 Console v0.41.0 to v0.43.0
Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(353.00 KB)
flyte_gcp_manifest.yaml(352.20 KB)
flyte_sandbox_manifest.yaml(160.43 KB)
flyte_test_manifest.yaml(25.49 KB)
v0.19.3-b3(Feb 18, 2022)
0.19.3-b3 Change Log

System

Properly control Athena token length in plugin

Relaxed StructuredDataset column compatibility

Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(353.00 KB)
flyte_gcp_manifest.yaml(352.20 KB)
flyte_sandbox_manifest.yaml(160.43 KB)
flyte_test_manifest.yaml(25.49 KB)
v0.19.3-b2(Feb 16, 2022)
0.19.3-b2 Change Log

System

Allow clusterName to be passed to flytepropeller through the Values.yaml file.

Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(352.99 KB)
flyte_gcp_manifest.yaml(352.20 KB)
flyte_sandbox_manifest.yaml(160.42 KB)
flyte_test_manifest.yaml(25.48 KB)
v0.19.3-b1(Feb 15, 2022)
0.19.3-b1 Change Log

System

Allow extraArgs to be passed to FlytePropeller, DataCatalog and FlyteAdmin deployments.

Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(352.99 KB)
flyte_gcp_manifest.yaml(352.20 KB)
flyte_sandbox_manifest.yaml(160.42 KB)
flyte_test_manifest.yaml(25.48 KB)
v0.19.2(Feb 5, 2022)
0.19.2 Change Log

UX

Added support for Archive/Unarchive Workflow Executions

Added Support for viewing node metadata on unexecuted nodes

Minor bug fixes

FlyteKit

Introducing the new StructuredDataset type. Turn it on by setting the env var FLYTE_SDK_USE_STRUCTURED_DATASET=TRUE. It'll be turned on by default in v1.0. Docs

This release also removes the legacy API.

Review the full changelog here.

System

Archive (issue). Expose the ability to archive/unarchive executions to remove clutter from the UI.

Security Updates:

Access token validation supports multiple-audience tokens (issue)

flytectl supports caching access tokens for multiple endpoints at the same time (issue)

You can use flytectl to update projects' display names and descriptions (issue)
flytectl update project --id data --name datadata --description "Team that manage Data Platform" --labels "team=data,managedby=flyte"

Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(352.99 KB)
flyte_gcp_manifest.yaml(352.20 KB)
flyte_sandbox_manifest.yaml(160.42 KB)
flyte_test_manifest.yaml(25.48 KB)
v0.19.2-b5(Jan 29, 2022)
0.19.2-b5 Release ChangeLog

System

Mount admin auth creds for cluster resource sync run in standalone mode.

Source code(tar.gz)
Source code(zip)
flyte_eks_manifest.yaml(352.99 KB)
flyte_gcp_manifest.yaml(352.20 KB)
flyte_sandbox_manifest.yaml(160.42 KB)
flyte_test_manifest.yaml(25.48 KB)

Kubernetes-native workflow automation platform for complex, mission-critical data and ML processes at scale. It has been battle-tested at Lyft, Spotify, Freenome, and others and is truly open-source.

Related tags

Overview

Flyte

Home Page · Quick Start · Documentation · Live Roadmap · RFC's & Proposals · Features · Community & Resources · Changelogs · Components

💥 Introduction

⏳ Five Reasons to Use Flyte

🚀 Quick Start

⭐️ Current Deployments & Contributors

🛣️ Live Roadmap

🔥 Features

🔌 Available Plugins

📦 Component Repos

Functional Tests Matrix

🛣️ RFC's (Request for Commments) & Proposals

🤝 Community & Resources

Communication Channels

Biweekly Community Sync

Blog Posts

Newsletter

Conference Talks & Pod Casts

Conference

Podcasts

💖 All Contributors

Comments

Why

Describe the bug

Expected behavior

Additional context to reproduce

Screenshots

Are you sure this issue hasn't been raised already?

Have you read the Code of Conduct?

Description

Are you sure this issue hasn't been raised already?

Have you read the Code of Conduct?

Motivation: Why do you think this is important?

Goal: What should the final outcome look like, ideally?

Describe alternatives you've considered

Propose: Link/Inline OR Additional context

Are you sure this issue hasn't been raised already?

Have you read the Code of Conduct?

Motivation: Why do you think this is important?

Goal: What should the final outcome look like, ideally?

Describe alternatives you've considered

Propose: Link/Inline OR Additional context

Are you sure this issue hasn't been raised already?

Have you read the Code of Conduct?

Motivation: Why do you think this is important?

Goal: What should the final outcome look like, ideally?

Describe alternatives you've considered

Propose: Link/Inline OR Additional context

Are you sure this issue hasn't been raised already?

Have you read the Code of Conduct?

Motivation: Why do you think this is important?

Goal: What should the final outcome look like, ideally?

Describe alternatives you've considered

Propose: Link/Inline OR Additional context

Are you sure this issue hasn't been raised already?

Have you read the Code of Conduct?

Describe the bug

Expected behavior

Additional context to reproduce

Screenshots

Are you sure this issue hasn't been raised already?

Have you read the Code of Conduct?

Releases(v1.3.0-b8)

v1.3.0-b8(Dec 30, 2022)

v1.3.0-b8

v1.3.0-b7(Dec 29, 2022)

v1.3.0-b7

v1.3.0-b6(Dec 29, 2022)

Flyte v1.3.0-b6 Changelog

v1.3.0-b5(Dec 20, 2022)

Flyte v1.3.0-b5 Changelog

v1.3.0-b4(Dec 15, 2022)

Flyte v1.3.0-b4 Changelog

v1.3.0-b3(Dec 7, 2022)

Flyte v1.3.0-b3 Changelog

v1.2.1(Nov 17, 2022)

Flyte v1.2.1 Release