Most ML pipeline tools force a tradeoff: a clean local dev loop or production-grade durability, rarely both. Flyte's bet is that the orchestrator should be a Kubernetes-native runtime where the workflow is just Python — the same code you test on your laptop is the code that runs, retries, and resumes across a cluster. That approach came out of Lyft, which ran pricing, ETA, and mapping on it before open-sourcing it in 2020.
What Sets It Apart
- Workflows are real Python, not a DSL. You compose tasks as typed functions, so versioning, caching, and reproducibility become mechanical rather than aspirational — a cache hit skips recomputation by input hash.
- Durable by default. Long-running jobs survive node failures: retries, fault tolerance, and resumption from the failure point are first-class, which matters when a 6-hour training run dies at hour 5.
- Infra-aware orchestration. Tasks declare their resource needs and Flyte provisions and autoscales accordingly, so a heavy GPU step and a cheap glue step coexist in one graph.
适合谁
Great fit if you run ML or data pipelines on Kubernetes and want strong typing, caching, and reproducibility without inventing your own scheduler. Look elsewhere if you have no Kubernetes footprint or just need lightweight cron-style task chaining — the operational surface is heavier than a simple DAG tool. Note that the open-source core targets self-hosted clusters; managed scale and sub-second inference paths come through Union.ai, the company Flyte's core team founded after Lyft open-sourced the project.
