Category

Apache Airflow

Programmatically author, schedule, and monitor data workflows as Python-defined DAGs; the scheduler handles dependencies, retries, and backfills. Pluggable executors (Local, Celery, Kubernetes) and a broad provider ecosystem for AWS, GCP, and databases.

vn.py (VeighNa)

End-to-end quantitative trading framework that covers feature engineering, model training, signal generation and live execution; includes vnpy.alpha for ML-driven strategy research and workflow management, aimed at production trading and research pipelines.

TensorFlow

Builds and deploys machine learning models across research, production, web, mobile, and edge environments. Its ecosystem spans Keras, TFX, LiteRT, TensorFlow.js, datasets, model hubs, and visualization tools.

Weights & Biases

Tracks every ML run — hyperparameters, metrics, checkpoints, dataset versions — into one dashboard you share as a live report, with Sweeps for tuning and a model registry. Weave extends it to LLM apps: tracing, evals, and production monitoring.

great_expectations

Expresses data quality checks as reusable, declarative "expectations" and auto-generates human-readable validation reports and docs; integrates with Python data stacks to enforce and monitor data reliability in ML and analytics pipelines.

ONNX

Defines a portable model format and operator set for moving trained machine learning models across frameworks, runtimes, and hardware targets without locking the model to one toolchain.

Ray (by Anyscale)

Scales any Python or ML workload across CPUs and GPUs with a few decorators, instead of rewriting code for Spark or MPI. Bundles libraries for distributed training, hyperparameter tuning, RL, batch inference, and online model serving on one cluster.

Streamlit

Turns a top-to-bottom Python script into an interactive web app: each widget interaction reruns the whole script, with cache decorators skipping redundant work. No callbacks or HTML needed; built for data dashboards, ML demos, and internal tools.

MLflow

Tracks ML and LLM experiments end to end: logs params, metrics, and artifacts, versions models in a registry, and records agent traces via OpenTelemetry. Framework-agnostic, runs locally or self-hosted, with 50+ built-in evaluation metrics and LLM judges.

NautilusTrader

Rust-native, event-driven trading platform for backtesting and live execution across crypto, forex, equities, and futures on 27+ venues. The same strategy code runs in nanosecond backtests and in production, giving true research-to-live parity.

Prefect

Orchestrates and schedules Python data pipelines and workflows with primitives for retries, caching, parameters, and deployments. Provides either a self-hosted server or managed Prefect Cloud for monitoring, observability, and integrations across common data tools.

MLRun

Turns plain Python functions into versioned, serverless ML jobs that run unchanged locally or on Kubernetes, with built-in tracking and deployment. Its feature store derives both offline (batch) and online (real-time) serving from one definition.