As teams combine hosted and self-hosted LLMs, agents that speak a single provider API face brittle integrations and costly provider lock‑in. Switchyard attacks that problem horizontally: it sits between clients and model backends, translates protocols, and routes turns to different targets according to pluggable algorithms so an agent can keep speaking its native API while traffic is split across many backends.
What Sets It Apart
- Protocol translation as first-class behavior — accepts OpenAI Chat, OpenAI Responses, and Anthropic Messages and converts requests/responses into each backend's native shape, so an existing agent needs no protocol changes.
- Typed, composable routing algorithms — supports random splits, LLM-as-classifier routing, stage-driven routing, and escalation modes; so you can implement cost tiers, A/B tests, or signal-driven escalation without changing client logic.
- Three deployment surfaces — launcher (integrates coding agents), standalone Rust proxy server, and an embeddable library for Rust apps; so teams can adopt it as a local proxy, a service, or embed routing into existing runtimes.
- Observability and operational controls — exports Prometheus metrics for requests, errors, latency, tokens, and routing overhead, enabling cost and reliability monitoring across heterogeneous backends.
Who It's For and Trade-offs
Great fit if you need to operate an agent or API gateway that must talk one provider API while you experiment with multiple backends (vLLM, NVIDIA NIM, Ollama, OpenAI-compatible endpoints) or run tiered/cost-aware routing. It helps teams run A/B benchmarks, implement weak/strong-tier escalation, or centralize provider translation.
Look elsewhere if you need a production-hardened, stable service today: Switchyard is explicitly pre-alpha and rapidly evolving, not intended for critical production traffic. Also, if you require non-Rust integrations that cannot call an external proxy or embed a Rust library, adoption will require additional integration work.
Where It Fits
Use Switchyard as the control plane for multi-backend LLM deployments or as a drop-in proxy for coding agents that assume a single provider API. Combine it with an LLM client that owns credentials and dispatch, or pair with switchyard-llm-client to let Switchyard make upstream calls for you.
