Serving a reasoning model is two jobs pretending to be one: prefill is compute-bound, decode is memory-bound, and running both on the same GPU wastes whichever resource the current phase isn't using. Dynamo's bet is that you should physically separate them — distinct GPU pools for prefill and decode, each free to pick its own parallelism strategy — and let a smart layer stitch the request back together across the network.
Key Capabilities
- Disaggregated prefill/decode: independent GPU pools per phase let each be tuned and scaled separately, instead of one compromise config; this is where the >2x Llama-70B-on-Hopper and up to 30x DeepSeek-R1-on-GB200-NVL72 gains come from.
- KV-aware Smart Router: tracks KV cache across the fleet via a radix tree and routes requests to where the cache already lives, so prefixes aren't recomputed.
- Distributed KV Cache Manager: spills cold KV blocks to CPU memory, SSD, or networked storage and keeps hot data on GPU, reclaiming HBM for active work.
- NIXL transfer library + SLO Planner: low-latency point-to-point GPU data movement plus a planner that switches between disaggregated and traditional serving based on live SLO metrics.
Great Fit / Look Elsewhere
Great fit if you run reasoning or MoE models at multi-node scale and your bottleneck is GPU utilization under bursty traffic — the engine-agnostic design (TensorRT-LLM, vLLM, SGLang, PyTorch) means you keep your existing backend. Look elsewhere if you serve a single small model on one or two GPUs: disaggregation and a distributed router add operational overhead that only pays off across a fleet.
