When a chatbot's traffic explodes, the instinct is to buy more GPUs. Mooncake's bet is the opposite: trade cheap, idle storage for expensive compute. By treating accumulated KVCache as a first-class, reusable asset rather than per-request scratch space, it turns the GPU cluster's underused CPU, DRAM, and SSD into a shared cache pool — so repeated prefixes are recomputed far less often.
Key Findings
- Splitting prefill and decode into separate clusters lets each scale and schedule independently, instead of fighting over the same GPU between two workloads with very different latency profiles.
- A KVCache-centric scheduler balances cache reuse against latency SLOs under real overload, rather than just maximizing raw throughput — the difference between a benchmark win and surviving a traffic spike.
- In Moonshot AI's production Kimi service it absorbed roughly 75% more requests under the same SLOs, and reported up to a 525% throughput gain in simulated long-context scenarios.
- The Transfer Engine (RDMA) and Mooncake Store are open-sourced and already wired into vLLM, SGLang, LMDeploy, and LMCache, so the ideas are reusable outside Kimi.
Who It's For
Great fit if you run a high-traffic, long-context inference service and are bottlenecked on prefill recomputation or GPU cost, and you can invest in RDMA networking and tiered storage to make a disaggregated KVCache pay off. Look elsewhere if your workload is low-volume, short-context, or single-node — the operational complexity of separate prefill/decode clusters and a distributed cache tier will outweigh the savings. The work won Best Paper at FAST 2025, so the design is well-documented for teams wanting to study rather than deploy it.
