Every long prompt your LLM has already processed gets recomputed from scratch the next time it shows up — the same RAG documents, the same system prompt, the same conversation history, burned again on the GPU. LMCache's bet is that the key-value cache produced during that work is reusable knowledge, not disposable per-request state, and should live wherever there is cheap memory: GPU, CPU RAM, local disk, or a shared remote store.
What Sets It Apart
- Non-prefix reuse: unlike standard prefix caching, CacheBlend reuses cached chunks that appear anywhere in a prompt (say, the third retrieved doc in a RAG query) and recomputes only the few tokens needed to stitch them together — far more cache hits on real workloads.
- Engine-independent: it runs as a standalone layer in front of vLLM, SGLang, and other mainstream engines instead of being welded to one, so a single cache pool serves a whole fleet.
- Tiered, shared storage: KV caches offload across CPU, disk, Redis/Valkey, and S3-compatible backends, and survive an engine crash rather than vanishing with the process.
- Prefill/decode disaggregation: KV moves over NVLink, RDMA, or TCP, letting prefill and decode run on separate hardware. The payoff is claimed up to 7x faster KV access and large TTFT drops on long-context, multi-turn, and agentic serving.
Who It's For
Great fit if you run production LLM serving at scale where the same context — RAG corpora, long system prompts, multi-turn sessions — recurs and TTFT and throughput actually matter, especially across a multi-node vLLM/SGLang fleet. Look elsewhere if your traffic is short, one-off prompts with little repeated context, or you run a single small model: the extra moving parts (external KV store, tiered offload) add operational overhead without enough cache reuse to pay for them.
