Mooncake

Distributed KV-cache store & transfer engine that decouples prefilling from decoding to scale vLLM serving clusters.

Visit Website

Introduction

Overview

Mooncake moves KV tensors across GPUs or nodes so that multiple inference servers can share prefilling work and latency.

Key Capabilities

Trace-based prefill disaggregation
P2P store & vLLM integration
Transfer-engine plug-in architecture

Back

Information

Websitekvcache-ai.github.io
AuthorsKVCache-AI Team
Published date2024/11/28

More Items

Ray

2017

RISELab (UC Berkeley), Anyscale Inc.

Ray is an open-source distributed compute engine that lets you scale Python and AI workloads—from data processing to model training and serving—without deep distributed-systems expertise.

ai-development ai-framework ai-train ai-serving

OpenVINO

2018

Intel

OpenVINO is an open-source toolkit from Intel that streamlines the optimization and deployment of AI inference models across a wide range of Intel® hardware.

ai-development ai-inference ai-serving

NVIDIA Dynamo

2025

NVIDIA

NVIDIA Dynamo is an open-source, high-throughput, low-latency inference framework that scales generative-AI and reasoning models across large, multi-node GPU clusters.

ai-development ai-inference ai-serving nvidia