Production LLM serving engines like vLLM and the full SGLang have grown into tens of thousands of lines, and their cleverest optimizations sit buried under layers of dispatch logic. Mini-SGLang takes the opposite bet: strip the same engine down to ~5,000 readable, fully type-annotated lines so you can actually trace how a request turns into tokens — without giving up state-of-the-art throughput.
What Sets It Apart
- It keeps the optimizations that matter, not a toy subset: radix-tree KV-cache reuse across shared prefixes, chunked prefill to cap peak memory on long contexts, overlap scheduling that hides CPU work behind GPU compute, and tensor parallelism for multi-GPU. These are the four things that separate a real serving engine from a for-loop over
model.generate(). - It wires in FlashAttention and FlashInfer kernels rather than reimplementing attention, so "mini" applies to the scheduler and runtime, not to the math.
- Because it derives directly from the SGLang codebase, the abstractions you learn here map onto the production system — it doubles as a reading guide for SGLang, not just a standalone engine.
Who It's For
Great fit if you're a systems researcher, a student, or an engineer who wants to understand modern inference internals — KV caching, scheduling, parallelism — by reading code that fits in your head, or who needs a hackable base for prototyping serving ideas. Look elsewhere if you need a battle-tested deployment with broad model coverage and a mature ecosystem; for that, run full SGLang or vLLM. It's Linux-only (x86_64/aarch64), with macOS and Windows reached through WSL2 or Docker.
