LLM serving is no longer just wrapping a model behind an API. The bottleneck has moved into scheduling, prefix reuse, long-context memory, expert parallelism, and post-training rollout workloads.
What Sets It Apart
RadixAttention and cache-aware runtime design make repeated-prefix workloads first-class. Broad model and hardware coverage spans language, multimodal, embedding, reward, and diffusion workloads across GPUs, CPUs, TPUs, NPUs, and clusters.
Who Should Use It
Great fit if you operate LLM or multimodal serving at scale, need advanced batching and parallelism, or build RL/post-training infrastructure. Look elsewhere for a desktop model runner or tiny API wrapper.
