vLLM solved one problem brilliantly — fast, memory-efficient autoregressive text generation — and that very specialization became a wall. Audio, image, and video models increasingly run on non-autoregressive backbones like Diffusion Transformers, which the original engine's KV-cache machinery simply doesn't fit. This project is vLLM's answer: a sibling engine that keeps the serving strengths while opening the door to omni-modality.
What Sets It Apart
- One serving stack for radically different model families: autoregressive LLMs, omni models (Qwen3-Omni, Cosmos), TTS (Qwen3-TTS, CosyVoice3), and diffusion generators — so you don't stitch together a separate runtime per modality.
- A heterogeneous pipeline abstraction lets a single request flow through mixed AR and non-AR stages, which is what real multimodal generation (text → speech, prompt → video) actually demands.
- Inherits vLLM's KV-cache management and tensor/pipeline/data/expert parallelism, so multimodal serving starts from a battle-tested distributed-inference base rather than a research prototype.
- Keeps the OpenAI-compatible API and streaming output, so existing vLLM clients can point at new modalities with minimal change.
Who It's For
Great fit if you need to deploy generation across modalities behind one OpenAI-style endpoint, or you already run vLLM and want speech, image, or video without adopting a second framework. Look elsewhere if you only serve text LLMs — core vLLM is leaner for that — or if your target model isn't in the supported set yet, since coverage is growing but still narrower than vLLM's mature text catalog.
