Most video-language checkpoints assume an offline workflow: wait until a full video is available, then reason. That approach fails for live cameras, screen shares, and interactive agents where questions can arrive at any moment and correct answers can change as the scene evolves. MOSS-VL-Realtime targets that gap by treating video as a continuous, timestamped stream and by making the model aware of absolute time so it can decide when to speak, stay silent, or revise.
Key Capabilities
- Realtime streaming inference: ingests PIL-compatible frames with absolute timestamps and integrates visual tokens into a running generation context so the model can respond to prompts before a full clip exists. This enables live descriptions, monitoring, and on-the-fly question answering.
- Interruptible interaction and proactive silence: users can query at arbitrary moments; the model may output a special silence token when evidence is insufficient, avoiding hallucinated responses and reducing noisy updates.
- Dynamic correction: as new frames arrive the model can amend earlier answers rather than being locked to an initial interpretation, which is crucial for tasks where later context disambiguates earlier frames.
- Timestamp-aware positional encoding (XRoPE) and cross-attention decoupling: timestamps are injected alongside visual patches to reason about order, duration, and pacing; vision encoding is decoupled from language reasoning to allow incremental integration of frames without full offline buffering.
- Practical defaults for streaming: 11B parameters, 256K text context, default 1 FPS sampling and a bounded frame queue; designed to balance context depth with realtime constraints.
Who it's for and tradeoffs
Great fit if you build live multimodal apps (surveillance analytics, assistive agents, live-captioning, interactive demos) that need answers or summaries while video is still arriving. It is also useful for research into streaming benchmarks, proactive silence strategies, and dynamic response updates. Look elsewhere if you need ultra-high-frame-rate video analytics, multi-session concurrency from a single instance, or minimal-latency production at massive scale without GPU optimizations: a single model instance supports one realtime session, latency depends on hardware, and the default design drops older pending frames when the frame queue bounds are reached. The model also emits realtime control tokens that downstream services must handle.
Where it fits
Positioned between offline video-VL checkpoints (which favor batch accuracy) and lightweight event-detectors (which favor latency). MOSS-VL-Realtime is a research-to-deployment checkpoint for applications that value temporal correctness, the ability to defer answering, and progressive correction over immediately committing to possibly wrong outputs.
