Most large vision–language models are optimized for offline, high-compute visual reasoning but struggle with low-latency, continuous perception. The core insight behind this work is pragmatic: compress the perceptual input where video codecs already concentrate useful signal (motion vectors and residual energy across I/P frames) and build a streaming-aware model around that representation. That shift yields drastic token savings without throwing away the spatiotemporal context that downstream multimodal reasoning needs.
Key Findings
- Codec-native tokenization: replacing uniform frame sampling with a Mage-ViT tokenizer that selectively encodes entropy-rich 16×16 patches (guided by motion vectors and residuals across sparse anchor and predicted frames) cuts visual token consumption by >75%, so the model processes far less input per second while retaining temporal cues.
- Streaming architecture: a bio-inspired dual-system design pairs a lightweight System 1 event gate for quick, low-cost perception with a causal System 2 decoder for deliberate multimodal reasoning — this enables proactive streaming perception and lower end-to-end latency.
- Empirical trade-offs: trained from scratch on ~560M unlabeled images and ~100M unlabeled video frames, the model matches strong static-image encoders on image tasks, substantially improves video understanding and spatial (2D/3D) reasoning, and achieves up to ~3.5× wall-clock inference speedup versus baseline pipelines.
- Practical pipelines and findings: the paper documents AI4AI data pipelines (prompt-code joint optimization and AI-driven training diagnosis) and seven empirical findings covering data efficiency, variable-resolution scaling, codec acceleration, VideoQA SFT redundancy, motion–spatial synergy, and Zero-Vision SFT for multimodal RL, offering operational guidance beyond the core model.
Who it's for — and when not to choose it
Great fit if you need low-latency, long-duration or continuous multimodal perception (e.g., live video understanding, real-time agents, robotics sensing) and want to reduce token/compute load by leveraging codec signals. Look elsewhere if your primary workload is offline high-resolution single-image tasks where codec-derived sparsity brings little benefit, or if you require models trained on very large paired image–text corpora for niche generative captioning benchmarks.
Where it fits
This work sits between efficient video encoders and multimodal foundation models: it targets systems that must run continuous perception with tight latency/bandwidth constraints while still supporting rich multimodal reasoning, rather than bulk offline pretraining on massive image–text pairs.
Brief methodological note
The approach combines a codec-aware front end (motion/residual-guided token selection at patch granularity), large-scale self-supervised pretraining, and a dual-stage inference flow that routes fast events to a lightweight gate and escalates selected contexts to a causal decoder for reasoning. The evaluation compares static, video, and spatial reasoning tasks and includes throughput and wall-clock benchmarks to reflect real streaming costs.