Most modern video VLMs rely on dense, uniform frame sampling and large pretrained image ViTs, which makes real‑time streaming perception slow and costly. This project flips that tradeoff by aligning visual tokenization with video codec structure and adding a lightweight, learned event gate: it keeps full anchor (I) patches but selects only motion‑salient predicted (P) patches, drastically lowering token counts while preserving spatio‑temporal context.
Key Capabilities
- Codec‑native sparse tokenization: uses I/P frame structure (16×16 patch grid) to retain anchor patches and only predicted-frame patches where the codec allocates bits, reducing visual tokens by over 75% (≈1/8 of dense sampling). So what: much lower compute per second of video while keeping temporal detail where it matters.
- Unified visual+language stack: pairs a from‑scratch Codec‑ViT visual encoder with a frozen Qwen3‑4B causal decoder via a lightweight projector, enabling images, short/long/ultra‑long video, and streaming within one checkpoint. So what: a single model handles offline reasoning and event‑gated commentary without separate variants.
- Proactive streaming gate: a small cognition gate scores rolling codec windows and triggers generation only on response‑worthy events, enabling low‑latency, event‑gated outputs in a single model pipeline. So what: fewer false positives and lower bandwidth/compute in continuous deployments.
- Training & data efficiency: variable‑resolution pretraining and codec alignment let the visual encoder train effectively on hundreds of millions (not billions) of images/frames while scaling monotonically with token budget; reported gains over comparable models are largest on localization and temporal tasks.
Who it's for & Trade‑offs
Great fit if you need real‑time or long‑horizon video understanding with constrained compute or streaming bandwidth, and you can integrate codec‑aware preprocessing into your pipeline. It is practical for research groups and engineering teams that want one unified checkpoint for image/video QA, temporal grounding, and event‑gated commentary.
Look elsewhere if you require a plug‑and‑play web UI or an out‑of‑the‑box image classifier with minimal engineering: codec pipelines, trust‑remote‑code model loading, and specialized inference settings (codec engines, gate thresholds, streaming server) add integration complexity. Also, as with many cutting‑edge VLMs, reproducible deployment requires attention to dependencies (ffmpeg, neural codec libs) and adequate GPU memory/throughput to realize the reported speedups.
Where it fits
Compared to dense frame sampling + pretrained ViTs, this approach favors accuracy–efficiency for temporal and localization tasks by moving compute to codec‑informed regions of interest. Compared to other multimodal models built on the same Qwen backbone, the main differentiator is the codec‑native visual tokenization and the single‑checkpoint proactive streaming gate.
Practical notes
- Single checkpoint covers image understanding, frame‑sampled video, traditional codec (H.264/HEVC), neural codec (DCVC‑RT), and streaming gate, with example inference scripts and a small pretrained gate weight included.
- Useful controls for deployment include gate threshold, codec backend choice, segment length, and attention implementation; careful tuning determines latency vs. recall in streaming use cases.
