AIAny
Icon for item

WeMM-Video: Pushing the Frontiers of Open-Source Omni-Modal Video Understanding

Unifies text, images, audio, and video for Chinese-English multimodal understanding and reasoning, with fast instruct and explicit thinking modes. The paper reports leading open-source results on several omni-modal video benchmarks.

Introduction

Many video models can describe frames, but understanding what happened often depends on connecting sound to the right moment and speaker. This work makes joint audio-visual reasoning its central test, rather than treating video as a sequence of images with optional transcripts. Its benchmark results are promising, though they remain claims from the model's authors until independently reproduced.

Key Findings
  • The 2B, 4B, 9B, and 35B-A3B variants share a bilingual, multimodal design and support both direct responses and explicit step-by-step reasoning. The range makes it possible to compare capability across model sizes.
  • Training uses approximately 1.2 trillion multimodal tokens and a staged post-training pipeline. The scale signals a substantial training effort, but the abstract does not isolate which training component contributes most to performance.
  • Across more than 120 benchmarks, the authors report strong audio-video understanding and competitive results on audio, silent video, images, and text. They claim the 35B-A3B model sets a new open-source high on omni-modal video tasks, while the 9B version surpasses Qwen3-Omni-30B-A3B-Instruct in their evaluations.
  • The paper also releases evaluation and reproduction materials, an important step for testing results across different video sampling, audio, and reasoning conditions.
Who Should Read It

A good fit for researchers comparing open multimodal models, especially on tasks where audio and video must be reasoned about together. Treat the reported rankings as a starting point, not a settled leaderboard: the available results are author-reported, and independent replication is still needed. Teams seeking established production performance or independently verified deployment costs should wait for broader external testing.

Information

  • Websitearxiv.org
  • OrganizationsTencent
  • AuthorsChangli Tang, Jie Yang, Tianyi Wang, Lei Li, Daoze Zhang, Zijie Xin, Zhan Liu, Li Li, Jinchang Xu, Fengyun Rao …
  • Published date2026/11/19

More Items

Tests multimodal models’ spatial reasoning through screen-based clicks and drags. Its programmatically generated CAPTCHA tasks also form a training environment, with fine-tuning gains reported across seven out-of-distribution benchmarks.

Coordinates visual and audio forensic agents to locate and verify signs of AI-generated video. Its dual-timescale training improves both the investigation strategy and the specialist inspectors.

A rendered object mask can look complete even when the selected 3D Gaussians omit occluded parts. VODA uses local depth clusters to supervise those Gaussians directly, and Removal-3D evaluates whether their removal matches the object’s expected 3D removal.