Most generative-video evaluations give models explicit instructions; this paper flips that setup: it withholds key event semantics and distributes the missing information across heterogeneous inputs, forcing an omni-modal generator to align partial cues and complete the scene via video output. The generated video then serves as a behavioral readout of whether the model truly recovered the latent event rather than merely following an explicit prompt.
Key Findings
- A focused benchmark of 517 expert-verified instances spanning four reasoning scenarios reveals a modest overall success rate of 41.97% for MiniMax-H3, showing substantial room for improvement when grounding generation in multimodal evidence.
- Performance varies by scenario: Video-based Decision Reasoning (VDR) performs best at 56.00%, Audiovisual Integrated Reasoning (AVIR) at 47.89%, Multi-view Spatial Reasoning (MSR) at 43.50%, while Audio-based Disambiguation Reasoning (ADR) is weakest at 27.40% — highlighting particular difficulty in resolving semantics from audio alone.
- Typical failures are semantically specific: inconsistent embodiment across views, wrong or missing camera motion, incomplete state transitions, lacking visual reactions to audio cues, and imprecise audiovisual localization. These indicate that visual plausibility alone does not guarantee physical-world consistency.
- The study introduces four complementary evaluation settings (multi-view, audio-image, prefix-video continuation, and audio-video integration) designed so each modality supplies only partial evidence; success requires cross-modal association, temporal inference, and audiovisual alignment.
Who it's for and trade-offs
Great fit if you need a diagnostic benchmark to probe multimodal grounding in generative video models or want concrete failure modes for research on cross-modal alignment and world-modeling. The dataset and protocol emphasize semantic correctness over pixel fidelity, so they expose reasoning weaknesses that conventional perceptual metrics miss. Look elsewhere if your goal is engineering high-fidelity short video synthesis without emphasis on multimodal inference: the paper targets reasoning evaluation rather than optimizing generation quality or latency.
Where it fits
This work sits at the intersection of multimodal generative models, video understanding, and evaluation methodology: it complements perceptual or user-study benchmarks by providing task-specific semantic success criteria that require joint interpretation of complementary modalities. The authors release the evaluation instances and an accompanying codebase to reproduce human-evaluation criteria.