Recent video generation models show emerging signs of visual reasoning, but standard evaluations often miss whether a model produces valid multi-step processes rather than only plausible final frames. VGI-Bench addresses this gap by providing a calibrated, photorealistic benchmark that stresses intermediate trajectory validity and fine-grained capability diagnosis.
Key Findings
- Benchmark design: 27 tasks and 810 instances organized by four mutually exclusive domains (Visual Organization, Physical Manipulation, Structured Puzzles, Spatiotemporal Dynamics) and annotated with non-exclusive skill tags (Spatial, Temporal, Planning, Attribute Grounding, Physics, Topology, Affordance). This structure enables coarse- and fine-grained diagnosis of model strengths and weaknesses.
- Evaluation behavior: Current video generation systems can partially solve some visually grounded reasoning tasks but struggle with coherent multi-step execution; top-performing model scored about 51.0% under the benchmark's criteria. Common failure modes include physical collapse, rule violations, and object/state inconsistency across frames.
- Design principles: The benchmark reduces domain gap by using photorealistic inputs, enforces intermediate-step validity (not just final-state plausibility), and calibrates difficulty through pre-generation filtering and human review to keep tasks challenging yet feasible for current models.
- Diagnostic insights: Performance is sensitive to input prompt/style; large-scale synthetic fine-tuning transfers only within covered skill distributions; denoising trajectories suggest models mostly refine early visual hypotheses with limited reliable self-correction later in generation.
Who it's for and trade-offs
Great fit if you need a targeted evaluation suite to measure multi-step visual reasoning in video generation systems, compare models across specific capability tags, or study failure modes tied to temporal and physical consistency. Look elsewhere if you only need single-frame realism metrics or large-scale generative diversity benchmarks—VGI-Bench prioritizes trajectory validity and reasoning diagnosis over unconstrained creative generation.
Where it fits
VGI-Bench complements pixel-level fidelity and frame-wise realism benchmarks by explicitly testing evolving processes and reasoning abilities. It is useful for model developers aiming to improve temporal consistency, planners or simulators that feed into video generators, and researchers studying how training data and denoising dynamics affect multi-step reasoning.