Keyframe-conditioned workflows are becoming a standard way for creators to steer video generation, but it's unclear whether current models can both faithfully execute prescribed keyframes and produce natural, temporally coherent videos. KeyFrame-Compass tackles this gap by providing a controlled benchmark and an automated evaluation suite that makes the trade-offs explicit and measurable.
Key Findings
- Models show a clear trade-off between faithful keyframe execution and natural video synthesis — improving one often harms the other, so model choice depends on whether fidelity or realism is the priority.
- Performance degrades as keyframe density increases, indicating current systems struggle to satisfy many tight constraints across time.
- Most open-source models fail to interpret storyboard-grid inputs as temporally ordered sequences, revealing a usability gap for common authoring formats.
- The benchmark exposes varied failure modes (presence, fidelity, ordering, localization, persistence, uniqueness), enabling targeted improvement rather than coarse, single-number comparisons.
Who It's For and Tradeoffs
Great fit if you evaluate or develop video generation models that must follow visual references (researchers, model developers, dataset curators). The benchmark is useful when you need fine-grained diagnostics of how well models realize keyframes under varying prompt granularity, conditioning format, and keyframe density. Look elsewhere if you only need generic perceptual quality scores or single-frame image metrics — KeyFrame-Compass focuses on temporal execution and reference compliance rather than purely aesthetic scoring.
Methodology
The benchmark contains 386 curated samples spanning three application domains, two video structures, two prompt granularities, two conditioning formats, and four keyframe densities to enable controlled comparisons. Keyframe execution is decomposed into six complementary automated metrics: presence, fidelity, temporal ordering, localization, persistence, and uniqueness. Overall video quality is assessed via evidence-grounded MLLM judgments augmented with specialized perception models, allowing joint measurement of constraint adherence and synthesis quality. Experiments on nine representative systems reveal systematic limitations and provide actionable axes for future model improvements.