Most recent video generators prioritize visual fidelity and temporal coherence, but they rarely get evaluated on whether a generated clip actually achieves a specified outcome while staying semantically grounded in a reference image. This paper reframes the problem as Semantic Task Completion Video Generation and provides both data and a repeatable, interpretable evaluation protocol to measure that capability.
Key Findings
- A focused evaluation target: the benchmark separates outcome achievement from low-level appearance fidelity and emphasizes whether the generated outcome matches the instructed goal and preserves task-relevant semantics from the reference. So what? Models can look realistic yet fail the task; this benchmark exposes that gap.
- SemComp-Data design: constructs image–instruction–outcome triplets by mining full-context real videos and uses a four-stage curation pipeline (candidate filtering, state mining, video extension, instruction structuring). So what? Tasks are authentically achievable for each instance and preserve fine-grained task-relevant alignment.
- VLM-based, evidence-grounded scoring: SemComp-Bench poses structured binary questions to a vision–language model and reports OA (Outcome Achievement) and GR (Generation Reliability) scores, with criterion-level pass rates for interpretable failure diagnosis. So what? This makes automated evaluation actionable and diagnostic rather than a single opaque metric.
- Empirical gap: evaluations on representative video generators show substantial failures in completing instructed tasks while preserving reference-grounded semantics. So what? Progress in visual fidelity does not imply task competence; targeted research on outcome grounding is needed.
Who it's for and tradeoffs
Great fit if you evaluate or train video generation models where the objective is to realize user-specified outcomes (e.g., instruction-driven editing, simulation of object state changes). The benchmark is most useful for diagnostics and dataset-driven improvements rather than measuring pure perceptual quality. Look elsewhere if your primary concern is unconditional aesthetic quality, frame-level temporal realism without task semantics, or short synthetic clips that lack real-world task feasibility evidence.