World-model rollouts demand judgments about physics, causality, and state evolution—things humans spot by reasoning over evidence, not by a single scalar metric. HarnessEval-W reframes evaluation as an agentic, hierarchical process: instead of applying a fixed rubric, it interprets each case, decomposes the evaluation question into measurable subproblems, launches specialized sub-agents equipped with context and diagnostic tools, and then validates and summarizes gathered evidence into a transparent evidence tree that justifies the final score.
Key Findings
- Evidence-first evaluation: Produces a complete, inspectable evidence tree for every judgment, so scores carry a verifiable reasoning chain rather than being opaque aggregates — this makes diagnoses auditable and actionable.
- Task decomposition and specialization: By splitting complex evaluation questions into subproblems and giving each sub-agent tailored tools, the pipeline captures diverse failure modes (physics, causality, consistency) more granularly than monolithic metrics.
- Human alignment and diagnostic richness: Across experiments on 18 representative world models and 330 cases, judgments align closely with human preferences while providing fine-grained, machine-readable diagnostics for each rollout.
- Extensibility as a live benchmark: The pipeline is open-sourced as a live benchmark that invites new evaluation skills and cases, enabling the community to grow the suite as world models evolve.
Who it's for and trade-offs
Great fit if you need transparent, auditable evaluation of generative or interactive visual/world models and want per-case diagnostic evidence rather than opaque scalar scores. It is useful for researchers comparing model failure modes, benchmark maintainers who want extensible evaluation skills, and developers diagnosing simulation or control errors.
Look elsewhere if you need ultra-low-cost, large-scale sweep metrics (HarnessEval-W is computationally heavier due to multiple sub-agents and verification steps), or if your evaluation target is simple text-only alignment where visual diagnostics add little value. The approach also depends on the reliability of underlying sub-agents and toolchains; extending to new domains requires building or adapting domain-specific sub-skills.