AIAny
Icon for item

HarnessEval-W: Agentifying the Evaluation of Visual Worlds

Automates evaluation of visual world models via a hierarchical agent pipeline that decomposes each case, spawns specialized sub-agents to collect diagnostic evidence, and outputs a verifiable evidence tree plus a final verdict; validated on 18 models across 330 cases and released as a live evaluation pipeline.

Introduction

World-model rollouts demand judgments about physics, causality, and state evolution—things humans spot by reasoning over evidence, not by a single scalar metric. HarnessEval-W reframes evaluation as an agentic, hierarchical process: instead of applying a fixed rubric, it interprets each case, decomposes the evaluation question into measurable subproblems, launches specialized sub-agents equipped with context and diagnostic tools, and then validates and summarizes gathered evidence into a transparent evidence tree that justifies the final score.

Key Findings
  • Evidence-first evaluation: Produces a complete, inspectable evidence tree for every judgment, so scores carry a verifiable reasoning chain rather than being opaque aggregates — this makes diagnoses auditable and actionable.
  • Task decomposition and specialization: By splitting complex evaluation questions into subproblems and giving each sub-agent tailored tools, the pipeline captures diverse failure modes (physics, causality, consistency) more granularly than monolithic metrics.
  • Human alignment and diagnostic richness: Across experiments on 18 representative world models and 330 cases, judgments align closely with human preferences while providing fine-grained, machine-readable diagnostics for each rollout.
  • Extensibility as a live benchmark: The pipeline is open-sourced as a live benchmark that invites new evaluation skills and cases, enabling the community to grow the suite as world models evolve.
Who it's for and trade-offs

Great fit if you need transparent, auditable evaluation of generative or interactive visual/world models and want per-case diagnostic evidence rather than opaque scalar scores. It is useful for researchers comparing model failure modes, benchmark maintainers who want extensible evaluation skills, and developers diagnosing simulation or control errors.

Look elsewhere if you need ultra-low-cost, large-scale sweep metrics (HarnessEval-W is computationally heavier due to multiple sub-agents and verification steps), or if your evaluation target is simple text-only alignment where visual diagnostics add little value. The approach also depends on the reliability of underlying sub-agents and toolchains; extending to new domains requires building or adapting domain-specific sub-skills.

Information

  • Websitearxiv.org
  • AuthorsWeiliang Chen, Haowen Sun, Jun Gao, Jiawei Chi, Hanyang Wang, Qiyu Dai, Yihao Li, Hao Li, Jingnan Gao, Yi-Hsin Hung …
  • Published date2026/08/17

Categories

More Items

Generates videos directly in pixel space without a pretrained autoencoder or cascaded refinement stages. It can jointly produce RGB frames and disparity maps, helping generated scenes follow target camera motion.

Turns LLMs’ physical knowledge into video training data using simulator constraints, physical verification, and iterative correction. It also repairs physical errors in generated videos and releases 100K videos spanning 2,500 physical events.

Continuously perceives live audio and video while deciding whether to respond, interrupt, or keep listening. It couples interaction reasoning with audiovisual generation for real-time, full-duplex exchanges.