Why this matters Most robot policy evaluation still depends on slow, costly real-world rollouts. This paper shows that reliable surrogate evaluators require more than photorealism: long-horizon consistency and action-faithful dynamics matter most. The work composes a focused benchmark (WMBench), a large empirical study, and a practical design roadmap implemented as GigaWorld-1 to make world-model evaluation scalable and aligned with real robot behavior.
Key Findings
- Evaluator quality is dominated by long-horizon, action-faithful rollout consistency rather than short-term visual realism — so models that preserve action controllability produce evaluations that better correlate with real-world success.
- Pretraining gains require balancing general-world knowledge with robot-specific controllability — so scale alone is insufficient; inclusion of robot-centric data and controllability priors improves evaluator alignment.
- Architecture and representation choices (action encoding, memory/masking, evaluator-focused post-training) strongly determine alignment with real-world policies — so design decisions must prioritize action fidelity and rollout stability over pixel perfection.
What this delivers
WMBench: a benchmark constructed from matched real-robot teleoperation data and policy rollouts, enabling controlled comparisons across model families, action encodings, rollout horizons, and metrics. Empirical sweep: evaluation of 7 video world models, 4 action representation schemes, and 324,000+ simulated policy rollouts paired with real executions, plus community submissions and 12,000+ hours of training videos. Outcome: a practical roadmap and an implemented model, GigaWorld-1, tuned for policy evaluation, with code, models, datasets, and tooling released to accelerate reproducible evaluator research.
Who it's for and tradeoffs
Great fit if you need scalable, reproducible evaluation of robot policies without extensive real-world rollouts — researchers building or benchmarking world models, teams running large-scale policy comparisons, and challenge organizers. Look elsewhere if your goal is high-fidelity sim-to-real control for low-level dynamics learning or safety-critical deployment without further real testing — world-model evaluation is a surrogate that reduces experimentation cost but does not replace final real-world validation.