Why this matters
As models move beyond visual replication to building systems that actually work, benchmarks must measure not just appearance but executable behavior. ExeWorldBench fills this gap by framing world modeling as a code-to-execution problem: the goal is to infer structure, constraints, and causal interactions from visuals or intent and realize them as runnable Blender scenes whose physical behavior can be evaluated automatically.
What Sets It Apart
- Task design that emphasizes executability: three tiers (Mechanism Executability, Compositional Executability, Intent-Driven Synthesis) progressively test single mechanisms, multi-stage causal chains, and goal-driven world construction — all judged by whether the generated Blender code produces the expected physical behavior rather than pixel similarity.
- End-to-end evaluation assets: each task includes reference images/frames, agent prompts for API-driven workflows, harness files for offline execution, and judge prompts/scripts to score Visual Structure and Mechanical Motion, enabling reproducible, automated benchmarking of visual-to-code and text-to-code agents.
- Focus on reasoning + realization: instead of penalizing superficial visual differences, the suite rewards correct component relationships, constraints, and motion dynamics, making it specifically useful for assessing embodied, physically grounded code-generation capabilities.
Suitable For & Tradeoffs
Great fit if you want to benchmark or develop systems that must infer object relationships, constraints, and dynamic behavior and then realize them as runnable 3D code (e.g., coding agents, multimodal VLMs that emit Blender scripts, research on physical reasoning). The dataset includes prompts and evaluation harnesses to streamline comparisons.
Look elsewhere if your goal is pure image generation or visual fidelity benchmarks: this suite prioritizes mechanical correctness and executable outcome over photorealistic rendering. It also assumes familiarity with Blender-based execution and a willingness to run the provided harnesses to obtain final scores.
