Most recent video world models focus on generating realistic frames but lack reliable mechanisms for enforcing long‑term rules and maintaining persistent off‑screen state. This work's core insight is to separate explicit state evolution from visual synthesis: use language-to-code agents and a lightweight execution engine to keep a persistent world state, then deterministically compile that state into spatiotemporal conditioning for an existing video generator so rendering and logic no longer compete.
Key Findings
- Decoupling state and rendering: Representing entity state and transitions as executable programs lets the system enforce rules and maintain attributes (including off‑screen and non‑visual properties) across long interactions, rather than relying on frame‑to‑frame visual cues.
- Intermediate representation: State‑augmented 3D oriented bounding boxes (OBBs) plus camera trajectories are compiled into pixel‑aligned spatiotemporal conditioning signals, bridging explicit state and a pretrained generative renderer.
- Practical outcomes: The pipeline supports direct per‑entity control, predefined game mechanics, and persistent identity across occlusion and long horizons. On the introduced CombatStateBench the method reports 94% Count Accuracy and 98% State Accuracy, substantially exceeding prior interactive video world models.
How it works
An agent translates natural language instructions into small executable programs that specify entities, states, and transition rules. A lightweight engine executes these programs to update a persistent global world state. That state is converted into state‑augmented 3D OBBs and combined with a target camera trajectory; this is deterministically compiled into pixel‑aligned spatiotemporal conditioning signals that a pretrained video model uses as the generative renderer. The architecture preserves rich visual detail while enforcing rule‑consistent, long‑horizon dynamics.
Who it's for and tradeoffs
Great fit if you need interactive visual worlds with explicit, enforceable rules (e.g., playable prototypes, controlled simulations, benchmarks) and want separation between logical state and visual fidelity. Look elsewhere if you require end‑to‑end learned dynamics from raw pixels, fully photorealistic unconstrained rendering without conditioning, or systems that avoid any programmatic/state representation. The approach depends on a reliable language→program translator and on a pretrained video generator for visual realism; limitations include reliance on the quality of those components and potential domain gaps when transferring to complex real‑world footage.