Interactive video is not automatically a playable game: visual generation alone does not guarantee that attacks land, characters obey input, or the world stays coherent. The key idea here is to keep gameplay state explicit and use video generation as its renderer, rather than asking a video model to invent both the rules and their appearance at once.
Key Findings
- Characters are controlled through articulated skeletons, while scene changes are represented as geometry edits. These structured proxies give the generator concrete signals to follow.
- A shared state tracks positions, poses, health, action phases, and cooldowns; a planner and rule-based controller resolve combat. This separates mechanical consistency from the video model’s visual synthesis.
- Training uses 50.3 hours of proxy–video pairs spanning 750 scenes and 178 characters. The reported zero-shot results target unfamiliar scenes, characters, and motions, not just variations of a single game setup.
- An adaptive masking mechanism ties character references to their corresponding regions, helping reduce appearance leakage during generation and character transformations.
Who It’s For—and the Trade-off
A useful read for researchers building controllable video world models or interactive game agents, especially those exploring how symbolic state can constrain generative visuals. It is less suited to anyone seeking a ready-to-play game engine: setup includes scene reconstruction and motion retargeting, uncommon body structures may lack compatible motions, and proxy-to-video consistency can weaken over long rollouts.