Lets vision-language models control robots via a compact semantic action interface that maps intent to discrete action units; supports zero-shot use of closed-source VLMs, low-cost fine-tuning of open VLMs, and GUI-based demonstration collection.
Translates natural-language instructions into executable programs that maintain an explicit, persistent global world state and compiles state-augmented 3D oriented bounding boxes into pixel-aligned conditioning signals for pretrained video generators. The approach decouples state evolution from rendering, enabling programmable entity control, off-screen state, and long-horizon interactive scenarios.