Pretrained vision–language models encode broad digital-world reasoning. This work asks whether that intelligence can be translated to physical robot control without task-specific robot training. The key insight: expose a human‑intuitive, semantically meaningful discrete action space and let a frozen VLM act in closed loop, then ground its interface use and strategies with a few in‑context demonstrations.
Key Findings
- Interface-first approach: a compact set of discrete motion primitives (unit translations, rotations, gripper open/close) turns manipulation into semantically interpretable decisions the VLM can reason over.
- Closed-loop visual decision-making: the VLM repeatedly observes the scene, selects a discrete primitive, executes it, and adapts—avoiding direct low-level control prediction.
- Strong zero-shot and one-shot gains: on RoboTwin 2.0 C2R zero-shot success is 53.2% (vs. baseline π0.5 at 46.0%), and a single in-context demonstration raises it to 73.6% (+20.4 points). On RoboDojo success rises from 35.67% to 47.17% (+11.5 points) with one demonstration.
- Real-world transfer: same framework controls a Franka robot for block‑in‑basket and block‑stacking tasks without robot-specific parameter updates.
What Sets It Apart
- So what: instead of costly robot-specific policy training, intelligence is transferred by aligning the robot interface to the VLM's reasoning space, enabling immediate deployment of frozen multimodal models.
- So what: a single in-context demo grounds both primitive semantics and task strategy, providing large gains with zero fine-tuning.
- So what: discrete, semantically labeled primitives make behavior interpretable and embodiment-agnostic, easing transfer across simulated benchmarks and a real arm.
Who it's for and trade-offs
- Great fit if you want to evaluate or deploy pretrained VLMs for manipulation tasks quickly, prefer interpretable action abstractions, or need cross-benchmark transfer without collecting robot data.
- Look elsewhere if you require smooth continuous low‑level control, high-frequency closed-loop control at actuator rates, or tasks that demand learned high-precision dynamics compensation—those still benefit from robot-specific training and low-level controllers.
Method overview
The pipeline freezes a VLM, provides image observations plus a discrete-action interface, and uses prompt-style in-context demonstrations to teach the model how to map observations to primitives and higher-level strategies. An embodiment-specific interpreter deterministically grounds primitives into actual robot motions.
