Most embodied systems either train end-to-end or bolt on isolated tools; this work argues a different lever: the harness that mediates model↔robot interaction. The core claim is that a compact, well-designed harness—centered on iterative perception–reasoning–action loops, semantic action abstractions, and multimodal observations—can unlock strong manipulation skills in small, open models with minimal simulated data.
Key Findings
- Iterative perception–reasoning–action loops: repeatedly acquiring targeted perceptual evidence and re-planning improves robustness on occluded, novel, and long-horizon tasks, so the system recovers from intermediate failures without exhaustive end-to-end retraining.
- Semantic action abstractions: exposing high-level, symbolic-like action primitives (rather than raw low-level commands) reduces search complexity for the reasoning model and improves transfer across objects and embodiments.
- Multimodal observations: combining focused visual crops, proprioceptive signals, and symbolic descriptors yields better grounding and generalization than vision-only prompts.
- Data-efficient distillation: the authors distill the harnessed behavior into a 4B open model using fewer than 2,000 simulated trajectories, achieving performance comparable to proprietary baselines and demonstrating transfer to real robot setups.
Who It's For and Trade-offs
Great fit if you want a model-agnostic interface to add embodied manipulation to LLM-style agents, especially when simulator data is limited and you need fast iteration or sim-to-real transfer. The approach favors modular pipelines that let perception, planning, and control be improved independently. Look elsewhere if your target requires extreme low-level precision, continuous control optimizations that depend on heavy RL fine-tuning, or if you cannot provide reasonably realistic simulation/sensor fidelity—harness gains hinge on the quality of perceptual modules and abstractions. The harness also introduces engineering complexity (designing action APIs, verifiers, and multimodal views) that may not suit very small experimental projects.