Most robot learning work either trains end-to-end policies or hands engineers closed-world wrappers. This paper asks a different question: how far can a vision-language model act as a full-stack coding agent when the only interface to hardware is a deliberately thin harness? The core insight is that if the agent must write and preserve its fixes as code and must rely only on raw RGB and structured logs, the resulting loop surfaces where reasoning suffices and where physical contact remains the bottleneck.
Key Findings
- Thin-harness design: the system translates agent-generated commands into joint motions and nothing more, forcing the agent to handle perception-to-3D reasoning, planning, and error diagnosis in code. So what: this keeps learned behaviors interpretable and reusable as small tools instead of opaque weights.
- Self-debugging coding loop: after failures the agent reads structured logs, diagnoses causes, and edits or creates short tools (e.g., triangulation routines, plane intersections, wrist-camera moves). So what: the agent’s experience is preserved as code that generalizes across layouts rather than ephemeral policy updates.
- Empirical performance: on 42 RoboDojo tasks (tools built on 10 training layouts per task then frozen), the system attains 39.7% success and a 48.0 progress score; pushing tasks reach 92.5% while tight insertion is only 8.3%. So what: reasoning and perception suffice for many tasks, but contact-rich manipulation remains the dominant failure mode.
- Practical stack: tools use only RGB and camera calibration (triangulation, ray-plane intersection); experiments used Codex with GPT-6-Astra at medium reasoning effort. So what: strong VLM reasoning plus minimal robot-side plumbing yields a new, audit-friendly route to embodied capability.
Who it's for and tradeoffs
Great fit if you want an interpretable, code-centric route to embodied agents: researchers exploring VLM-based robot cognition, teams that prefer deterministic robot-side execution with agent-authored perception modules, and benchmark-driven evaluations like RoboDojo. Look elsewhere if your primary goal is high-rate contact-rich skills (force control, tight insertions) or fully end-to-end learned controllers; HexaShell’s limit is physical contact rather than reasoning. The approach also assumes access to multi-view RGB and camera calibration and depends on a powerful VLM for code synthesis and debugging.