Most robotic control work either trains embodiment-specific policies or end-to-end models that fuse perception and low-level control. Show-Harness takes a different tack: it treats foundation vision-language models (VLMs) as high-level decision-makers and exposes a small, semantic action language that the VLM can reason over, while deterministic embodiment-specific interpreters ground those semantics into actual robot motions.
Key Findings
- Semantic action abstraction: Exposes a compact set of discrete action units (directional moves, grasp/release, terminal) so VLMs reason at a symbolic level instead of low-level trajectories — this isolates high-level planning from embodiment details.
- Deterministic grounding: Uses embodiment-specific interpreters to convert semantic units into local robot commands, keeping the VLM responsible for fine-grained decisions while ensuring repeatable, hardware-safe execution.
- Practical unlocks: Demonstrates two deployment modes — zero-shot control of closed-source frontier VLMs and rapid adaptation of small open VLMs with only a few GPU-hours — lowering the barrier to real-robot use.
- GUI demonstration pipeline: GUMI extends the same semantic action space to GUI-based demo collection, enabling humans and agents to "play" multiple embodiments without specialized teleoperation hardware.
- Empirical behavior: Reported experiments show robust generalization across tasks, embodiments, and environments, with performance gains over representative agentic and VLA baselines.
Who it's for and trade-offs
Great fit if you want to leverage existing VLMs as planners for manipulation tasks, rapidly prototype multi-embodiment agents, or collect demonstrations without robot teleops. Look elsewhere if your primary need is millimeter-level control, extremely low-latency closed-loop control, or tasks that cannot be meaningfully abstracted into discrete semantic units — the approach trades end-to-end low-level control for modularity and generalization.
Where it sits
Conceptually positioned between end-to-end vision-to-action systems and fully modular tool-based harnesses: the harness provides a small, model-agnostic semantic interface that unlocks large pretrained VLM reasoning for embodied tasks while relying on deterministic grounding modules for embodiment-specific safety and precision.