Detailed physical plans are expensive for a language model to spell out: even a trajectory becomes a long sequence of coordinate tokens. The key insight here is that the shape of the answer is often known in advance. Predicting a complete geometric primitive as one structured block can therefore be both faster and more accurate than reproducing the target model’s token sequence.
Key Findings
- A lightweight drafter predicts all coordinates in a visual primitive in one pass, with layouts adapted to primitive type, dimensions, and length. This supports both fixed-size boxes and variable-length trajectories.
- Across evaluated 2D and 3D robot trajectory tasks, the method lowers errors relative to its autoregressive targets; reported 3D waypoint error reductions reach 61% in some settings.
- On MolmoSpaces, a world action model conditioned on these plans achieves 70.2% average success across four task suites, versus 62.6% for the baseline.
- Throughput gains depend on the task and hardware: per-step latency falls about 5.1–5.2× on MolmoSpaces on Jetson Thor and DGX Spark, while preliminary single-rollout measurements report roughly 25× faster 3D point-motion generation.
How It Works
The decoder uses each primitive’s schema to determine the coordinate layout before prediction. It then drafts the complete set of coordinates jointly, rather than requiring agreement with the autoregressive model’s exact token sequence. This shifts the target from “say the same tokens” to “produce a structurally valid, spatially accurate plan.” The approach also extends to numeric fields serialized as text, including dense motion predictions.
Who It’s For—and the Trade-Off
A good fit for researchers and builders exploring low-latency visual planning for robots or autonomous systems, especially where plans have explicit geometric structure. It is less compelling when outputs are free-form language, or when an application cannot provide suitable primitive schemas and annotated geometry. The reported results are benchmark- and hardware-specific; the large point-motion speedup is preliminary and should not be treated as a universal end-to-end gain.