A policy can benefit from predicting motion without running a predictor at execution time. The key idea here is to use a spatial world model as a training-time teacher, then transfer its dynamics-aware representations into the policy’s latent queries. This targets a practical tension in dexterous robotics: anticipating occluded finger motion is useful, but explicit future generation can add deployment complexity and latency.
Key Findings
- The training-time world model learns from offline depth, hand geometry, and demonstrations, predicting 3D hand-point motion and visual feature changes. This gives the policy access to spatial dynamics that are difficult to infer from a single RGB view alone.
- The world model is frozen while its predictive queries are aligned with policy queries. At deployment, only the policy remains, so it does not need depth input or a future-decoding stage.
- Across 11 DexJoCo tasks, success rises from 49.4% for the GR00T N1.7 base policy to 60.5%. In three real-robot tasks, the method reaches 58.3% overall, compared with 50.0% for the strongest reported baseline.
- A linear probe also recovers future hand motion more accurately from the trained policy’s features: at 0.54 seconds, displacement error is 10.01 mm versus 13.59 mm for the architecture-matched QwenVLA action-only policy. This supports the claim that predictive structure persists in the deployed representation, rather than only improving task scores.
Method and Trade-offs
The distinctive choice is to separate where foresight is learned from where it is used: depth and explicit spatial prediction shape the policy during training, but are removed from inference. The approach still depends on demonstrations and offline depth for world-model training; removing offline depth reduces reported success. Its evidence also has a limited scope—11 simulation tasks and three physical tasks—so results do not establish performance across the full range of dexterous manipulation.
Who It Suits
A strong fit for robotics researchers studying dexterous manipulation, latent world models, or ways to transfer predictive representations into policies without paying for future decoding at runtime. Look elsewhere if your setting lacks demonstration data or offline depth, requires broad real-world validation, or needs a method that does not add a separate world-model training stage.
