A robot’s action space does not need to look like a human’s for it to learn from human experience. The key is to express both through what an observer would see: camera motion for locomotion and projected body keypoints for manipulation. This makes the model’s training signal less dependent on any one robot’s joints, while keeping predicted actions visually grounded.
Key Findings
- A shared kinematic interface connects unlike bodies. Camera poses encode walking and turning; projected skeletons encode manipulation. This gives the model a common representation across people, grippers, and dexterous humanoids.
- The data scale is a central result, not just a training detail. Xp-Data brings together 10,000 hours of human and 3,000 hours of robot egocentric data with synchronized multimodal annotations. The reported experiments find that increasing human pretraining data improves both human-video and robot evaluations.
- The benchmark tests transfer across embodiments. The paper reports leading results on five robot benchmarks, as well as held-out human video, across zero-shot, few-shot, and post-training settings. Xp-Teleop adds long-horizon humanoid tasks recorded with synchronized video, robot state, tactile signals, and language.
- A predictive model can support more than offline evaluation. A distilled version is reported to stream 640×480 video at 21.3 fps on one GPU, enabling live VR interaction and suggesting uses in policy evaluation, planning, and world–action models.
Method and Trade-offs
The model is a latent video diffusion system conditioned on an initial frame, text, and two action signals: camera trajectory and projected skeletons. Its cross-embodiment promise depends on consistent, high-quality annotations—including robot forward kinematics and manually corrected keypoints—so building comparable data is demanding. The model predicts what may happen; it is not, by itself, a general-purpose controller or proof that imagined rollouts will always match physical outcomes.
Who It’s For
Great fit for researchers studying embodied world models, human-to-robot transfer, egocentric video, and model-based planning. Look elsewhere if you need a ready-to-run robot policy, broad deployment tooling, or results for an embodiment and task not covered by the paper’s data and benchmarks.