Most progress in vision–language–action (VLA) models hinges on abundant, high-quality robot trajectories—but collecting such data at scale is costly. ACE-EGO-0 attacks this bottleneck by turning egocentric human video into usable robot-format supervision and by designing training and representation choices that make noisy human signals complementary rather than harmful. The key insight: with a camera-space unified action representation and reliability-aware loss weighting, large amounts of imperfect human data can reliably teach high-level semantics and improve downstream robot manipulation.
Key Findings
- Unified camera-space action representation: ACE-EGO-0 expresses actions in a camera-centered coordinate system, conditions on morphology, and uses time-aligned action chunking. So what: this reduces embodiment gaps between human and robot data, enabling direct joint pretraining without brittle inverse-kinematics retargeting.
- Scalable video→action pipeline: the paper builds an automated pipeline to extract pseudo-action trajectories from egocentric videos at scale. So what: it unlocks thousands of hours of human manipulation supervision (1.48K hours used) that supplements robot/sim data in pretraining.
- Reliability-aware training with human auxiliary loss: noisy pseudo-actions are downweighted and supervision is concentrated on reliable signals. So what: this makes co-training robust—joint pretraining consistently improves fine-tuning and yields better task transfer than robot-only baselines.
- Empirical gains and transfer: instantiated on 4.53K hours of robot/sim data plus 1.48K hours of pseudo-labeled human data, ACE-EGO-0 achieves state-of-the-art results on benchmarks (RoboCasa GR1 TableTop, RoboTwin 2.0) and shows strong real-world bimanual manipulation transfer. So what: scaled human supervision can materially boost VLA performance in practice.
Who it's for and tradeoffs
Great fit if you need to scale VLA pretraining for manipulation and want to leverage large egocentric human video collections to reduce robot data needs—especially for learning task semantics and compositional behaviors. It suits research groups or labs with access to egocentric datasets and interest in representation-level solutions (camera-space actions, morphology conditioning).
Look elsewhere if you need turnkey robot policies without dataset engineering: ACE-EGO-0 requires building or adopting a video-to-action extraction pipeline and tuning reliability-weighting; its gains depend on the quality and domain match of the human videos. It also does not remove the need for some robot or simulation data for final fine-tuning and real-world deployment.