Open-AoE turns low-cost smartphone egocentric video into alignment-ready supervision for embodied learning: instead of a raw video collection, it ships a productionized capture→process→reconstruct pipeline that yields per-frame MANO hand parameters, metric SLAM camera poses, and temporally localized atomic actions.
What Sets It Apart
- Large, contributor-driven scale and diversity: the first release targets ~2,000 hours collected across hundreds of contributors and devices, giving varied viewpoints, lighting, and everyday manipulation contexts useful for generalization tests.
- Multi-signal, training-ready outputs: each clip includes undistorted intrinsics, hands.npz (MANO parameters in camera/world coords, per-frame validity masks), camera_traj.npz (framewise cam→world homogeneous transforms + intrinsic matrix), and ego_action_annotation.json (segment-level atomic verbs/objects/bboxes). These artifacts are synchronized so you can directly project 3D hand/world points to image pixels or feed aligned sequences into VLA/WAM/world-model training pipelines.
- Reusable pipeline & tools: the release includes the cloud-side processing pipeline and a downstream toolchain (visualization, cross-embodiment retargeting, robot replay, and model conversion/recipes), lowering the engineering cost to go from mobile capture to robot/ML training data.
- Practical conventions for use: intrinsics are provided in undistorted form (use these for projection), pred_valid masks must be applied to filter unreliable frames, timestamps and frame indices are stored as strings and require numeric conversion, and coordinates are in meters with explicit camera/world transform relations.
Who it's for and Tradeoffs
Great fit if you need aligned, multi-modal supervision for embodied tasks — examples: training vision-language-action policies, learning world-action models, hand-object interaction reconstruction, or human-to-robot retargeting. The dataset is useful when you want a ready-made pipeline and tooling to avoid building a private SLAM/hand-reconstruction stack.
Look elsewhere if you require per-frame depth maps (current release omits per-frame depths), strict lab-controlled capture (data is smartphone-captured in natural settings), or a permissive open-source license (dataset license is listed as “other” and users should verify usage terms). Also note frame rate and resolution conventions (30 FPS; mostly 1280×720 or 1920×1080) and that some fields (intrinsics, distortion) are corrected in undistorted outputs—follow the provided usage notes to avoid projection mistakes.
