Physical AI needs more than realistic scenes or demonstrations alone: a robot must connect what an environment looks like with how people act in it. The dataset’s central idea is to capture both in a shared geometric frame, making real-world spaces and human behavior jointly useful for simulation, robot learning, and evaluation.
What Sets It Apart
- The collection combines 3D scene reconstructions with first-person task recordings, rather than treating environment assets and behavior data as separate resources. This can support experiments that transfer from observed human actions to simulated robot trajectories.
- Its 1,589 released scans and collection inventory of 4,230 recordings across 74 tasks provide varied indoor and public-space settings. The recordings include estimated body and hand motion, device trajectories, and task annotations.
- The reconstructions also support depth and geometry, 3D object detection, and spatial reasoning benchmarks, allowing the same environments to serve both data generation and evaluation.
Who It’s For—and the Trade-Off
A good fit for researchers building embodied-AI or robot-learning pipelines that need real-scene geometry paired with human demonstrations. It is less suitable for commercial use under the CC BY-NC 4.0 license, or for work requiring ground-truth motion capture: body and hand motion are estimated, and automatically generated captions can be inaccurate. The first online batch contains 126 scenes, not the full released collection.