Why this matters The dataset packages realistic, action-conditioned video rollouts together with the source-native controls and sparse 3D pose reconstructions, so researchers can study control-to-visual dynamics without constructing alignments from disparate sources. Its preservative release (no transcoding of MP4s; committed annotation archives) favors reproducible experiments that combine temporal controls, captions, and geometric cues.
What Sets It Apart
- Episode-level fidelity: 30,969 episodes preserve original MP4 bytes alongside a deterministic annotations.tar containing action.json, caption.json, and a COLMAP text sparse model (cameras.txt, images.txt, points3D.txt). This keeps media, controls, and sparse 3D geometry collocated for per-frame analysis.
- Action-native controls: Frame-synchronous keyboard control records (boolean key maps plus per-frame delta translations/rotations) let you train or evaluate action-conditioned predictors using the same control space exposed at inference time.
- Research-oriented layout and provenance: Metadata.jsonl indexes anonymous HMAC sample_ids and hub URIs to the media; the dataset emphasises reproducibility (commit-pinned annotation objects) and advises selective payload download.
- License & scale: Released under Apache-2.0, covering ~30.9k episodes and ~185.8k source objects; payload size and per-sequence duration/fps/frame-count statistics are not asserted by the payload publication.
Who it's for and tradeoffs
Great fit if you want to: train or evaluate action-conditioned video models, explore controllable world models that couple controls with visuals, or combine sparse 3D geometry (COLMAP poses) with temporal control signals and captions. Look elsewhere if you need fully labeled dense 3D reconstructions, guaranteed per-frame temporal alignment metadata beyond the provided schema, or a small-footprint curated subset — the release preserves raw payloads and does not claim exhaustive coverage of physical or demographic factors.
Practical notes
- Size and access: The dataset indexes media via the Hub and makes annotations available as uncompressed TAR archives; consumers should validate TAR members before extraction. Selective download via commit IDs is recommended for reproducibility.
- Intended uses: action-conditioned video prediction, controllable world-model research, representation learning, and agent learning. The authors caution that this is not a symbolic simulator and that suitability, bias, safety, and legal obligations must be assessed for downstream applications.
