Omnimodal world model that jointly processes and generates text, images, video, audio, and action trajectories for physical AI. Uses a mixture-of-transformers to combine autoregressive reasoning and diffusion-based multimodal generation; released open-source with checkpoints, datasets and benchmarks for robotics and simulation.
Evaluates multimodal LLMs on streaming egocentric video for spatial intelligence using 1,680 human-annotated questions across 348 videos; organizes tasks into four hierarchical levels (perception → tracking → simulation → allocentric mapping) and highlights allocentric mapping as the main bottleneck.
Generates minute-level, multi-shot synchronized audio+video from a single text prompt, using a paired cross-modal memory to preserve character appearance and voice across shots. Uses DMD-distilled few-step inference for ~7.5× speedup; requires high-GPU memory and is released under the LTX-2 community license.
Native multimodal model for image/text/video→text tasks with million‑token context support. Uses a sparse-attention operator to cut long‑context compute and latency, and targets agentic, coding, and long-horizon conversational workloads.
Large-scale training corpus for knowledge- and reasoning-intensive video understanding: 315K video reasoning examples over 145K CC-licensed expert-domain videos, with human-in-the-loop chain-of-thought rationales to strengthen post-training for video reasoning. ([arxiv.org](https://arxiv.org/abs/2606.05259))
Provides 1,036,431 identity–text–video triplets with per-video JSON annotations and reference keyframes to train and evaluate identity-preserving customized video generation models. Data is drawn from ~320K Pexels HD videos; videos must be downloaded separately per Pexels' terms.
Stores a persistent 3D scene cache directly in a diffusion model's latent space to produce temporally and spatially consistent videos. Constructs memory via depth-guided back-projection and queries it with direct latent-space warping — achieving large speed and memory gains versus pixel-space 3D baselines.
End-to-end framework for controlled character animation that transfers motion from driving videos to reference characters without intermediate pose or background representations. Introduces the MotionPair‑60K end-to-end motion-transfer dataset, in‑context mask conditioning and mode‑specific RoPE for task unification, plus Bias‑Aware DPO to mitigate synthetic-detail errors.
Provides 500+ hours of human whole-body teleoperation demonstrations for humanoid robot learning in real homes, with synchronized video, joint states, action traces and language annotations. Includes 23K+ episodes, fine-grained subtask labels, and raw ROS/MCAP plus compressed LeRobot formats.
Provides 500+ hours of human whole-body teleoperation recordings of a Unitree G1 in real homes, packaged in LeRobot v3.0 for robot learning. Contains 23K+ episodes, ~40M frames, multi-view 480p@30 video, 29-DoF states, actions and language annotations; CC BY 4.0 and large download size.
End-to-end pose-driven image-to-video model that animates a reference character from a driving video, supporting cross-identity replacement and multi-character scenarios without intermediate pose representations; performs best at 704p and ships as a diffusers-compatible checkpoint.
Continuously watches live video and autonomously decides each second whether to speak, stay silent, or delegate; released together with an 8B vision-first model, time-aligned interaction data, training recipe, and a deployable real-time system. Designed for vision-triggered, low-latency streaming scenarios and evaluated across six real-world streams.