Provides a real-scale 3D Hong Kong sandbox to evaluate whether multimodal LLM agents can turn local street-view perception into sustained spatial action, supporting closed-loop first-person interaction, an interactive map, and controlled tests of grounding, long-range navigation, and robustness.
Rewrites physical scenes as executable world programs (e.g., MuJoCo scene descriptions) and uses an agentic abductive loop to propose, execute, render, verify, and iteratively refine those programs from videos or text. Verified executable worlds supply scalable physical supervision for training vision–language models.
Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Models
Proposes VLAct, a representation-centric continued pre-training method for Vision-Language-Action models that preserves VLM priors and enforces cross-embodiment action semantics to turn limited robot trajectories into transferable visual-action representations; shows strong gains and sample efficiency on multiple VLA benchmarks using modest compute.
Generates synchronized spoken dialogue and explicit full-body co-speech motion (facial expressions, hands, upper- and lower-body) end-to-end from the same hidden states, replacing the speech-then-motion cascade. Trains with a scalable pseudo-labeling pipeline (422,856 ranked pairs) and supports real-time inference (RTF 0.78) while matching teacher motion metrics within ~2%.
Proposes GenFirst, a generation-before-reconstruction end-to-end training strategy for latent generative models that avoids latent collapse by prioritizing generative objectives and then progressively strengthening reconstruction, validated with strong gFID/GenEval results on ImageNet-256 and text-to-image tasks.
Converts posed indoor RGB(-D) video into editable, simulation-ready 3D scene graphs by parsing multi-view evidence into per-object bundles, generating complete object assets from that evidence, and placing them with GizmoAct, a VLM policy that refines 9-DoF poses through closed-loop GUI actions.
Develops a vision-language foundation model for autonomous driving that unifies 3D BEV perception, visual question answering, and motion planning without changing the pretrained VLM architecture. Key elements include an external BEV perception head for 3D detection and occupancy, a Planning Expert using flow-matching for trajectory prediction, and a staged training recipe combining driving and general VLM data.
Embodied AI2026
Learns generalizable World Action Models for robotic manipulation by scaling causal egocentric video pretraining and grounding learned dynamics with heterogeneous robot trajectories. Key features: a three-stage curriculum (video pretraining, video-action mid-training with a unified action representation, and target-robot specialization) and a Slow–Fast dual-system for 30 Hz real-time action prediction.
Builds high-fidelity image generators by pairing a 6B Diffusion Transformer with a frozen LLaDA2.0-Mini vision-language module, relying on extensive image-only pretraining and mid-training; model weights, training code, and recipes are released.
Trains a world-action model that predicts future visual states to guide zero-shot robotic manipulation; introduces CoAE, SVP, IDM and KASO to pretrain generative and action components from scratch on manipulation data, scaling up to 30,000 hours and improving zero-shot success.
Generates compositional 3D scenes as collections of individual object meshes by conditioning a single-object 3D generative prior on multi-view posed observations. Key features include anchor-aligned canonical frames, multi-view DINOv3 feature lifting with an IBR-style fusion, and LoRA adaptation to complete heavily occluded objects; includes a large UE-MeshyScene benchmark.
Trains end-to-end driving without human trajectory supervision by decoupling perception and action: DriveVFM distills multiple frozen vision foundation models into a single camera backbone, and DriveRL trains a privileged closed-loop RL teacher whose rollouts supervise a camera-only planner, yielding state-of-the-art closed-loop benchmark results.