AIAny
Icon for item

Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligence

Pretrains a DiT-based Mixture-of-Experts video foundation model for embodied intelligence by augmenting internet videos with robot-centric footage and using a multi-dimensional reward system to prioritize physical realism and task completion while scaling MoE for better capacity vs. inference trade-offs.

Introduction

Most video generative models optimize for visual fidelity and creativity; they rarely encode the physical constraints and action semantics a robot needs to act in the world. The core insight of this work is that by combining a scalable Mixture-of-Experts architecture with curated robot-oriented video data and task-aware reward signals, a video foundation model can be pretrained to better support embodied perception and control rather than just content creation.

Key Findings
  • Adopts a Mixture-of-Experts (MoE) variant of DiT and scales it from scratch, enabling higher modeling capacity with a more favorable inference cost trade-off compared to equivalently larger dense models — meaning stronger sequence modeling without linear increases in runtime for many deployments.
  • Introduces a data profiling and augmentation pipeline that injects manipulation, navigation, and egocentric robot footage into standard internet video corpora, improving the model's exposure to action-specific dynamics and agent-centric viewpoints.
  • Uses a multi-dimensional reward framework during pretraining that goes beyond aesthetics and motion consistency to explicitly enforce physical rationality and task completion signals, aligning learned representations with downstream robotic objectives.
  • Releases LingBot-Video as an open, large-scale MoE video foundation model focused on embodied intelligence, aiming to bridge generative video capabilities and actionable world dynamics for downstream robot tasks.
Who It's For and Tradeoffs

Great fit if you aim to bootstrap perception or planning modules for embodied agents from large-scale video pretraining, or need a foundation model that encodes action affordances and egocentric dynamics. It favors scenarios where physical plausibility and task-aligned behavior matter more than purely photorealistic or highly creative video synthesis. Look elsewhere if your primary goal is highest-fidelity content generation for visual media, or if you require a minimal-weight model for extremely constrained hardware: MoE designs reduce average inference cost but add implementation complexity and may require specialized runtime support.

Where It Fits

This paper positions itself between generative video research and embodied AI: unlike conventional video foundation models that prioritize creativity, it reorients pretraining objectives and data toward robot-relevant phenomena, making it a candidate foundation for downstream robotics perception, action prediction, and simulation-to-real transfer experiments.

Information

  • Websitearxiv.org
  • AuthorsShuailei Ma, Jiaqi Liao, Xinyang Wang, Jingjing Wang, Chaoran Feng, Zijing Hu, Chong Bao, Zichen Xi, Yuqi Gan, Weisen Wang …
  • Published date2026/07/08

More Items

Distills a diffusion teacher’s conditional image distribution by matching exponentially tilted student means to teacher posterior means. It supports flexible sampling budgets on the trained timestep grid while preserving generation diversity.

Introduces a multimodal lens for tracing how vision-language models represent visual concepts and selectively access them in response to questions. Counterfactual tests examine how those concepts affect later reasoning; the work does not claim models are conscious.

Tests multimodal models’ spatial reasoning through screen-based clicks and drags. Its programmatically generated CAPTCHA tasks also form a training environment, with fine-tuning gains reported across seven out-of-distribution benchmarks.