Empirically studies how transformer-based native multimodal pre-training scales under fixed compute, deriving compute- and data-allocation power laws and an efficiency frontier for model size, token count, and data mixture; evaluates cross-modal transfer and multimodal in-context learning.
Presents Metis, a prototype memory foundation model that embeds a persistent native memory state into the backbone so historical experience is compressed and accessed via memory attention. Key features: forward-only, gradient-free online memory updates; memory-specific mid-training objectives; and a dual text/code memory design.
Systematically studies how language and vision interact during unified multimodal pretraining, identifies mechanisms that enable modality synergy versus competition, demonstrates the benefit of early joint training, and derives efficient pretraining recipes validated at scale.
Surveys how foundation models reshape AI across the game lifecycle, organizing literature into six roles (play & act; model players & games; design; build & maintain; runtime generation & adaptation; test & evaluate). Emphasizes cross-role links, limits to transfer, and gaps in evaluation and persistent state.
Provides a domain-agnostic world-modeling framework that factorizes latent targets into orthogonal predictive components, with dedicated prediction branches and synthesis for multi-domain forecasting and intervention. Demonstrates improved dynamics and long-horizon rollouts across seven domains and includes experimental biological validation.
Reduces self-attention complexity to O(N log N) by using a coarse-to-fine (pyramid) Top-K block selection with LogSumExp scoring, implemented with hardware-aware Triton kernels for fused routing and scoring—aimed at long-context LMs and retrieval tasks.
Zhentao Tan, Jianrong Zhang +2Affiliation: Zhejiang University. *Corresponding author., Affiliation: [email protected], [email protected], [email protected]
Learns dynamics-aligned latent representations for parametric PDE forecasting using masked-latent predictive pretraining, a geometry projector that aligns latent trajectory geometry, and a physics-structured latent predictor that separates shared evolution from parameter-dependent responses.
Analyzes why self-evolving reasoning models collapse under repeated self-training and proposes R-Quest: a feedback-driven pipeline that trains solvers to reject invalid questions and uses a frozen base model to detect task-level repetition, filtering training data to sustain multi-round gains.
Xiaohan Qin, Xiangdong Zhang +8School of AI, Shanghai Jiao Tong University, Shanghai Innovation Institute +3
Enables multi-head, depth-wise residual retrieval in Transformers by expanding a single residual stream into multiple context-aware streams using causal depthwise convolutions. Improves pretraining convergence and downstream benchmark scores with minimal overhead, yielding notable iso-loss compute speedups in large MoE models.
Predicts future egocentric video from actions for human and robot embodiments. It represents movement with camera poses and manipulation with projected skeletons, trained on a unified collection of human and robot experience.
Uses natural language to describe actions and visual changes, letting one predictor learn from simulation, robot trajectories, images, and videos. It supports cross-domain planning and can initialize robot policies.