Vision often gets added to language models as an afterthought, but the training dynamics reveal deeper asymmetries: this paper shows that when modalities are unified early, models avoid relying on language priors, enable productive knowledge transfer, and can be trained far more compute-efficiently than naive recipes suggest. Its core insight is that modality interactions follow regular patterns that can be shaped by data complexity and architectural choices to yield synergy rather than competition.
Key Findings
-
Knowledge Flow: Language, visual understanding, and visual generation transfer knowledge across modalities in distinct and asymmetric ways — knowing these directional effects lets designers target which capability to boost (e.g., better visual generation vs. language-grounded vision).
-
Synergy vs. Competition: Whether modalities help or hurt each other depends largely on data "complexity". Architectural choices that promote shared attention and normalization but preserve modality-specific feed-forward layers consistently encourage synergy across different visual tokenizer designs.
-
Early Unification: Training modalities together from the start outperforms late alignment or sequential schemes. Delaying visual integration produces a "vision laziness" effect where models over-rely on language priors and under-develop visual representations.
-
Practical Recipes and Scaling: The paper derives efficient pretraining recipes that reach strong generative performance using only about 5% of the compute of baseline approaches, and validates the core findings at scale with multiple 13.5B mixture-of-experts models trained on ~2T tokens.
Who it's for and tradeoffs
Great fit if you design or evaluate foundation models that combine vision and language and need principled guidance on when to unify modalities, which architecture choices encourage positive transfer, and how to reduce compute for generative visual capabilities. Look elsewhere if your application is narrowly supervised, small-scale vision-only work, or you cannot afford the dataset engineering needed for early joint training — the benefits rely on appropriate multimodal data and careful architectural choices. The large-scale validations use MoE models and substantial data, so reproducing scale effects may need similar resources.
Methodology (brief)
The conclusions come from controlled experiments on synthetic and large-scale real-world datasets, ablations over architectural variants (attention, normalization, modality-specific FFNs), and scaling runs that confirm the small-scale phenomena persist when training larger MoE models.
