Why this matters
Most large models are pretrained with text-first recipes and then fused with vision features, leaving open how to allocate compute and data when training multimodal models from scratch. This paper provides the missing, practical scaling laws for native multimodal pre-training: it shows predictable compute-loss behavior, derives power-law relations for compute-optimal model size and token counts, and models how data mixture (text vs. multimodal) shifts optimal allocations — producing an explicit efficiency frontier practitioners can use when budgeting compute.
Key Findings
-
Predictable compute law for objective loss: training loss follows a stable compute-dependent curve, so you can estimate marginal returns from added FLOPs before actually training.
- So what: enables upfront tradeoff decisions between larger models and more tokens under a fixed budget.
-
Power-law scaling of optimal model size and token count: both scale as power laws with compute, but language and multimodal objectives exhibit distinct exponents.
- So what: optimal capacity and training length are not one-size-fits-all; language-heavy objectives favor different scaling than multimodal objectives.
-
Data-composition sensitivity: the multimodal allocation law depends strongly on the text vs. multimodal ratio, while the language allocation law is relatively invariant to mixture.
- So what: for text-heavy mixtures compute-efficiency emerges only at larger scales, pushing resource allocation toward larger models rather than more tokens.
-
Efficiency frontier and downstream effects: by modeling mixture effects the authors present an efficiency frontier (model size × tokens × mixture) and show native multimodal pretraining yields positive cross-modal transfer, improving pure-text spatial reasoning and enabling robust multimodal in-context learning.
- So what: practitioners can pick configurations from the frontier to hit desired downstream behaviors with known cost tradeoffs.
Who it's for and tradeoffs
Great fit if you design or budget large-scale multimodal foundation models and need quantitative guidance on model-vs-data tradeoffs under fixed compute. The paper is especially useful for researchers comparing native multimodal-from-scratch recipes against late-fusion baselines and for teams planning mixed text/image datasets.
Look elsewhere if your primary concern is small-scale prototypes, edge deployment constraints, or if you need implementation-level engineering recipes (this work is empirical and prescriptive at the scaling-law level, not an engineering blueprint). Also, the conclusions depend on the specific architectures, data mixtures, and compute regimes tested; practitioners should validate the frontier when moving to substantially different modalities or architectures.