Most foundation-model pretraining treats text as the primary, or sole, substrate for learning language intelligence. This paper challenges that assumption by asking: what if the visual presentation of knowledge — figures, typeset equations, and page layout — is an essential input rather than a lossy side-channel? The core insight is that training directly on visual document representations produces more informative pretraining signals than converting everything to plain text, and that this advantage scales across model families and benchmarks.
Key Findings
- Visual pretraining on the same underlying corpora consistently beats text-only pretraining across multiple backbone architectures and evaluation suites — indicating the gain is not a niche engineering tweak but a broadly applicable effect. This means models can learn layout- and figure-aware semantics that plain text loses, improving tasks that depend on visual context (e.g., interpreting plots, equations, multi-column layouts).
- The approach is unsupervised and operates without requiring perfect text extraction: it leverages raw visual tokens (images, layout cues) alongside or instead of extracted text, reducing dependence on brittle OCR pipelines. Practically, this opens a scalable pathway to incorporate web and document collections that are visually rich.
- Empirical results show the method scales: gains appear across small-to-large backbones and persist when evaluated on standard benchmarks for language understanding and multimodal tasks, suggesting visual pretraining can be a complementary ingredient in foundation-model recipes.
Who it suits and trade-offs
Great fit if you train or fine-tune foundation models on corpora containing dense visual information (scientific papers, textbooks, web pages, reports) and you need models to understand figures, typeset math, or layout-dependent semantics. It is less compelling if your domain is strictly plain text (chat logs, transcripts) or if compute/bandwidth budgets make image-based pretraining impractical.
Key trade-offs include increased storage and compute for image tensors and potential engineering complexity for batching and augmentations; however, the paper argues these costs are offset by the additional, recoverable information visual inputs provide. For production use, combine visual pretraining selectively (e.g., mixed visual/text curricula) where visual content actually matters.