A model’s training pairs may matter more for cross-modal alignment than its size. The paper’s central insight is that models align when their positive pairs connect observations of the same world states—not merely when they see more data or larger networks.
Key Findings
- Under idealized conditions, separately trained JEPAs learn the same representation up to rotation when their positive pairs identify the same world states. If those pairings differ, the resulting misalignment is bounded by the mismatch.
- Pairing two observations of one scene avoids spending representation capacity on modality-specific noise. By contrast, next-token prediction pairs a document prefix with its continuation, behaving like a “bad-view” JEPA.
- In experiments, good-view training reached comparable alignment with 42 times fewer samples than next-token prediction on the same corpus. A 109M-parameter text JEPA also exceeded evaluated language models up to 72B parameters on the reported alignment benchmarks.
- The authors demonstrate that a learned rotation can connect separately trained image and text encoders for image captioning, and that alignment can be recovered from unpaired image and caption sets in a benchmark setting.
Who It’s For—and the Trade-off
A useful read for researchers studying self-supervised learning, representation alignment, and multimodal training objectives. The theorem is a population-level result with assumptions including injective renderings and matched marginals; real data are lossy, and the experiments do not satisfy every theoretical condition. Good views also depend on knowing which observations describe the same scene, sometimes using captioner-generated text.