Why this matters
Real-world document parsing often fails for reasons that synthetic augmentation cannot isolate: geometric warps, screen moiré, harsh lighting or perspective skew have distinct causal effects. Real5-OmniDocBench converts those uncontrolled confounders into controlled variables by physically recreating every OmniDocBench v1.5 page under five photographic scenarios, so you can measure exactly which physical factor breaks a model and by how much.
What Sets It Apart
- One-to-one physical reconstruction: each of the 1,355 original test pages has five matched physical captures (totaling 6,775 images), and the dataset reuses OmniDocBench’s JSON annotations without modification. That strict correspondence makes cross-scenario comparisons directly comparable rather than approximate.
- Scenario-level causal analysis: scenarios are orthogonal (Scanning, Warping, Screen‑Photography, Illumination, Skew) and include diverse sub-conditions (e.g., multiple warping types). This lets researchers attribute performance drops to specific physical factors (geometric vs. optical vs. lighting) rather than aggregate “reality gap” noise.
- Diagnostic benchmark, not just leaderboard: designed to reveal actionable failure modes—e.g., compact, document-specialized VLMs can outperform much larger generalist models under physical stress—so it guides architecture and data-augmentation choices rather than only ranking models.
Who It's For and Trade-offs
Great fit if you need to evaluate or harden OCR and multimodal document parsers against real photographic distortions, compare augmentation strategies, or perform factor-wise robustness studies. It’s particularly useful for teams developing layout/content/structure parsers and for evaluating VLMs on end-to-end document understanding.
Look elsewhere if you need a broader domain coverage of languages, handwriting-heavy corpora, or dynamic video captures—Real5-OmniDocBench focuses on photographic distortions of printed/digital document pages and intentionally preserves the original OmniDocBench annotations rather than adding new types of ground truth.
Practical notes
- Evaluation is fully compatible with OmniDocBench metrics and scripts (TextEdit, Formula CDM, Table TEDS, Reading Order Edit, Overall score), enabling plug-and-play benchmarking.
- Because the dataset emphasizes controlled physical variation, it complements synthetic augmentation: use Real5 to validate whether synthetic methods actually close the real-world gap observed here.
