A parser can get nearly every character right and still mislead a reader by attaching a value to the wrong table header or inventing a blank form entry. This benchmark makes those high-impact failures visible: scores reflect error severity, not just the fraction of content matched.
What Sets It Apart
- Six dimensions cover both content and structure, including chart data, visual grounding, and government forms—areas that broad OCR tests can miss.
- More than 140 error types distinguish changed facts, missing or invented content, and broken relationships. Per-page diagnostics help explain why a score fell.
- Its 4,317 page-level cases span 2,645 source documents, with targeted collections for difficult tables, meaningful formatting, and forms. Full chart data annotations also make evaluations less dependent on sparse spot checks.
- Critical and major errors impose calibrated score caps, so a wrong value cannot be masked by a page that is otherwise mostly correct. The overall score averages the six dimensions equally.
Who It’s For—and the Trade-off
A good fit for teams comparing document extraction pipelines or diagnosing failures in OCR, layout analysis, and structured-data recovery. It is less useful as a general-purpose OCR score: its six dimensions and targeted subsets are deliberately specific, and results should only be compared using the same dataset revision and scorer version. Scores are not directly comparable with ParseBench-1.