Why this matters
Enterprise document extraction is both ubiquitous and brittle: teams need one system to handle many document families, auditors need evidence of every returned value, and real paperwork includes long repeated lists, scans, and handwriting. ExtractBench makes these production requirements testable by treating the schema as the input spec and requiring schema-valid JSON plus page- and box-level grounding for every extracted value.
What Sets It Apart
- Broad, realistic coverage: 370 documents (4,869 pages) spanning 8 business domains and 67 document types, with each document type defined by a single schema shared across examples.
- Traceability-first evaluation: unified value F1 for extraction accuracy, plus word-level and page-level grounding F1 where human-verified boxes exist, so reviewers can audit model outputs without re-searching the source PDF.
- Challenge tagging and per-field scoring: every case is tagged on five independent axes (task challenge, perception, table structure, length, domain) and each schema field declares its scoring comparator and evidence rules.
- Realistic inputs: mix of born-digital PDFs, synthetic long lists, scanned pages, degraded recaptures, and some handwriting; evaluation is deterministic and reproducible via the open evaluation harness.
Who it’s for and trade-offs
Great fit if you build or benchmark document-extraction systems for enterprise workflows (invoices, regulatory filings, fund-holdings, procurement, remittances) and need to measure both exhaustive list completeness and grounded provenance. Not a large pretraining corpus—its value is as an evaluation suite and reproducible testbed. The dataset and code are open (Apache 2.0) but include downloadable PDFs and per-case JSON schemas and field rules, so you must run the provided harness to reproduce official scores.
Where it fits
Use ExtractBench to compare extraction pipelines, validate grounding and human-in-the-loop correction tools, or stress-test agents on long-list and needle-in-haystack challenges. It complements smaller PDF-to-JSON benchmarks by focusing on exhaustive lists, grounding, and measured cost across many document families.
