Perception is often the confounding factor in multimodal LLM failures: holistic benchmarks mix perceptual mistakes with reasoning and knowledge gaps. PerceptionBench tackles this by isolating the earliest failure points — atomic perceptual capabilities — so you can tell whether a model truly "sees" or merely guesses.
What Sets It Apart
- Error-driven design: items were constructed from an error taxonomy derived from diagnosing failures across 42 existing visual benchmarks, ensuring the dataset targets real-world perception breakdowns rather than synthetic puzzles. This yields 10 clearly defined atomic perceptual capabilities.
- Capability-level balance and difficulty stratification: 3,000 verified questions (60% decomposed from prior-benchmark failures, 40% newly authored) are subsampled to balance capabilities and difficulty, enabling fine-grained capability profiling instead of a single aggregate score.
- Audit-ready evaluation: unified prompting and high-agreement automated judging (GPT-oss-120B with 99.7% agreement on a 300-sample audit) let maintainers run consistent leaderboards and compare sixteen frontier MLLMs under the same protocol.
Who it's for and tradeoffs
Great fit if you need to diagnose or benchmark the visual perception limits of multimodal LLMs, develop targeted perception improvements, or compare capability-level strengths across models. Not a replacement for application-level benchmarks when reasoning, knowledge, or multi-step tasks are the primary focus. Practical tradeoffs: the released set is 3,000 items (sampled from a larger 17,000+ pool), which gives clarity for capability diagnostics but is modest for large-scale training; the dataset uses a CC-BY-NC-4.0 license, restricting commercial use.
