A benchmark score can hide two very different problems: the test may overrepresent some difficulty levels, or it may have no items in regions where a model is improving. Those gaps can make useful training progress invisible, even when the benchmark is widely used.
Key Findings
- The paper measures difficulty distributions with item response theory and compares each benchmark with a uniform reference using Wasserstein distance. This makes implicit difficulty weighting measurable rather than an assumption.
- Across 25 benchmarks in seven domains, 10 show a distribution mismatch of at least 0.10, while 8 have a largest uncovered gap of at least 0.10. The results suggest that leaderboard scores may reflect what a benchmark happens to emphasize, not a difficulty-neutral comparison.
- BenchFill rewrites existing items toward the midpoint of large gaps. With three added items, it reduces the sum of the three largest gaps by an average of 32.67%, close to the 35.06% idealized oracle and above the 17.52% uniform-parent baseline.
Who Should Read It
Great fit for benchmark designers and teams using benchmark scores to track training progress: it offers diagnostics for both score weighting and local coverage. Look elsewhere if you need a universal definition of the intended difficulty distribution; the uniform reference is a neutral default, not necessarily the right target for every evaluation. Difficulty estimates also depend on the models used to calibrate items.