Random subsets often estimate a full benchmark well—but their occasional large misses make evaluation results hard to trust. This paper argues that this instability persists even as more foundation models are evaluated, and shows how to reduce the risky tail without replacing random sampling with an expensive selection pipeline.
Key Findings
- Across 70 text, multimodal, and agent benchmarks, the method lowers estimation error by 16.4% on average versus strong baselines, while using 51.3% of their average construction cost.
- It reduces the chance of high-error outcomes by 50.2%, targeting a weakness hidden by average accuracy: a few unrepresentative subsets can skew conclusions.
- Under score-range, model-family, and temporal shifts, out-of-distribution estimation error falls by 21.8% on average, suggesting the gains are not limited to familiar evaluation settings.
How the Selection Works
A small set of probe models scores a candidate pool once. Many candidate subsets are then compared using cached probe scores, and the subset whose average probe profile is closest to the pool average is chosen for target-model evaluation. This keeps the search cheap: candidate comparisons require no additional model calls, and the selected subset is shared across target models.
Who It’s For—and the Trade-off
A good fit for researchers and evaluation teams that need to run large model comparisons on a fraction of a benchmark, especially when unreliable outliers matter as much as average error. It is less compelling when probe-model scores are unavailable, the benchmark is already cheap to evaluate, or the target setting differs sharply from the probe pool; its selection signal depends on probe scores being informative.