Many data-analysis benchmarks can check whether an agent computes a number, but not whether its explanation matches how the observed data came to be. This benchmark addresses that gap by building worlds from explicit mechanisms first, then generating the files agents must analyze. The result is a test of both data work and evidence-based insight, with a reference process behind each rubric.
What Sets It Apart
- Worlds are held out, not just individual questions. The 100 tasks come from 50 worlds excluded from training, reducing the chance that evaluation rewards familiarity with a reused environment.
- Each world yields two analytical directions. A cause task asks what upstream factors explain a target’s variation; a consequence task traces how that target affects downstream outcomes. This separates explanation from impact analysis.
- The evidence is deliberately messy and distributed. Tasks draw on 4–12 subsystems, with a median of about 31 files per task. Agents must parse, clean, join, and interpret observed data rather than consult the world’s hidden specification.
- Scoring distinguishes computation from interpretation. Task-specific rubrics assess atomic calculations and supported insights, while audited generative mechanisms make the intended targets recoverable from the files.
Who It’s For—and the Trade-off
A good fit for researchers evaluating tool-using data agents, especially when correctness means more than producing a plausible statistic. It is less suitable as a measure of everyday user-request coverage: the tasks and data are synthetic, and rubric-based LLM judging may miss valid insights that were not listed or vary with the judge. The full collection is also large—about 50.7 GB—so running the benchmark has meaningful storage and compute requirements.
