Why this matters
Benchmarks for AI-driven science require agent-facing inputs and controlled variation in how much methodological guidance is given. This seed-42 package supplies one deterministic instance per final task so researchers can run repeatable evaluations of agent autonomy and compare performance when procedural guidance is present (B1) versus progressively removed (B2–B4).
What Sets It Apart
- Fixed-seed, agent-visible artifacts: each task folder contains four prompt variants and the data the agent can access, enabling identical inputs across runs for fair comparisons.
- Project-level, multi-domain scope: 60 tasks spanning 11 scientific domains focus on long-horizon research workflows rather than isolated microbenchmarks, emphasizing planning, experimentation, and result validation.
- Designed for execution testing: intended to be run by agent-execution frameworks (sandboxed runners) and paired with private reference answers/scorers for official evaluation; the public package omits those private assets to allow open experimentation while preserving official scoring integrity.
Who It's For and Tradeoffs
Great fit if you evaluate or develop LLM-based research agents, compare prompt-level robustness (B1–B4), or need reproducible agent-facing inputs across many scientific tasks. Look elsewhere if you only need small-scale unit tests or labeled gold-standard answers — the public seed set omits reference answers and official scorers, so closed-loop benchmarking against official metrics requires access to the private evaluation service used by ASI-Bench.
Where It Fits
Use this dataset as the deterministic input layer in agent evaluation pipelines, e.g., local development, ablation studies, and cross-agent comparisons. For leaderboard-grade scoring or to reproduce published official results, pair these instances with the ASI-Bench scoring backend and reference assets used by the project maintainers.
