Why this matters
Benchmarks that test end-to-end autonomous research require realistic, reproducible task instances. This seeded instance pack supplies ready-to-run inputs and matched prompt conditions so researchers can evaluate how LLM-based agents perform the full research loop—idea, method selection, execution, and artifact production—under controlled, repeatable conditions.
What Sets It Apart
- Full instance-level artifacts: each task directory contains four prompt variants (B1–B4), agent-visible input files, and reference artifacts and metadata where available, so experiments can be run without re-synthesizing inputs.
- Project-scale, multi-domain coverage: instances correspond to 60 project-level scientific tasks spanning 11 domains, enabling cross-domain comparisons of agentic capabilities rather than toy problems.
- Controlled prompt conditions: the B1–B4 scheme isolates method guidance vs. objective-only prompts and introduces task-irrelevant noise, which helps diagnose where agents rely on procedural scaffolding versus problem understanding.
- Reproducible and redistributable: instances are released under Apache‑2.0 to permit reuse in external evaluations and leaderboards.
Who it's for and tradeoffs
Great fit if you want to benchmark or develop autonomous research agents, compare prompt/scaffold effects, or run reproducible evaluations across many scientific tasks. The dataset removes the burden of instance generation so teams can focus on agent design and evaluation pipelines.
Look elsewhere if you need full evaluation stacks (the dataset supplies instances and references but not private evaluation assets or centralized scoring servers), or if you require human-curated gold-standard solutions for every instance—some reference artifacts are provided, but authoritative scoring infrastructure is separate.
Where it fits
Use this alongside ASI‑Bench code and evaluation harnesses to run agentic experiments, ablations over prompt conditions, or leaderboard submissions. It complements other research-focused benchmarks that target specific ML tasks by stressing long-horizon, multi-step scientific workflows.
