AIAny
Icon for item

ASI-Bench: At the Dawn of Artificial Superintelligence

Evaluates whether AI systems can independently carry out project-level scientific research by progressively removing human methodological guidance across 60 tasks in 11 domains. Built with expert review, sandbox execution, and multi-agent–model scoring to measure innovation and autonomous experimental execution.

Introduction

Why this matters

Most existing AI benchmarks test knowledge retrieval, single-step reasoning, or tool use under heavy human guidance. The critical gap is whether models can originate novel ideas, choose methods, run experiments, and produce verifiable results with diminishing human direction. ASI-Bench attacks that gap by turning whole research projects into benchmark tasks and gradually withdrawing methodological guidance to reveal how far current systems can autonomously progress.

Key Findings
  • Project-level design: 60 research projects spanning 11 scientific domains, each evaluated end-to-end so that success requires method selection, experiment design, execution, and verification — not just single-answer generation. So what? This forces evaluation of long-horizon planning and iterative experimentation rather than isolated reasoning steps.

  • Degraded performance with less guidance: across 18 agent–model configurations, average score falls from 50.91 (full guidance) to 29.10 (only method specified) and 26.62 (agent chooses method). So what? Modern agents remain heavily dependent on human methodological scaffolding and struggle to carry projects end-to-end.

  • Rigorous pipeline: tasks were constructed with >31,000 human hours, expert review, AI-assisted auditing, sandbox execution, and scorer validation. So what? Results are less likely to reflect env or evaluation artifacts and more likely to expose true gaps in autonomous scientific capability.

  • Open, extensible benchmark: tasks and submission channels are open for community contributions. So what? Provides a shared, evolving testbed to track progress toward more autonomous, creative AI research.

Who it's for & tradeoffs

Great fit if you develop or evaluate agentic research systems, study AI autonomy or safety, or need a stress test for long-horizon, experimental workflows. The benchmark is useful for measuring innovation-oriented abilities (method selection, experiment iteration, reproducibility).

Look elsewhere if you need light-weight, low-cost evaluation: ASI-Bench is costly to run at scale (sandboxed experiments, human review) and focuses on project-level science rather than single-task efficiency or throughput. Also, because tasks emphasize verifiable experiments, they may favor domains with reproducible, sandboxable workflows over purely conceptual or purely production-oriented tasks.

Where it fits

Compared with other agentic benchmarks, ASI-Bench uniquely emphasizes progressive withdrawal of methodological guidance and end-to-end project success as the unit of evaluation. Use it when your goal is to probe an agent's capacity for autonomous scientific discovery and experimental execution rather than isolated planning or tool orchestration.

Information

  • Websitearxiv.org
  • AuthorsJunwei Zhou, Zhen Sun, Binyu Li, Jiangyu Zhou, Yuexi Pan, Hengyu Wang, Honghe Ren, Xiaohan Jia, Xueyang Zhou, Xiaoyu Cao …
  • Published date2026/08/18

Categories

More Items

Blocks tool-using agents from directing emails, payments, and other side effects to unauthorized targets by checking typed tool-return fields. Deterministic rules also remove instruction-like text before the model reads it.

Provides six open language models from 0.9B to 375B-A23B for reasoning and agentic tasks. The paper also introduces sparse attention and a decoding speedup that needs no separate draft model.

Proposes a framework for robots that combine foundation models with an embodied runtime to interpret intent, call skills, verify outcomes, and adapt. It also outlines an evaluation framework and a roadmap toward sustained autonomy.