AIAny
Icon for item

ASI-Bench Generated Instances (Seed 42)

Provides fixed-seed benchmark instances (prompts and agent-visible inputs) for ASI-Bench to run reproducible evaluations of LLM agents on scientific tasks. Includes four matched prompt levels (B1–B4) across 60 project-level tasks in 11 domains; excludes reference answers and private scorers; Apache-2.0 licensed.

Introduction

Why this matters

Benchmarks for AI-driven science require agent-facing inputs and controlled variation in how much methodological guidance is given. This seed-42 package supplies one deterministic instance per final task so researchers can run repeatable evaluations of agent autonomy and compare performance when procedural guidance is present (B1) versus progressively removed (B2–B4).

What Sets It Apart
  • Fixed-seed, agent-visible artifacts: each task folder contains four prompt variants and the data the agent can access, enabling identical inputs across runs for fair comparisons.
  • Project-level, multi-domain scope: 60 tasks spanning 11 scientific domains focus on long-horizon research workflows rather than isolated microbenchmarks, emphasizing planning, experimentation, and result validation.
  • Designed for execution testing: intended to be run by agent-execution frameworks (sandboxed runners) and paired with private reference answers/scorers for official evaluation; the public package omits those private assets to allow open experimentation while preserving official scoring integrity.
Who It's For and Tradeoffs

Great fit if you evaluate or develop LLM-based research agents, compare prompt-level robustness (B1–B4), or need reproducible agent-facing inputs across many scientific tasks. Look elsewhere if you only need small-scale unit tests or labeled gold-standard answers — the public seed set omits reference answers and official scorers, so closed-loop benchmarking against official metrics requires access to the private evaluation service used by ASI-Bench.

Where It Fits

Use this dataset as the deterministic input layer in agent evaluation pipelines, e.g., local development, ablation studies, and cross-agent comparisons. For leaderboard-grade scoring or to reproduce published official results, pair these instances with the ASI-Bench scoring backend and reference assets used by the project maintainers.

Information

  • Websitehuggingface.co
  • OrganizationsApexintelligence-AI
  • Published date2026/07/28

Categories

More Items

Hugging Face

Evaluates whether GUI agents can locate interface elements from Korean instructions and website screenshots. Its 3,319 human-annotated examples span 142 websites, providing a focused test of visual grounding on Korean web interfaces.

Hugging Face

Evaluates document parsers across text, formatting, tables, charts, layout grounding, and forms. Severity-aware scoring prevents a small number of critical errors from being hidden by otherwise high page-level accuracy.

Hugging Face

Provides first-person video of skilled manual work with synchronized motion-sensor data for robot learning and dexterous manipulation research. The first release spans 10,124 hours from 12,227 contributors, with task-level annotations.