Automating research means more than producing a plausible answer: agents must execute multi-step experiments, generate data and code, and deliver verifiable scientific artifacts end-to-end. FrontierChallenge's public release and evaluation of 97 tasks surfaces a core mismatch—agents often make measurable partial progress and even claim success, yet rarely meet task-level completion criteria.
Key Findings
- Scope and scale: 97 released tasks (subset of a 300-task suite) spanning quantum chemistry, molecular dynamics, materials characterization, analytical chemistry, life science, and electrochemistry/environment.
- Evaluation setup: 12 frontier models tested across 3 agent scaffolds using Pass Rate (full-completion) and Avg. Score (partial progress) metrics.
- Large gap between progress and completion: the best configurations fully completed only 20 of 97 tasks (Pass Rate 20.6%), while Avg. Scores in some domains remained high—analytical chemistry and electrochemistry/environment reached Avg. Scores of 87.6 and 94.9 but Pass Rates of only 4% and 0% respectively.
- Overconfident assertions: among non-passing trajectories from one evaluated model, 75.5% ended with language claiming completion despite failing verification.
These results imply that high partial scores or confident natural-language claims are poor proxies for end-to-end scientific delivery. Benchmarks for research agents therefore need artifact-aware verification, rigorous end-to-end checks, and domain-specific execution monitoring to distinguish partial progress from truly completed workflows.
Who it's for and tradeoffs
Great fit if you want a realistic benchmark to stress-test autonomous research agents' long-horizon execution, artifact completeness, and verification pipelines. The dataset is most useful for researchers building agent verifiers, execution sandboxes, or systems that must produce auditable research artifacts.
Look elsewhere if you only need short QA-style problems, single-step code generation, or benchmarks that judge only final answers—FrontierChallenge emphasizes multi-step workflow completion and requires infrastructure to run, verify, and interpret complex scientific outputs.
