Most agent benchmarks report final scores, which hide where improvements originate and whether accumulated experience actually helps. This paper argues that diagnosing research-loop behavior requires within-run, rule-based metrics and controlled comparisons; applying this view reveals where current agents succeed, fail, and transfer experience (for better or worse).
Key Findings
- Within-run decomposition matters: measuring Solution Framing, Execution, and Feedback Control uncovers distinct process bottlenecks that final scores mask, so one model’s strong final result can stem from very different internal behaviors.
- Experience reuse is unstable: agents sometimes leverage prior runs to improve decisions but can also propagate misleading artifacts across tasks, so selective memory mechanisms are needed to realize consistent gains.
- Harness design affects reliability more than raw capability: shared tooling and iteration policies change performance stability and reproducibility, so engineering the harness is as important as model choice.
- Methodological novelty is rare: the best solutions typically adapt or combine established techniques rather than invent new algorithms, implying current systems function more like optimizers than independent researchers.
Who it's for and tradeoffs
Great fit if you care about building or evaluating autonomous ML/engineering agents and need diagnostic metrics beyond final-task scores. The paper gives concrete, testable metrics and an experimental protocol (36 long-horizon tasks, 3 rollouts each) that help pinpoint where to intervene (training, inference-time selection, memory policies, harness design). Look elsewhere if you only need single-turn benchmarks or short-horizon agent traces—the framework targets sustained iterative improvement and requires more complex verifiers and wall-clock budgets.
Method and scope
The study evaluates seven frontier models across 36 expert-curated long-horizon tasks with rule-based verifiers and three complementary capability axes (Solution Framing, Execution, Feedback Control). It reports avg@3 and best@3 per model-task pair and includes controlled intra- and inter-task comparisons to measure experience transfer and harness effects. The protocol emphasizes deterministic, verifier-driven metrics over subjective LLM judgments to enable fine-grained diagnosis and actionable recommendations for agent design.