AIAny
Icon for item

The Resolution Hypothesis: Agent Evolution’s Key Is Identifying Improvements

Finds that agent evolution often stalls not for lack of better prompts or skills, but because noisy evaluations fail to identify real improvements. The study tests seven methods across six models.

Introduction

Agent evolution can look like progress while an agent’s actual capability barely changes: repeated selection on noisy scores favors lucky results as well as genuine improvements. This study’s central finding is that detecting gains—not merely proposing candidates—is often the bottleneck.

Key Findings
  • Across seven evolution methods and six models, candidate selection was often no better than chance at distinguishing helpful artifacts from harmful ones; execution-based selection chose the better artifact in 49.7% of head-to-head comparisons.
  • Method rankings were also unstable. In more than 60% of configuration pairs, the difference between methods was smaller than the noise in a single-run comparison, making one-run leaderboards unreliable.
  • On re-evaluation, apparent in-loop gains of 6.9–9.0 percentage points shrank to between +0.4 and −3.2 points. The acceptance process can accumulate lucky scores without accumulating real capability.
  • The practical implication is a shift in research priorities: improve evaluation resolution and repeat experiments enough to separate real gains from variance before claiming one evolution method is better.
Who Should Read It—and the Trade-Off

A useful read for researchers building self-improving agents or comparing prompt, skill, memory, and harness optimization methods; it gives concrete reasons to measure run-to-run variance rather than trust a single score. Its evidence is not a verdict on every kind of learning: model weights stay frozen, and the candidate-level analysis is centered on Terminal-Bench, so the findings may not transfer directly to weight-updating reinforcement learning or other benchmark domains.

Information

  • Websitearxiv.org
  • OrganizationsShanghaiTech University, ByteDance, Beijing Institute for General Artificial Intelligence (BIGAI)
  • AuthorsYuanhao Zeng, Shaocheng Lan, Ao Lu, Kuang Yan, Heng Lin, Xinang Xu, Yuan Luo, Yueting Wang, Yexin Li, Ruo Shan …
  • Published date2026/11/17

More Items

Scales verifiable gameplay data by having a Builder and visual and coding players co-develop web games. Progress-weighted training improves agent performance on both its own benchmark and an external game benchmark.

Studies scaling language models with parallel Transformer branches and introduces Joint-KV to share key-value caches across branches. Experiments report improved modeling quality at comparable training compute and faster long-context decoding.

Examines whether Transformers are necessary when LLMs write model weights without gradient training. When allowed to design the architecture, the tested LLMs compile facts into sparse tables, reaching up to 54.4% accuracy on ARC.