Agent evolution can look like progress while an agent’s actual capability barely changes: repeated selection on noisy scores favors lucky results as well as genuine improvements. This study’s central finding is that detecting gains—not merely proposing candidates—is often the bottleneck.
Key Findings
- Across seven evolution methods and six models, candidate selection was often no better than chance at distinguishing helpful artifacts from harmful ones; execution-based selection chose the better artifact in 49.7% of head-to-head comparisons.
- Method rankings were also unstable. In more than 60% of configuration pairs, the difference between methods was smaller than the noise in a single-run comparison, making one-run leaderboards unreliable.
- On re-evaluation, apparent in-loop gains of 6.9–9.0 percentage points shrank to between +0.4 and −3.2 points. The acceptance process can accumulate lucky scores without accumulating real capability.
- The practical implication is a shift in research priorities: improve evaluation resolution and repeat experiments enough to separate real gains from variance before claiming one evolution method is better.
Who Should Read It—and the Trade-Off
A useful read for researchers building self-improving agents or comparing prompt, skill, memory, and harness optimization methods; it gives concrete reasons to measure run-to-run variance rather than trust a single score. Its evidence is not a verdict on every kind of learning: model weights stay frozen, and the candidate-level analysis is centered on Terminal-Bench, so the findings may not transfer directly to weight-updating reinforcement learning or other benchmark domains.