A higher benchmark score can conceal a less dependable agent: across all 16 evolved agents studied, some previously solved tasks were lost even as new ones were gained. This paper treats evolution not as a one-way accumulation of capability, but as a redistribution of success whose reliability, cost, and transfer behavior must be measured separately.
Key Findings
- Reliability and coverage are different outcomes. Across the tested lineages, aggregate scores often improved while individual task outcomes shifted in both directions. One general-task case gained 5.2 percentage points in reliability yet lost ten previously covered tasks.
- Gains can cost more to deploy. Eight of the eleven reliability-improving champions had higher per-trial costs than their seeds. The paper also shows why cost per attempt alone is incomplete: fewer attempts can sometimes make a more expensive agent cheaper overall for reaching a target level of coverage.
- Improvements do not transfer uniformly. Coverage generally declined on the general-task transfer target, while it rose in most coding-task transfer cases using an outer-layer-only harness. Reliability and coverage could move in opposite directions.
- Evolution rules matter, not just the selected champion. The highest-scoring agent did not produce the largest mean improvement in any of the four evaluation pools. The authors use these diagnostics to develop EvoAudit, which revises selection, analysis, mutation, and admission decisions.
Method and Scope
The study compares AHE, HyperAgents, A-Evolve, and GEPA with GPT-5.4 across MiniSWE and Codex harnesses. MiniSWE permits source-level changes; Codex allows changes to the outer layer, such as prompts, skills, configuration, and orchestration. Evaluation covers general tasks and coding benchmarks, with repeated trials and additional analyses of task-level changes, deployment cost, and transfer.
The results are evidence about these methods, harnesses, model, and benchmarks—not a universal ranking of agent-evolution approaches. The practical takeaway is to report gains and regressions separately, test transfer on the intended target, and account for the cost of reaching useful coverage rather than relying on a single aggregate score.
Who Should Read It
A useful read for researchers building self-improving agents or evaluating agent workflows, especially those deciding how to admit and preserve revisions. It is less suited to readers seeking a new base model or a broad survey of agent research: the contribution is an empirical diagnosis of evolution procedures under a specific experimental setup.