Agent self-improvement is hard to compare when each method brings its own execution and scoring machinery—and especially when a method can alter the evaluator that judges its changes. The central design choice here is to make the evaluator and evidence archive framework-controlled, so experiments can vary the evolution policy without quietly changing the rules of success.
Key Findings
- Reusable method components: Recipes combine lifecycle operators and declare which resources may change. This lets researchers compare distinct evolution policies without rebuilding the experimental pipeline for each one.
- A fixed basis for comparison: The paper instantiates AHE, Hyperagents, A-Evolve, and GEPA without modifying the framework core, and evaluates them under fixed model and evaluator configurations. This helps distinguish policy differences from infrastructure differences.
- Meaningful but uneven gains: Across MiniSWE and Codex seed agents on Terminal Bench 2 and τ³ Banking, the largest reported overall pass@1 gain is 13.5 percentage points. Rankings vary across settings, and gains do not reliably carry over to held-out tasks—so the headline result is not evidence that one strategy dominates generally.
- Inspectable experiment history: Candidate changes, evaluation conditions, and outcomes are linked through lineage. This makes failed trials and provenance part of the research record, not just the final score.
How the Boundary Works
A recipe defines the mutable surface and evolution policy; the framework controls execution, evaluation, recovery, and archive writes. Since candidates cannot change the scoring contract, a reported improvement is more interpretable. This boundary is the paper’s key contribution: it makes agent evolution a more reproducible experimental subject rather than merely a collection of optimization tricks.
Who It’s For
A good fit for researchers comparing agent-improvement methods, or builders who need auditable candidate histories across repeated runs. Look elsewhere if you need a turnkey agent product, or if your goal is to demonstrate broad generalization: the reported results are benchmark- and configuration-dependent, and transfer to held-out tasks is not guaranteed.
