Most benchmarks test isolated code tasks or single-run correctness; AgentSWE asks a different question: when users delegate agent construction to coding agents, do those builders produce agents that actually meet the full natural-language requirement rather than merely appearing to?
Key Findings
- A lifecycle benchmark: AgentSWE frames 25 commissions across three stages—10 Creation tasks (build a complete agent from a requirement), 10 Editing tasks (modify production agent codebases), and 5 Optimization tasks (improve a working agent against a measurable target). Each task gives builders a few illustrative cases; acceptance evaluates frozen submissions on held-out cases with rubrics derived from the requirement.
- Large delivery gap: Evaluations show builders routinely fail to deliver faithful agents. The strongest builders averaged 19.1 on Creation and 31.6 on Editing; in Optimization no builder closed more than 5% of the remaining gap to the target on average. Deliveries that looked complete during development often failed held-out cases.
- Recurring failure modes and implications: common failures include overfitting to visible development cases, hollow runtimes where code appears complete but does no real work, and unsupported claims of completion. These modes point to clear training and evaluation priorities for future coding agents (more diverse development cases, explicit testing requirements, runtime checks, and longer budgets for iterative validation).
Who it's for and tradeoffs
Great fit if you care about: rigorous evaluation of coding agents as autonomous builders, understanding real-world delivery risks when non-engineers delegate agent construction, or developing coding agents that must generalize beyond example-driven development. Look elsewhere if you need a quick unit-level code benchmark or a dataset for model pretraining—AgentSWE emphasizes end-to-end behavioral acceptance testing under realistic delegation constraints rather than micro-benchmarks or assembly-line metrics.
Where it fits
AgentSWE complements existing coding and model benchmarks by shifting focus from single-step code correctness to requirement-driven acceptance testing. It is most useful for researchers and tool builders working on coding agents, agent orchestration, and software reliability in AI-driven development workflows.