Most LLM agent failures on long-horizon tasks come from small local errors that compound across steps; addressing them requires more than per-trajectory fixes. AutoSaddler reframes harness design as an offline learning problem: it collects execution traces, diagnoses deep failure modes across batches, generates targeted code-like patches to the harness (prompts, tool configs, middleware), and accepts updates only after validation to ensure generalization beyond individual trajectories.
Key Findings
- Consistent gains across benchmarks: iterative harness updates yield +9.0, +9.6, and +10.0 percentage points on GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0 respectively, over corresponding base harnesses. This demonstrates durable improvement on long-horizon tasks rather than trajectory-specific fixes.
- Three practical design principles: (1) deep debugging — analyze multi-step failure traces rather than shallow reflections, (2) structured interventions — treat harnesses as editable code and make targeted, constrained patches across prompts, tools, and middleware, and (3) generalization-aware selection — validate candidate updates on held-out tasks to prefer broadly useful changes.
- Method pipeline: accumulate traces in mini-batches, diagnose failed trajectories to extract root causes, generate structured candidate patches guided by diagnosis, evaluate each candidate on training mini-batch and a validation set, and commit updates that improve validation performance.
Who it's for and trade-offs
Great fit if you maintain LLM agents for long-horizon or multi-step workflows and want a principled way to harden external harnesses without exhaustive manual prompt/tool search. It helps teams that can run batched evaluations and accept harness edits as code. Look elsewhere if you need runtime, per-interaction self-repair from the base model itself (AutoSaddler optimizes external harnesses offline) or if you cannot run validation suites to test generalization. The framework focuses on harness-level fixes and does not replace agent model fine-tuning or tool reengineering; it complements those approaches by making harnesses more robust and maintainable.