Why this matters
Tight context budgets and harness choices—not just model weights—often decide whether an autonomous coding agent completes long-horizon software tasks. This paper isolates harness components (planning, action space, context management) with a fixed execution loop so we can compare their direct impact on success, cost, and agent trajectories.
Key Findings
- Context management scales with budget pressure: under tight context windows it prevents context-overflow failures and yields the largest accuracy gains; its benefit shrinks as context windows grow, meaning larger windows trade off some need for complex management.
- Staged elision + LLM summarization is most efficient: applying rule-based elision before calling summarization controls peak context and reduces summarization calls while keeping mean success similar to other managed strategies.
- Planning shifts role by model capability: for weaker models planning acts as an accuracy scaffold (keeps trajectories alive to attempt edits) at added cost; for stronger models it mainly reduces redundant verification steps and lowers cost with little accuracy change.
- Action-space / tool design drives granularity and cost: predefined tools help models weak at shell usage; bash-capable models can use a bash-only interface to combine operations, achieving much lower cost on command-line-centric tasks.
Method snapshot
The study uses a lightweight modular harness with a fixed agent loop and controlled ablations across three axes (planning, action space, context management). Experiments span 176 matched settings, four LLMs, and two benchmarks (SWE-Bench Verified, Terminal-Bench 2.1), and report success rates, costs, and trajectory-level behavior to explain why components help.
Who it's for and trade-offs
Great fit if you design or evaluate coding-agent runtimes and need empirical guidance on which harness machinery to invest in under constrained budgets. The paper gives actionable rules: prioritize context management when tokens are scarce, prefer staged elision for efficiency, and match tool interfaces to a model's shell proficiency.
Look elsewhere if your primary concern is raw model architecture or pretraining methods—the paper isolates runtime components and does not propose new model architectures. Also, some proposed machinery (e.g., recoverable elision) adds complexity without consistent accuracy gains, so weigh engineering cost against marginal benefit.
Where it fits
This work complements source-code audits of production harnesses and benchmark studies by treating the harness as a modular object of study. Use it to make model- and budget-aware harness decisions or as a reproducible framework for testing future harness components.