Long-running coding agents repeatedly replay context, rerun verification steps, and reread large tool outputs — most savings come from changing the harness lifecycle, not the model. SoL‑Pi treats harness improvement as a searchable, recursive self-improvement problem: generate many candidate harness changes, pre-screen them, and validate survivors across diverse executable environments so improvements transfer beyond development settings.
Key Findings
- Empirical gains: On a 51-task EdgeBench suite, SoL‑Pi retains roughly 94% of Pi's average score while reducing recorded token traffic by about 44.7–49.0% and cutting API cost by roughly one third.
- Mechanisms discovered: Four narrowly scoped, composable mechanisms survived the automated search — Action Fusion (combine edit+test into one tool call), Online Context Compact (semantic compaction at subtask boundaries), ObservationPack (archive large payloads with lightweight handles and paged recall), and Evidence-Preserving Reducer (delegate first-pass log reading to cheaper agents with verifiable receipts).
- Cost implications: Measured savings translate to estimated hourly cost reductions for a professional researcher on the evaluated backends (ranges reported per model/backend), demonstrating practical token- and cost-efficiency without large capability regressions.
Who it's for & trade-offs
Great fit if you run or research long-horizon coding agents and care about reducing API/token spend without substantially lowering task completion or evidence retention. SoL‑Pi is useful when harness-level changes are allowed and you can integrate optional mechanisms into Pi-style harnesses. Look elsewhere if your primary bottleneck is model quality rather than harness overhead, if you cannot modify the harness, or if strict reproducibility requires every tool output to be replayed into model context (some mechanisms archive content and recall pages on demand).
Where it fits
SoL‑Pi is a harness-level efficiency layer, not a new agent model. It complements improvements in base LLMs and retrieval stacks by reducing redundant context and model round-trips. Use it when operational cost and token throughput matter for continuous or unattended agent runs.
Mechanisms (brief)
- Action Fusion: collapse predictable edit→test sequences into a single tool invocation to avoid an extra model round-trip.
- Online Context Compact: trigger context compaction at semantic subtask boundaries when future savings justify the rewrite cost.
- ObservationPack: replace repeated full-text replays with stable handles and paged recall, keeping exact content available without replaying it into every request.
- Evidence-Preserving Reducer: delegate initial log scans to cheaper agents and bind verified receipts to archived logs so frontier agents only see quoted, verified lines.