Agent harnesses (or inference harnesses) have become central to modern agent performance, but their complexity breaks assumptions used by standard RL pipelines. This work argues for a practical, infrastructure-aware recipe: keep the harness opaque at execution time, capture model interactions at the boundary, and recover usable trajectories for policy optimization. That combination preserves training–inference consistency while enabling large-scale, concurrent rollouts.
Key Findings
- Black-box training is practical: isolating each task+harness in short-lived sandboxes allows thousands of concurrent, interference-free rollouts, which keeps infrastructure costs and failure modes manageable.
- Trajectory recovery matters: a proxy that records model calls and organizes them into prefix trees reconstructs multi-turn interaction traces efficiently, improving sample utilization compared with naively logging final outcomes only.
- Algorithm adaptations are effective: adapting both critic-free GRPO and critic-based PPO to operate over recovered tree-structured trajectories yields steady, non-divergent improvements (e.g., substantial Pass@1 gains on ClawGym-Bench using Qwen3-30A3B through different harnesses).
- Mix-harness training works: a single policy can be jointly optimized across heterogeneous harnesses (OpenClaw, Claude Code, etc.), enabling broader generalization without per-harness retraining.
Who this helps and tradeoffs
Great fit if you need to train or improve agent policies that execute through complex, stateful harnesses (multi-turn tool use, GUIs, or multi-process orchestrators) and you must preserve the real harness at inference time. The approach reduces coupling between harness engineering and RL trainer design, and it scales across clusters via sandboxing.
Look elsewhere if you require dense, step-level supervision derived from internal harness signals (the black-box stance deliberately avoids instrumenting harness internals), or if your environment cannot be reliably snapshotted into isolated sandboxes. The method relies on a robust call-capture proxy and deterministic-enough harness execution to make recovered prefix trees meaningful, so highly non-deterministic harnesses may limit gains.
Where it fits
This paper sits between engineering-first harness work (which prioritizes inference capability) and classical RL research (which assumes environment-level observability). It is most useful for teams that deploy harnessed agents in production and want an empirically validated, scalable RL path that does not require reimplementing the harness as an RL-native environment.