AIAny
Icon for item

ClawGym II: Exploring Black-Box RL on Agent Harness

Presents a unified black-box reinforcement learning framework to train and optimize agents running inside complex execution harnesses. Uses sandbox-parallel rollouts, a serving proxy that captures model calls and reconstructs multi-turn trajectories as prefix trees, and adapted GRPO/PPO optimizers to achieve stable, scalable RL across heterogeneous harnesses.

Introduction

Agent harnesses (or inference harnesses) have become central to modern agent performance, but their complexity breaks assumptions used by standard RL pipelines. This work argues for a practical, infrastructure-aware recipe: keep the harness opaque at execution time, capture model interactions at the boundary, and recover usable trajectories for policy optimization. That combination preserves training–inference consistency while enabling large-scale, concurrent rollouts.

Key Findings
  • Black-box training is practical: isolating each task+harness in short-lived sandboxes allows thousands of concurrent, interference-free rollouts, which keeps infrastructure costs and failure modes manageable.
  • Trajectory recovery matters: a proxy that records model calls and organizes them into prefix trees reconstructs multi-turn interaction traces efficiently, improving sample utilization compared with naively logging final outcomes only.
  • Algorithm adaptations are effective: adapting both critic-free GRPO and critic-based PPO to operate over recovered tree-structured trajectories yields steady, non-divergent improvements (e.g., substantial Pass@1 gains on ClawGym-Bench using Qwen3-30A3B through different harnesses).
  • Mix-harness training works: a single policy can be jointly optimized across heterogeneous harnesses (OpenClaw, Claude Code, etc.), enabling broader generalization without per-harness retraining.
Who this helps and tradeoffs

Great fit if you need to train or improve agent policies that execute through complex, stateful harnesses (multi-turn tool use, GUIs, or multi-process orchestrators) and you must preserve the real harness at inference time. The approach reduces coupling between harness engineering and RL trainer design, and it scales across clusters via sandboxing.

Look elsewhere if you require dense, step-level supervision derived from internal harness signals (the black-box stance deliberately avoids instrumenting harness internals), or if your environment cannot be reliably snapshotted into isolated sandboxes. The method relies on a robust call-capture proxy and deterministic-enough harness execution to make recovered prefix trees meaningful, so highly non-deterministic harnesses may limit gains.

Where it fits

This paper sits between engineering-first harness work (which prioritizes inference capability) and classical RL research (which assumes environment-level observability). It is most useful for teams that deploy harnessed agents in production and want an empirically validated, scalable RL path that does not require reimplementing the harness as an RL-native environment.

Information

  • Websitearxiv.org
  • AuthorsHuatong Song, Fei Bai, Ming Yang, Renyuan Li, Jia Deng, Jujie He, Zhange Zhang, Daixuan Cheng, Yan Xing, Qi Yun …
  • Published date2026/08/17

More Items

Proposes a framework for robots that combine foundation models with an embodied runtime to interpret intent, call skills, verify outcomes, and adapt. It also outlines an evaluation framework and a roadmap toward sustained autonomy.

Turns screen observations into keyboard and mouse actions to complete long-horizon game tasks. Sparse reasoning, persistent memory, and online reinforcement learning support real-time interaction and multi-step decisions.

Tests decision models as actors, trajectory judges, and reinforcement-learning teachers for long-horizon agents. It finds that a model can be a weak actor yet still improve policy learning by evaluating actions after seeing their consequences.