Why this matters Many gains attributed to larger models actually come from how models are embedded in runtime harnesses that structure context, tools, memory, and control. JIT-Agent treats harness design as a trainable capability: instead of hand-crafting runtimes per model or task, it trains a dedicated model to generate, repair, and self-evolve harnesses on the fly so that an unchanged foundation model can behave differently across tasks.
Key Findings
- Trains a harness-intelligence model that emits harnesses under a fixed four-module protocol and refines them using performance signals archived from prior runs — so harness improvements compound over time rather than being handcrafted.
- Shows sizable empirical gains: JIT-Agent-equipped agents surpass baseline strong models on multiple benchmarks (examples reported: +9.1 on DeepSearchQA and +4.3 on OdysseyBench; GLM-5.2 reported gains up to +20.2 points), and consistently improves families such as DeepSeek V4, Mimo-V2.5, and Qwen3.6.
- Supports harness customization (task-specific generation), automated repair for execution stability, and self-evolution via distillation of past harness configurations, enabling transfer across tasks and models.
Who it's for and trade-offs
Great fit if you maintain or evaluate multi-model agent fleets and want a repeatable way to improve runtime behavior without model fine-tuning. It suits teams that can instrument execution traces and archive harness configurations for continual distillation. Look elsewhere if you cannot collect or store execution traces, require strict provenance for every runtime change, or if latency and compute cost of a harness-generation step are hard constraints—JIT-Agent adds an on-the-fly generation layer that incurs overhead and depends on a training loop for the harness model.
Where it fits
Positions harness design as an orthogonal scaling axis to model size: instead of only investing in larger LLMs, teams can invest in harness intelligence that amplifies existing models, particularly in tool-using, long-horizon, and retrieval-augmented agent setups.
Mechanism overview
At a high level JIT-Agent: (1) encodes task context and past harness archive, (2) generates a four-module harness specification, (3) optionally repairs harnesses after validation failures, and (4) distills successful harnesses back into the archive for future generations. The paper evaluates this pipeline across controlled benchmarks and multi-model families to quantify transfer and compounding benefits.