Why this matters
Most capability transfer relies on training-time distillation or weight updates. This work shows an orthogonal lever: design the execution harness at test time so a stronger model can scaffold a weaker one. The surprising core insight is that much of what looks like ``model thinking'' can be offloaded into deterministic program logic and routing rules, producing large, persistent gains without retraining.
Key Findings
- Large test-time gains: Using a small validation set (5%) to iteratively refine harnesses, builder models produced inference-time harnesses that nearly doubled average target-model accuracy (from ~0.49 to ~0.91) across four Theory-of-Mind benchmarks. This improvement persists when the finalized harness is applied to the full test set.
- Mechanism of improvement: Most gain comes from converting brittle probabilistic reasoning into deterministic code paths, benchmark-specific routing, and strict answer-format enforcement rather than encouraging the target model to sample or reason more extensively.
- Monotonic effect of builder effort and platform robustness: Stronger or more deliberative builder reasoning produces better harnesses; differences between execution platforms are modest compared to the influence of builder capability. The weakest target models benefit the most in absolute terms.
- No parameter updates required: All adaptation occurs by changing the surrounding executable harness; solver and target models remain frozen, and no labeled supervision is needed beyond using execution traces for refinement.
Who should care and trade-offs
Great fit if you can afford to run a stronger builder model at validation/test time and your task has repeatable workflows or strict output formats (e.g., benchmarks, structured QA, tool-using agents). This approach is attractive when retraining is expensive or infeasible and when failures stem from orchestration/formatting rather than missing internal knowledge.
Look elsewhere if the task requires genuine internal model capability that cannot be encoded as deterministic scaffolding (open-ended creative tasks, continual learning scenarios), or when deploying a heavier builder model at inference time is impractical for latency or cost reasons. Harness design also depends on reliable execution-derived proxy signals; judge reliability can bottleneck unsupervised refinement.
Where it fits
Positions test-time harness engineering as a complementary axis to traditional distillation and parameter-efficient methods: instead of changing model weights, modify the program that manages context, verification, and recovery. For practitioners, this suggests a practical route to make smaller or cheaper models perform like larger ones on structured tasks by investing effort into harness design rather than into further training.