Why this matters now Most harnesses are handcrafted and fixed, so they either over-block some models or under-protect some domains. EvoSafeHarness treats the harness as a tunable, search-able artifact: it jointly synthesizes a natural-language policy and executable enforcement logic tailored to a frozen model and target domain, guided by model rollouts, domain semantics, and adversarial review.
Key findings
- Joint search of policy text and executable code yields stronger safety–utility frontiers than one-size-fits-all defenses: on DecodingTrust-Agent, average attack success rate (ASR) falls from 45.6% to 10.0% at a 3.3-point utility cost. This shows targeted harnesses can dramatically cut attacks with modest utility loss.
- On AgentDojo, EvoSafeHarness attains 82.8% benign utility at 0.0% ASR, outperforming comparable defenses at the same operating point and transferring unchanged to unseen AgentDyn suites.
- Robustness under adaptive attacks: the method keeps mean ASR below 20% under adaptive PAIR attacks with a refinement budget of 16, indicating the evolved harnesses generalize beyond benchmark-specific rules.
- Analysis insight: domain semantics determine which safety relations and trajectory state must be monitored; model and runtime behavior determine where and how strict enforcement should be applied. In short, “what to check” is domain-driven while “how strictly to check” is model-driven.
What sets it apart
- Co-optimization of natural-language rules and executable logic so the harness is both interpretable and enforceable.
- Guided generation using fresh-context adversarial review to avoid brittle, benchmark-overfitted rules.
- Designed for deployment: produces artifacts that can be validated and executed in harness pipelines and that transfer across models and related domains without retraining the base LLM.
Who should care and trade-offs
Great fit if you deploy LLM agents across heterogeneous models or domains and need a deployable, inspectable safety layer that balances utility and safety per deployment. Look elsewhere if you require zero-runtime changes to the harness (EvoSafeHarness assumes the harness can be updated) or if you cannot run the candidate-generation and validation loop (search and adversarial review add compute and engineering overhead).