Early environment changes can steer agent behavior far into the future; evaluating agents only on short, static tasks misses these cumulative effects. This work operationalizes that insight by treating the environment state as the primary red‑teaming unit and by evolving target‑visible state (while keeping objectives and hidden safety contracts fixed) to systematically increase attack difficulty without changing semantics.
Key Findings
- Scalable dataset: generates and validates over 10,000 stateful scenarios spanning 50 domains, synthesized from a corpus of 500K+ Tools, MCPs, and Skills. Tasks are long‑horizon (median 97 tool calls) and target agent–model pairs through lightweight adapters to form 75 agent–model configurations.
- EMHA attack policy: introduces Evolutionary Markov Hypergraph Attack (EMHA), a black‑box, feedback‑driven policy that coordinates authorized state transitions along hypergraph paths to evolve environments without model parameter updates.
- Empirical impact: across all configurations, EMHA attains a pooled strict Attack Success Rate (ASR) of 85.0%. Environment evolution yields small gains on simple tasks (≈1.8–2.7%) but large gains on the most complex tasks (≈17.2–17.6%), showing long‑horizon workflows amplify the effects of state evolution.
- Implementation matters: after controlling for model and benign task completion, target‑agent identity explains an additional 7.6% of ASR variation, indicating runtime implementations significantly influence safety outcomes.
Who it's for and tradeoffs
Great fit if you evaluate or develop tool‑using agents and need a unified, high‑complexity benchmark to surface cumulative, stateful failures. The arena is useful for red‑teamers, safety researchers, and teams comparing runtime implementations under identical task semantics. Look elsewhere if you need real‑world deployment monitoring (OpenART focuses on controlled, executable scenarios rather than production telemetry), or if your target agents cannot be instrumented to support the lightweight adapters and tool/skill interfaces the benchmark assumes.
Where it fits
OpenART complements short, static benchmarks by focusing on persistent state evolution and long interaction horizons. It is positioned as a scalable research platform and benchmark rather than a turnkey production safety system.
How EMHA and environment evolution work
EMHA models environment evolution as hypergraph traversal: evaluator feedback authorizes or rejects candidate state transitions, and EMHA iteratively refines future transitions using past success signals. The pipeline keeps benign objectives and hidden safety contracts fixed while only mutating the target‑visible state, enabling controlled difficulty escalation without altering task semantics.