Most existing agent benchmarks rely on sandboxed or single‑turn tasks that conflate multiple capabilities and obscure why agents fail. UniClawBench addresses this by driving evaluation around five explicit capabilities and running 400 bilingual, real‑world tasks in live Docker containers with stepwise completion checkpoints and a closed‑loop grading setup (executor agent, hidden supervisor, user agent). The key insight: real‑world agent performance emerges from an interaction between base model capabilities and agent‑framework design, so evaluating them separately reveals different bottlenecks.
Key Findings
- Capability-driven taxonomy: Tasks are organized by Skill Usage, Exploration, Long‑Context Reasoning, Multimodal Understanding, and Cross‑Platform Coordination — this makes failures diagnosable (so what: you can tell whether poor performance stems from perception, long‑horizon planning, or interface handling).
- Live, stepwise evaluation: Running tasks inside isolated Docker environments with incremental checkpoints prevents static gold‑answer bias and captures partial progress (so what: supports measuring intermediate competence rather than binary pass/fail).
- Closed‑loop grading design: Using an executor agent, a hidden supervisor, and a user agent simulates multi‑turn human feedback without leaking criteria (so what: yields more realistic interaction traces and makes automated scoring more robust to prompt leakage).
- Framework vs base model tradeoffs: Experiments show both base LLM/multimodal capabilities and agent orchestration choices materially affect outcomes (so what: improving orchestration alone may not overcome foundational model gaps, and vice versa).
Who it's for and tradeoffs
Great fit if you need diagnostic, real‑world evaluation of proactive agents — for example, authors of agent frameworks, LLM/multimodal model developers, and benchmark researchers who care about stepwise behaviours and cross‑platform workflows. Look elsewhere if you need lightweight, large‑scale synthetic benchmarking or very fast iteration: UniClawBench requires running isolated containers and orchestrated multi‑agent loops, which increases compute and engineering overhead. It is bilingual and focused on practical tool use and coordination, so domains outside interactive tool use (e.g., pure language modeling or specialized scientific tasks) may be underrepresented.