High accuracy can hide a model that changes its answer when choices are reordered—or confidently selects an option when none is correct. That gap matters for fast-decision models, where reliability cannot be inferred from benchmark scores alone.
Key Findings
SO-Arena evaluates 2,919 tasks spanning 11 domains and five capability families, combining static judgments with interactive web and tool-use tasks. Its reliability measures test two distinct failure modes: answer instability under controlled option reordering, and failure to recognize that no valid answer is available. Across 21 models, stronger overall performance did not guarantee more reliable decisions; the top-performing System One model ranked only seventh in reliability. The benchmark also compares System One and System Two approaches on reasoning-intensive tasks, showing that scaling and extra reasoning help unevenly across domains.
Who It’s For
A good fit for researchers comparing fast-response models, studying abstention and answer consistency, or evaluating agents beyond static accuracy. Look elsewhere if you need a general-purpose leaderboard or a single score that predicts deployment safety: the benchmark’s domain-specific tasks and separate reliability measures are diagnostic, not a substitute for evaluating a model in its intended setting.