A model can often pick the right answer yet still be unreliable when it must recover every valid option or express uncertainty accurately. That gap matters when model outputs drive tools, workflows, or scored judgments—not just benchmark accuracy. This evaluation makes those distinctions measurable across varied decision settings.
Key Findings
- The benchmark contains 5,448 questions across 16 categories, with single-select, multi-select, and soft-probability targets. This lets evaluations test more than whether a model can name one plausible answer.
- Option sets range from 2 to 1,024, while inputs reach 64,817 tokens. The combination probes both large choice spaces and decisions grounded in long contexts.
- Across 28 decision models, the strongest reported hard multi-select accuracy is 36.81%. Models can also perform substantially worse on option logic and constraint planning than on world knowledge, highlighting the difficulty of jointly checking interacting conditions.
- The benchmark includes newly constructed tasks alongside source-grounded rewrites and recompositions. This creates fresh decision contexts, though exposure to the underlying public source material cannot be ruled out.
Who It’s For
A strong fit for researchers comparing structured decision models, especially when completeness, constraints, or calibrated probabilities matter. Look elsewhere if you need multimodal or interactive-agent evaluation: this benchmark focuses on text inputs with a fixed state, query, and option set. Its multi-select and soft-label tasks are also concentrated in particular task families, so results should not be treated as a universal measure of decision quality.
