The key problem this benchmark targets is not just accuracy but calibrated probabilistic outputs across multiple, typed decisions over the same input. Practical systems often need a distribution (how sure is the model?) rather than a single label; Typed Decisions forces models to return those distributions and measures whether a shared encoder plus typed outputs improves the accuracy/calibration/latency trade-off.
What Sets It Apart
- Focus on typed probabilistic outputs: each question is
noul(yes/no probability),choice(label distribution + confidence), orscore(ordered rubric with an expected score). This makes calibration and full-distribution scoring first-class objectives rather than afterthoughts. - Shared-state, multi-question cases: each case contains one
stateand 5 questions, so models must amortize encoding cost while producing multiple correlated distributions. - Soft gold from teacher sampling: gold labels are the mean of three teacher samples (temperature 0.7), so evaluations reward reproducing label uncertainty (report KL, Brier, log loss, ECE alongside accuracy).
- Compact benchmark slice and clear ceilings: the
testsplit is 400 cases (2,000 decisions) across four workflows (agent traces, customer service, invoices, security incidents), with reference ceilings (perfect scenario understanding, teacher self-agreement) to contextualize scores.
Who it's for and tradeoffs
Great fit if you build or evaluate systems that must output calibrated probabilities (System One-style APIs, decision engines, automated triage). It is especially useful to compare specialist classifiers (fitted per-workflow) versus generalist System One models scored zero-shot. Look elsewhere if you only need argmax labels or generative text evaluation—this benchmark penalizes models that ignore distributional uncertainty and rewards those that match the teacher's spread. Note: the gold measures agreement with a teacher endpoint, not absolute ground truth; high scores can reflect learning teacher idiosyncrasies.
