MMLU-Pro matters because existing multi-domain benchmarks have become saturated: simple knowledge recall no longer distinguishes advanced LLMs. By substantially increasing distractors and integrating reasoning-heavy problems, MMLU-Pro forces models to rely less on surface cues and more on multi-step reasoning.
What Sets It Apart
- Ten-option multiple-choice format (typical) instead of four, reducing random-guess probability and raising task difficulty. This change alone makes accidental success far less likely and increases discriminative power in leaderboards.
- Reasoning-first construction: a mix of original MMLU questions (cleaned), curated STEM problems, TheoremQA items, and SciBench exam questions yields a dataset with a stronger reasoning component where Chain-of-Thought (CoT) prompting consistently outperforms direct prompting.
- Expert-in-the-loop curation: options were augmented (GPT-4 assisted) then vetted by >10 domain experts; ambiguous or erroneous items were removed or corrected in ongoing maintenance.
Core Details
- Scale and scope: 12,032 questions across 14 disciplines (math, physics, chemistry, law, engineering, biology, economics, psychology, business, health, computer science, philosophy, history, other).
- Observed effects: reported model accuracy drops of roughly 16–33% compared to original MMLU, and prompt-sensitivity decreased (score variance across prompt styles fell from ~4–5% to ~2%), indicating greater stability and discrimination. CoT prompting yields substantially better results than direct prompting on this dataset.
- Construction notes: initial filtering removed trivial/noisy MMLU items; choices were expanded to ten with plausibly hard distractors; dataset files include question text, options, answer index, CoT content, category, and source. A public leaderboard and evaluation scripts are maintained for reproducible benchmarking.
Who It's For and Trade-offs
Great fit if you need a tougher, more discriminative benchmark to compare LLM reasoning capabilities across scientific and professional domains, or to validate CoT-style prompting and model robustness. Look elsewhere if your goal is small-scale, knowledge-only evaluation or quick sanity checks: the larger option set and reasoning focus make per-question evaluation slower and can be harsher on models that were tuned primarily for recall rather than multi-step inference.
