MMLU-Pro: A Reasoning-Centric Benchmark

• MMLU-Pro is an advanced multi-task language understanding benchmark that emphasizes reasoning over memorization by integrating rigorous exam-style questions and a broadened answer set.
• It leverages chain-of-thought prompting to significantly enhance evaluation accuracy, showing improvements up to 19% over direct answer methods.
• Its experimental design reduces prompt sensitivity and differentiates nuanced performance gaps among top language models, ensuring robust and actionable evaluations.

MMLU-Pro is an advanced multi-task language understanding benchmark specifically designed to address limitations observed in earlier benchmarks such as MMLU, particularly the saturation of performance among leading LLMs and their reduced discriminative power. MMLU-Pro achieves this through several structural enhancements, integrating more reasoning-intensive questions, expanding the multiple-choice answer set, and implementing rigorous experimental protocols. Its primary goal is to more accurately evaluate and differentiate evolving capabilities in language comprehension and reasoning across a broad set of domains.

1. Design Principles and Dataset Construction

MMLU-Pro was constructed to overcome two major drawbacks in the original MMLU benchmark: the predominance of knowledge recall over true reasoning, and the presence of trivial or noisy questions. To this end, MMLU-Pro:

• Removes trivial and noisy items that offered little challenge or domain relevance, substantially reducing surface-level knowledge recall as an evaluation metric.
• Curates questions from multiple independent sources—including the STEM Website, TheoremQA, and SciBench—for greater breadth, with the original 57 MMLU categories condensed into 14 broader subject areas.
• Expands the set of answer choices per question from 4 to 10, thereby reducing random chance accuracy from 25% to 10% and compelling models to contend with more plausible distractors.
• Generates additional answer options using automated LLMs (such as GPT-4-Turbo), followed by expert review for plausibility and reasoning depth. This controlled distractor generation avoids cueing shortcut strategies prevalent with shorter options.
• Focuses on exam-style, college-level, and multi-step problems with heightened reasoning requirements, shifting the benchmark away from fact memorization toward complex analytical deliberation.

2. Reasoning-Focused Evaluation Protocol

MMLU-Pro emphasizes questions where deduction, multi-step logic, and explicit chain-of-thought (CoT) processes are necessary for success:

• The majority of questions require explicit reasoning, including mathematical and domain-specific multi-hop argumentation, rather than direct recall.
• Exemplars in physics and mathematics employ advanced formulas—such as the loan amortization equation:

$$M = \frac{P \cdot r \cdot (1 + r)^n}{(1 + r)^n - 1}$$

which enforces comprehensive reasoning to reach a correct solution.

• CoT prompting emerges as a critical evaluation strategy. On the original MMLU, CoT yields only marginal improvement for leading models (e.g., a 1.5% gain for GPT-4o), but on MMLU-Pro reasoning is indispensable—CoT boosts GPT-4o accuracy by up to 19.1%. This robust improvement underscores the intentional design for reasoning-centric model evaluation.

3. Experimental Results and Benchmark Robustness

The experimental protocol is characterized by extensive prompt variation and rigorous accuracy measurement:

• Leading models (GPT-4o, etc.) experience a marked drop in accuracy—from 86–87% on MMLU to ∼72.6% on MMLU-Pro, with losses ranging from 16% to 33% depending on model and domain.
• A suite of 24 distinct prompt styles is used to evaluate response stability. MMLU-Pro demonstrates much lower sensitivity to prompt changes (mean variability ~2%), compared to MMLU’s 4–5% and peaks up to 11%. This enhanced robustness suggests that MMLU-Pro is less vulnerable to surface-level prompt engineering and possesses greater validity for comparative studies.
• Figures labeled “Performance Comparison” and “Performance Variability under Different Prompts” quantitatively reinforce these findings.

4. Expanded Discriminative Power

A core limitation of MMLU was saturation at the top of the leaderboard, compressing score gaps between leading models and challenging the field’s ability to distinguish relative capabilities. MMLU-Pro directly addresses this constraint:

• With harder questions and more distractors, accuracy differentials become more pronounced between high-performing models—for instance, the gap between GPT-4o and GPT-4-Turbo increases from ~2% on MMLU to ~9% on MMLU-Pro.
• This widened performance spread allows detection of nuanced differences in reasoning, generalization, and subject-specific mastery, enabling more granular progress tracking.

5. Choice Set Expansion: Implementation and Impact

Increasing the answer set from four to ten has both theoretical and practical consequences:

• It formally lowers random guessing probability, sharpening the signal provided by true model understanding.
• Distractor generation is performed using controlled approaches (GPT-4-Turbo plus human review) to ensure that all options require genuine discrimination rather than superficial keyword matching.
• Plausible distractors, when implemented carefully, force models to engage with the full question context and to weigh confounding alternatives, accentuating reasoning depth.

6. Chain-of-Thought Reasoning: Methodology and Outcomes

Chain-of-thought (CoT) methodology is embedded in both the construction and analysis protocols:

• Stepwise reasoning prompts (“think step by step”) are explicitly incorporated for both training/finetuning and evaluation.
• On MMLU-Pro, CoT is no longer a marginal improvement but a necessity. Models using CoT demonstrate up to 19% higher accuracy than direct answer approaches.
• The benchmark’s question design, including formulas and domain-specific breakdowns, is structured to leverage CoT advantage, distinguishing models on their substantive reasoning ability.

7. Robustness to Evaluation Artefacts and Future Directions

The benchmark’s design actively mitigates issues arising from test artefacts (such as answer order sensitivity):

• Insights from related work (“Changing Answer Order Can Decrease MMLU Accuracy” (Gupta et al., 27 Jun 2024)) emphasize the necessity for robustness—e.g., systematic shuffling of answer labels and averaging accuracy over shuffles—so as to minimize artefactual score inflation.
• Robust measurement protocols (e.g., test–retest reliability) are suggested as future avenues for optimizing MMLU-Pro leaderboard reliability.
• The methodology is extensible to diagnostics for anchoring effects, shortcut learning, and category-specific vulnerabilities.

Summary Table: Key Enhancements in MMLU-Pro

Dimension	MMLU (Original)	MMLU-Pro (Enhanced)
# Answer Choices	4	10
% Reasoning Qs	Majority recall	Majority reasoning
Avg Accuracy (Top)	86–87%	72.6% (drop by 16–33%)
Prompt Variability	4–5% (peak 11%)	~2%
CoT Benefit	+1.5% (GPT-4o)	+19.1% (GPT-4o)
Discriminative Gap	~2% (top models)	up to ~9%

Context and Significance

MMLU-Pro represents a substantive shift in benchmark development for LLM evaluation, transforming standard accuracy metrics into multidimensional, reasoning-centered assessments. By systematically reducing knowledge-recall artefacts, expanding choice sets, and robustly integrating CoT methodology, MMLU-Pro provides a powerful tool for both diagnostic and comparative evaluation of model reasoning capabilities. Its protocol innovations facilitate more accurate progress tracking for the field, encourage model improvements centered on analytical reasoning, and address pressing concerns on benchmarking artefacts. The benchmark is positioned as a reference standard for future multi-task, multi-domain language understanding efforts.