Is MMLU still meaningful?

Known problems

• Labs stopped reporting it before any model crossed 90% in Epoch AI's data.^{[2][2] Benchmark data bundle Epoch AI Primary}
• The first model passed 50% in December 2021, and the climb to 75% took another 15 months.^{[2][2] Benchmark data bundle Epoch AI Primary}
• Anthropic's Claude Opus 5.5 launch table (22 Sep 2026) has no MMLU row.^{[3][3] Introducing Claude Opus 5.5 Anthropic Vendor}
• About 6.49% of MMLU questions contain errors, and 57% of the audited Virology questions do.^{[4][4] Are We Done with MMLU? Edinburgh Primary}

What to look at instead

The knowledge line runs MMLU → MMLU-Pro → GPQA Diamond → HLE → HLE-Diamond (Sep 2026).^{[1][1] Humanity's Last Exam site Primary[5][5] HLE-Verified, arXiv 2602.13964 Primary}

• MMLU-Pro Status:Saturated No. Top scores sit near 90%, and no 2026 frontier results come from a primary source.
• GPQA Diamond Status:Near ceiling Barely. Top models sit within one point of the 97% ceiling, inside the noise.
• Humanity's Last Exam (HLE) Status:Flawed Use with care. Nearly half of a sampled set was defective, so top scores partly reflect broken items.
• HLE-Diamond Status:New / unproven Promising but unproven. It launched in September 2026 and has no independent audit yet.

Where it appears in the report

Chapter 1, Benchmarks

• Figure 1. Each New Benchmark Fills Up Faster Than the Last
• Figure 2. Recent benchmarks went from 50% to 90% in under a year
• Table 2. First model to cross each threshold in “Go deeper: The full saturation timeline (14 benchmarks)”
• Table 3. How each area replaced its saturated benchmarks, oldest first in “Go deeper: Lineage of replacements, by area”
• Table 5. Reasoning and knowledge benchmarks: what they measure, who maintains them, the top score on 5 November 2026 and the known flaw in “Go deeper: Reasoning and knowledge benchmarks (17)”
• Figure 14. OpenAI and Anthropic share only five headline benchmarks

Chapter 2, Evaluation science

• Error bars and adaptive tests make a score trustworthy at lower cost
• Figure 6. Item response theory cuts the number of test items by 30 to 140 times
• Table 4. Key statistics and reliability numbers in “Go deeper: Statistics: key numbers and open problems”