Is MMMU still meaningful?
Status:Flawed Checked against the report, 5 November 2026
Verdict
No. Its test answers went public in February 2026, so new scores may reflect contamination.
Why flawed: Its test answers became public in February 2026, which ends contamination control.[1][1] MMMU benchmark site Primary The chapter lists MMMU among retired or discredited tests.
What “Flawed” meansKnown problems
- Future MMMU scores deserve less weight than scores on private sets.[1][1] MMMU benchmark site Primary
- Text-only models could answer some of its questions, so MMMU-Pro removed them.[3][3] MMMU-Pro, arXiv 2409.02813 Primary
- Four model families already scored 80.6–84.2 by November 2025.[2][2] Qwen3-VL technical report, arXiv 2511.21631 Primary
What to look at instead
At launch, models scored 16.8–26.9 points lower on MMMU-Pro than on MMMU.[3][3] MMMU-Pro, arXiv 2409.02813 Primary
- MMMU-Pro Status:Near ceiling Partly. Scores sit near 85% and differ by source, so compare models within one source only.
- HallusionBench Status:Still useful Yes. Frontier models still score only 60–67, so it separates them on hallucination.
Where it appears in the report
Chapter 7, Multimodal, voice and embodied
- Figure 1. Tests Beyond Text Fill Up in Months, Too
- Table 2. Image and document benchmarks with their best scores in “Go deeper: Image and document benchmarks, full table”
- Figure 11. Robotics, GUI agents and deepfake detection have the weakest independent checks
Sources
Copied from the chapter source lists. Each number links back to the source in its chapter.
- [1]
- [2] Qwen3-VL technical report, arXiv 2511.21631. Lab report; Table 2 compares four model families. Primary arxiv.org Chapter 7, source 1
- [3]