Is MMMU-Pro still meaningful?

Known problems

• Sources disagree on the same model. Google lists Gemini 3.1 Pro at 80.5%, and BenchLM lists it at 83.9%.^{[3][3] Google DeepMind, Gemini Pro page Vendor[1][1] BenchLM, MMMU-Pro Secondary}
• Tool and Python variants mix into rankings and inflate scores, for example GPT-5.4 at 89.3% with tools (aggregator).^{[4][4] Another Wrapper, MMMU-Pro with tools Secondary}
• A July 2026 news claim of 94% for GPT-5.4 Pro stays unverified.^{[5][5] af.net on GPT-5.4 Pro Single source}
• Models still stay "close to random guesses" on rare image types, such as music sheets and chemical structures.^{[6][6] MMMU benchmark site Primary}

What to look at instead

Only a re-run in one fixed harness would settle the gap between lab and aggregator numbers.^{[3][3] Google DeepMind, Gemini Pro page Vendor[1][1] BenchLM, MMMU-Pro Secondary}

• HallusionBench Status:Still useful Yes. Frontier models still score only 60–67, so it separates them on hallucination.

Where it appears in the report

Chapter 7, Multimodal, voice and embodied

• Figure 1. Tests Beyond Text Fill Up in Months, Too
• Figure 2. MMMU-Pro scores move with the source that reports them
• Table 1. Each modality has one number that tells its story
• Table 2. Image and document benchmarks with their best scores in “Go deeper: Image and document benchmarks, full table”
• Vision tests filled up, and the vision arena can no longer rank the frontier