Is CMMLU still meaningful?

Known problems

• Most of 18 models scored below 50% at launch.^{[2][2] CMMLU, arXiv 2306.09212 Primary} A 2026 base model now scores 90.0.^{[1][1] ERNIE-5.5-Flash-Base model card Primary}
• Chinese-language benchmarks appear only in base-model tables of the 2026 cards the report checked.^{[3][3] DeepSeek-V4.1-Flash model card Primary[1][1] ERNIE-5.5-Flash-Base model card Primary}

What to look at instead

SuperCLUE uses private questions refreshed each round.^{[4][4] DataLearner, SuperCLUE page Secondary}

• SuperCLUE Status:Still useful Yes for Chinese-market ranking, since private questions resist contamination, but scores from different rounds do not compare.
• C-Eval Status:Saturated No. A 2026 base model scores above 92, and labs now cite it only in base-model tables.

Where it appears in the report

Chapter 6, China and Asia

• China has its own leaderboards, but frontier claims run on Western tests
• Table 2. First-generation Chinese benchmarks in “Go deeper: First-generation Chinese benchmarks (2023–2026)”
• Figure 3. Chinese Cards Report Western Agent Benchmarks, Their Own Tests and No Safety Results