Is C-Eval still meaningful?

Known problems

• DeepSeek-V4.1-Flash reports Chinese benchmarks only in its base-model table (Oct 2026).^{[1][1] DeepSeek-V4.1-Flash model card Primary}
• Chinese flagship cards make frontier claims on Western tests such as Terminal-Bench, SWE-bench Pro and HLE.^{[1][1] DeepSeek-V4.1-Flash model card Primary[3][3] GLM-5.3 model card Primary}

What to look at instead

SuperCLUE uses private questions refreshed each round.^{[4][4] DataLearner, SuperCLUE page Secondary} Chinese frontier claims now run on Western agentic tests.^{[1][1] DeepSeek-V4.1-Flash model card Primary[3][3] GLM-5.3 model card Primary}

• SuperCLUE Status:Still useful Yes for Chinese-market ranking, since private questions resist contamination, but scores from different rounds do not compare.
• SWE-bench Pro Status:Flawed No. About a third of its tasks are broken, and agents have cheated on it.
• Humanity's Last Exam (HLE) Status:Flawed Use with care. Nearly half of a sampled set was defective, so top scores partly reflect broken items.

Where it appears in the report

Chapter 6, China and Asia

• China has its own leaderboards, but frontier claims run on Western tests
• Table 2. First-generation Chinese benchmarks in “Go deeper: First-generation Chinese benchmarks (2023–2026)”