Is CMT-Benchmark still meaningful?
Status:Flawed Checked against the report, 5 November 2026
Verdict
No. Epoch AI found flaws in 30 of 50 sampled items.
Why flawed: Epoch AI rated CMT-Benchmark Flawed on 14 October 2026: 30 of 50 sampled items are flawed.[1][1] Benchmark Review registry Epoch AI Primary
What “Flawed” meansKnown problems
- 30 of 50 sampled items are flawed, the highest share in Epoch AI's published samples.[1][1] Benchmark Review registry Epoch AI Primary
What to look at instead
The report names no direct successor.
Where it appears in the report
Chapter 1, Benchmarks
- Figure 5. Thirteen Flawed verdicts in ten weeks
- Figure 6. In several famous benchmarks, a third to a half of audited tasks are broken
- Table 4. Epoch AI Benchmark Review verdicts and the main defects found, August to October 2026 in “Go deeper: Epoch's verdicts, benchmark by benchmark”
Sources
Copied from the chapter source lists. Each number links back to the source in its chapter.
- [1] Benchmark Review registry, Epoch AI. Lists 13 Flawed, 5 Verified and 8 unrated benchmarks in November 2026. Primary epoch.ai Chapter 1, source 3