Is SimpleQA Verified still meaningful?
Status:Still useful Checked against the report, 5 November 2026
Verdict
Yes. It passed Epoch AI's audit, though it rewards guessing over saying "I don't know".
Why still useful: Epoch AI rates it Verified: only 5 of 50 sampled items had score-affecting defects.[1][1] Epoch's benchmark reviews The Neuron Secondary
What “Still useful” meansKnown problems
- It counts abstentions as wrong, which rewards guessing.[1][1] Epoch's benchmark reviews The Neuron Secondary
What to look at instead
The report names no direct successor.
Where it appears in the report
Chapter 1, Benchmarks
- Figure 6. In several famous benchmarks, a third to a half of audited tasks are broken
- Table 4. Epoch AI Benchmark Review verdicts and the main defects found, August to October 2026 in “Go deeper: Epoch's verdicts, benchmark by benchmark”
- Table 5. Reasoning and knowledge benchmarks: what they measure, who maintains them, the top score on 5 November 2026 and the known flaw in “Go deeper: Reasoning and knowledge benchmarks (17)”
Sources
Copied from the chapter source lists. Each number links back to the source in its chapter.
- [1] Epoch's benchmark reviews, The Neuron. It reports nine Flawed and four Verified ratings on 18 Sep 2026. Secondary theneuron.ai Chapter 1, source 36
- [2] Benchmark data bundle, Epoch AI. Dated scores (5 Nov 2026) and ceilings for most benchmarks here. Primary epoch.ai Chapter 1, source 1