Is HealthBench Professional still meaningful?
Status:Flawed Checked against the report, 5 November 2026
Verdict
No. Half of the sampled tasks had clear errors, and boards disagree on scoring.
Why flawed: Epoch AI rated it Flawed on 10 September 2026: it found clear errors in 25 of 50 sampled tasks.[1][1] HealthBench Professional review Epoch AI Primary
What “Flawed” meansKnown problems
- Some tasks score 100% for declining to answer.[1][1] HealthBench Professional review Epoch AI Primary
- OpenAI picked items that were hard for its own models.[3][3] HealthBench Professional, arXiv 2604.27470 Primary
- Third-party boards conflict: one ranks Claude Fable 5 first on a length-adjusted score, another gives Claude Opus 5 73.4% raw.[4][4] HealthBench Professional LLM Stats Secondary[2][2] HealthBench Professional BenchLM Secondary
What to look at instead
The report names no direct successor.
Where it appears in the report
Chapter 1, Benchmarks
- Figure 5. Thirteen Flawed verdicts in ten weeks
- Figure 6. In several famous benchmarks, a third to a half of audited tasks are broken
- Table 4. Epoch AI Benchmark Review verdicts and the main defects found, August to October 2026 in “Go deeper: Epoch's verdicts, benchmark by benchmark”
- Table 13. Health benchmarks, November 2026 in “Go deeper: Health benchmarks (6)”
- Figure 14. OpenAI and Anthropic share only five headline benchmarks
Sources
Copied from the chapter source lists. Each number links back to the source in its chapter.
- [1] HealthBench Professional review, Epoch AI. It finds clear errors in 25 of 50 sampled tasks. Primary epoch.ai Chapter 1, source 111
- [2] HealthBench Professional, BenchLM. It lists raw scores from one aggregator. Secondary benchlm.ai Chapter 1, source 114
- [3] HealthBench Professional, arXiv 2604.27470. It describes clinician tasks and item selection. Primary arxiv.org Chapter 1, source 110
- [4] HealthBench Professional, LLM Stats. It lists length-adjusted scores. Secondary llm-stats.com Chapter 1, source 113