Benchmark checker Law, finance, health, science

Is HealthBench still meaningful?

Still useful Checked against the report, 5 November 2026

Yes, as a template. But the dataset has not changed since August 2025.

Why still useful: The chapter lists it as active. Its Hard subset stood at 32% at launch (May 2025).^{[1][1] HealthBench, arXiv 2505.08775 Primary}

What “Still useful” means

Known problems

The dataset has not changed since August 2025.^{[2][2] HealthBench dataset Hugging Face API Primary}
Physicians agreed with each other on only 55–75% of criterion grades, and that sets the ceiling for judges.^{[3][3] HealthBench meta-evaluation OpenAI Primary}
Rubrics as Rewards trains on rubric scores, and it gained up to 31% on HealthBench over a simple judge reward.^{[4][4] Rubrics as Rewards Scale AI Primary}

What to look at instead

The Professional version followed, but Epoch AI rates it Flawed.^{[5][5] HealthBench Professional review Epoch AI Primary}

HealthBench Professional Status:Flawed No. Half of the sampled tasks had clear errors, and boards disagree on scoring.

Where it appears in the report

Chapter 1, Benchmarks

Domain benchmarks test expert skill, not lawful conduct
Table 13. Health benchmarks, November 2026 in “Go deeper: Health benchmarks (6)”
Table 22. Each part of a multi-turn legal-conduct benchmark already exists in a public benchmark or method in “Go deeper: The parts of a better multi-turn legal-conduct benchmark”

Chapter 2, Evaluation science

AI judges match humans on easy tasks and fall near chance on hard ones
Table 2. Judge agreement only means something next to the human baseline
Figure 4. A validated grading pipeline has five parts
Table 3. Key grading numbers in “Go deeper: Grading: key numbers and open problems”

Chapter 3, Tools and technology

Figure 3. Well-Known Eval Projects Freeze Every Few Months

Sources

Copied from the chapter source lists. Each number links back to the source in its chapter.

[1]
HealthBench, arXiv 2505.08775. It describes physician rubrics on health conversations. Primary arxiv.org Chapter 1, source 108
[2]
HealthBench dataset, Hugging Face API. It shows the last change to the dataset. Primary huggingface.co Chapter 1, source 109
[3]
HealthBench meta-evaluation, OpenAI. It compares model-physician and physician-physician agreement. Primary arxiv.org Chapter 2, source 33
[4]
Rubrics as Rewards, Scale AI. It uses rubric scores as a reinforcement-learning reward. Primary arxiv.org Chapter 2, source 29
[5]
HealthBench Professional review, Epoch AI. It finds clear errors in 25 of 50 sampled tasks. Primary epoch.ai Chapter 1, source 111