Skip to content

Benchmark checker Law, finance, health, science

Is MedQA still meaningful?

Saturated Checked against the report, 5 November 2026

No. It is saturated, and exam questions match few real clinical queries.

Why saturated: The chapter rates MedQA Saturated, and Vals AI has archived it.^{[1][1] Benchmark list Vals AI Primary}

What “Saturated” means

Known problems

Only a third of clinician queries resemble exam questions.^{[2][2] MedArena, arXiv 2603.15677 Primary}

What to look at instead

HealthBench Status:Still useful Yes, as a template. But the dataset has not changed since August 2025.

Where it appears in the report

Chapter 1, Benchmarks

Table 13. Health benchmarks, November 2026 in “Go deeper: Health benchmarks (6)”

Sources

Copied from the chapter source lists. Each number links back to the source in its chapter.

[1]
Benchmark list, Vals AI. It lists Vals's legal, finance, health, science and cyber boards. Primary vals.ai Chapter 1, source 15
[2]
MedArena, arXiv 2603.15677. It collects clinician preferences between models. Primary arxiv.org Chapter 1, source 107