Benchmark checker Safety and security

Is XSTest and OR-Bench still meaningful?

Near ceiling Checked against the report, 5 November 2026

Only as a hygiene check. Static prompts are easy to tune for.

Why near ceiling: The chapter rates them Near ceiling; labs use them as hygiene checks.^{[1][1] XSTest, arXiv 2308.01263 Primary}^{[2][2] OR-Bench, arXiv 2405.20947 Primary}

What “Near ceiling” means

Known problems

Static prompts are easy to tune for.^{[1][1] XSTest, arXiv 2308.01263 Primary}^{[2][2] OR-Bench, arXiv 2405.20947 Primary}

What to look at instead

The report names no direct successor.

Where it appears in the report

Chapter 1, Benchmarks

Table 16. Harmful-content, jailbreak and over-refusal benchmarks, November 2026 in “Go deeper: Harmful content, jailbreaks and over-refusal (8)”

Sources

Copied from the chapter source lists. Each number links back to the source in its chapter.

[1]
XSTest, arXiv 2308.01263. It tests over-refusal on safe prompts. Primary arxiv.org Chapter 1, source 122
[2]
OR-Bench, arXiv 2405.20947. It tests over-refusal at scale. Primary arxiv.org Chapter 1, source 123