Benchmark checker Safety and security

Is HarmBench still meaningful?

Saturated Checked against the report, 5 November 2026

No longer for the frontier. Refusal violations sit near zero, and nobody maintains it.

Why saturated: Chat refusal violations sit near 0–2% in 2026 model cards, and the repository has seen no push since August 2024.^{[1][1] Grok 4.20 model card xAI Vendor}^{[2][2] HarmBench repository GitHub Primary}

What “Saturated” means

Known problems

The repository has seen no push since August 2024.^{[2][2] HarmBench repository GitHub Primary}
UK AISI still found universal jailbreaks "in every system tested" in December 2025.^{[4][4] Frontier AI Trends Report findings UK AISI Primary}

What to look at instead

AgentHarm Status:Still useful Yes, as part of the standard stack. It uses simulated tools only.
AgentDojo Status:Still useful Yes, run it yourself. The public results page stopped in February 2025.

Where it appears in the report

Chapter 1, Benchmarks

The safety signal moved from static prompts to agent behaviour and real uplift
Table 16. Harmful-content, jailbreak and over-refusal benchmarks, November 2026 in “Go deeper: Harmful content, jailbreaks and over-refusal (8)”
Maintenance in “Go deeper: Judges, normalisation, validity and maintenance”

Sources

Copied from the chapter source lists. Each number links back to the source in its chapter.

[1]
Grok 4.20 model card, xAI. It reports safety, honesty and dangerous-capability results. Vendor data.x.ai Chapter 1, source 117
[2]
HarmBench repository, GitHub. It shows the date of the last push. Primary github.com Chapter 1, source 119
[3]
HarmBench, arXiv 2402.04249. It standardises automated red-teaming. Primary arxiv.org Chapter 1, source 118
[4]
Frontier AI Trends Report findings, UK AISI. It summarises two years of frontier testing. Primary aisi.gov.uk Chapter 1, source 129