Benchmark checker Safety and security

Is AgentHarm still meaningful?

Still useful Checked against the report, 5 November 2026

Yes, as part of the standard stack. It uses simulated tools only.

Why still useful: The chapter lists it as active, and labs still report it: Grok 4.20's violation rate is 0.30 (Apr 2026).^{[1][1] Grok 4.20 model card xAI Vendor}

What “Still useful” means

Known problems

It uses simulated tools only.^{[2][2] AgentHarm, arXiv 2410.09024 Primary}

What to look at instead

The report names no direct successor.

Where it appears in the report

Chapter 1, Benchmarks

The safety signal moved from static prompts to agent behaviour and real uplift
Table 16. Harmful-content, jailbreak and over-refusal benchmarks, November 2026 in “Go deeper: Harmful content, jailbreaks and over-refusal (8)”

Sources

Copied from the chapter source lists. Each number links back to the source in its chapter.

[1]
Grok 4.20 model card, xAI. It reports safety, honesty and dangerous-capability results. Vendor data.x.ai Chapter 1, source 117
[2]
AgentHarm, arXiv 2410.09024. It tests compliance with malicious agent tasks. Primary arxiv.org Chapter 1, source 124