Is Agents' Last Exam still meaningful?
Status:Flawed Checked against the report, 5 November 2026
Verdict
Use with care. Shortcuts exist in 22 of 50 audited tasks, and lab scores disagree.
Why flawed: Epoch AI rated it Flawed on 18 October 2026: 22 of 50 tasks allow a shortcut to a full score.[1][1] Agents' Last Exam review Epoch AI Primary
What “Flawed” meansKnown problems
- 22 of 50 audited tasks allow a shortcut to a full score.[1][1] Agents' Last Exam review Epoch AI Primary
- OpenAI reports 59.3% for GPT-6 Astra, while Google reports a 34.2% binary pass rate for Gemini 4 Argon.[4][4] GPT-6 Astra launch OpenAI Vendor[5][5] Gemini 4 Argon model card Google DeepMind Vendor
What to look at instead
The report names no direct successor.
Where it appears in the report
Chapter 1, Benchmarks
- Figure 4. Best score on benchmarks that still resist, November 2026
- Figure 5. Thirteen Flawed verdicts in ten weeks
- Figure 6. In several famous benchmarks, a third to a half of audited tasks are broken
- Table 4. Epoch AI Benchmark Review verdicts and the main defects found, August to October 2026 in “Go deeper: Epoch's verdicts, benchmark by benchmark”
- Table 7. General and computer-use agent benchmarks, November 2026 in “Go deeper: General and computer-use agent benchmarks (18)”
- Figure 14. OpenAI and Anthropic share only five headline benchmarks
Sources
Copied from the chapter source lists. Each number links back to the source in its chapter.
- [1] Agents' Last Exam review, Epoch AI. It finds shortcuts to a full score in 22 of 50 tasks. Primary epoch.ai Chapter 1, source 21
- [2] Agents' Last Exam results, KuCoin news. It reports overall and hardest-tier pass rates. Secondary kucoin.com Chapter 1, source 20
- [3] Agents' Last Exam, arXiv 2606.05405. It describes the tasks, software and code graders. Primary arxiv.org Chapter 1, source 28
- [4] GPT-6 Astra launch, OpenAI. It gives OpenAI's capability, cyber and alignment tables. Vendor openai.com Chapter 1, source 55
- [5] Gemini 4 Argon model card, Google DeepMind. It gives benchmark rows and Frontier Safety Framework results. Vendor deepmind.google Chapter 1, source 56