Is OSWorld 2.1 still meaningful?
Status:Flawed Checked against the report, 5 November 2026
Verdict
No. Grading errors could affect half of the score.
Why flawed: Epoch AI rated it Flawed on 28 September 2026: errors could affect up to 50.4% of the score.[1][1] OSWorld 2.1 review Epoch AI Primary
What “Flawed” meansKnown problems
- Epoch AI's audit of 50 tasks found 4 impossible tasks, 24 false negatives and 12 false positives.[1][1] OSWorld 2.1 review Epoch AI Primary
What to look at instead
- OSWorld 2.0 Status:Still useful Yes. Full-credit scores stay low, though partial credit tells a kinder story.
Where it appears in the report
Chapter 1, Benchmarks
- Figure 5. Thirteen Flawed verdicts in ten weeks
- Table 4. Epoch AI Benchmark Review verdicts and the main defects found, August to October 2026 in “Go deeper: Epoch's verdicts, benchmark by benchmark”
- Table 7. General and computer-use agent benchmarks, November 2026 in “Go deeper: General and computer-use agent benchmarks (18)”
- Figure 14. OpenAI and Anthropic share only five headline benchmarks
Sources
Copied from the chapter source lists. Each number links back to the source in its chapter.
- [1] OSWorld 2.1 review, Epoch AI. Impossible tasks, false negatives and false positives. Primary epoch.ai Chapter 1, source 8
- [2] Introducing Claude Opus 5.5, Anthropic. It gives Anthropic's launch table and caveats. Vendor anthropic.com Chapter 1, source 12