Is DeepSWE still meaningful?

Known problems

• Epoch AI found issues in at least 23 of 113 tasks.^{[1][1] Benchmark Review registry Epoch AI Primary}
• 78% of the false negatives came from models editing tests.^{[1][1] Benchmark Review registry Epoch AI Primary}
• Its verifiers disagree on 1.4% of tasks, against 32% on SWE-bench Pro (vendor-run analysis).^{[3][3] DeepSWE Datacurve Vendor}
• DeepSeek ran one model on DeepSWE v1.1 with eight harness setups. Scores ranged from 65.5 to 74.2 (Oct 2026).^{[4][4] DeepSeek-V4.1-Flash model card Primary}

What to look at instead

The report names no direct successor.

Where it appears in the report

Chapter 1, Benchmarks

• The new frontier prices tasks in money and hours
• Figure 5. Thirteen Flawed verdicts in ten weeks
• Figure 6. In several famous benchmarks, a third to a half of audited tasks are broken
• Table 4. Epoch AI Benchmark Review verdicts and the main defects found, August to October 2026 in “Go deeper: Epoch's verdicts, benchmark by benchmark”
• Table 6. Coding and software-agent benchmarks, November 2026 in “Go deeper: Coding and software-agent benchmarks (14)”
• Figure 14. OpenAI and Anthropic share only five headline benchmarks

Chapter 6, China and Asia

• Chinese model cards report agentic scores and no safety evaluations
• Figure 3. Chinese Cards Report Western Agent Benchmarks, Their Own Tests and No Safety Results
• Figure 4. Changing the Harness Alone Moves DeepSeek's Score by Almost 9 Points