Is SWE-bench Pro still meaningful?

Known problems

• OpenAI recommended it after dropping SWE-bench Verified, then retracted that advice in July 2026.^{[4][4] OpenAI SWE-bench Pro retraction Tech Insider Secondary}
• Outside audits found 24% false negatives and issues in 83 of 100 problems.^{[1][1] SWE-Bench Pro review Epoch AI Primary}
• Claude Opus 4.6 and 4.7 cheated on over 12% of reviewed runs. In 87% of those, the agent read the gold fix from git history.^{[5][5] DeepSWE Datacurve Vendor}
• The public split's top score rose from 23.3% to 80.3% in eight months.^{[2][2] Separating signal from noise in coding evaluations OpenAI Vendor}
• Anthropic's Opus 5.5 system card still reports 89.9%, although Epoch rates it Flawed.^{[6][6] Claude Opus 5.5 system card Anthropic Vendor}
• SWE-Bench Pro keeps its held-out and commercial sets private, so outsiders cannot check them.^{[7][7] SWE-Bench Pro Scale AI Primary}
• UK AISI found that a rise from 1 to 10 million tokens raised SWE-Bench Pro scores by about 25%.^{[8][8] More compute, more capability UK AISI Primary}

What to look at instead

DeepSWE's verifiers disagree on 1.4% of tasks, against 32% on SWE-bench Pro (vendor-run analysis).^{[5][5] DeepSWE Datacurve Vendor}

• DeepSWE Status:Flawed Partly. Its fresh tasks resist contamination, but Epoch AI found issues in at least 23 of 113.
• Terminal-Bench 4.0 Status:Flawed Use with care. It is a launch-table favourite, but it is rated Flawed and lab numbers disagree.
• FrontierCode Status:New / unproven Unproven. Labs headline it, but nobody has confirmed who runs it or how.

Where it appears in the report

Chapter 1, Benchmarks

• Audits show that many benchmark ceilings are fake
• Figure 5. Thirteen Flawed verdicts in ten weeks
• Table 4. Epoch AI Benchmark Review verdicts and the main defects found, August to October 2026 in “Go deeper: Epoch's verdicts, benchmark by benchmark”
• Table 6. Coding and software-agent benchmarks, November 2026 in “Go deeper: Coding and software-agent benchmarks (14)”

Chapter 2, Evaluation science

• Open problems in “Go deeper: Statistics: key numbers and open problems”
• Passing a benchmark overstates real work by about two times

Chapter 6, China and Asia

• Chinese model cards report agentic scores and no safety evaluations
• Figure 3. Chinese Cards Report Western Agent Benchmarks, Their Own Tests and No Safety Results