Is SWE-bench Verified still meaningful?

Known problems

• OpenAI stopped reporting it on 23 February 2026, citing contamination and flawed tests.^{[3][3] Why we no longer evaluate SWE-bench Verified OpenAI Vendor}
• OpenAI's audit covered 27.6% of tasks. In those, 59.4% had tests that reject correct fixes.^{[1][1] SWE-bench Verified review Epoch AI Primary}
• Every frontier model OpenAI tested could reproduce the original fixes.^{[1][1] SWE-bench Verified review Epoch AI Primary}
• No lab headlines it in its autumn 2026 launch table.^{[4][4] Introducing Claude Opus 5.5 Anthropic Vendor[5][5] GPT-6 Astra benchmarks explained Vellum Secondary}
• In March 2026, maintainers would merge agent pull requests at rates about 24 points below the automated scores.^{[6][6] Many SWE-bench-passing PRs would not merge METR Primary}
• In a single-source preprint, 28.5% of 49 tasks accepted an incorrect patch. Scores ran 14.14 points higher on the hackable tasks.^{[7][7] Weak code verifiers on SWE-bench Verified Single source}

What to look at instead

Coding moved to freshly written, partly private task sets. DeepSWE and Terminal-Bench 4.0 also carry Flawed ratings.^{[8][8] DeepSWE Datacurve Vendor[4][4] Introducing Claude Opus 5.5 Anthropic Vendor[9][9] Benchmark Review registry Epoch AI Primary}

• DeepSWE Status:Flawed Partly. Its fresh tasks resist contamination, but Epoch AI found issues in at least 23 of 113.
• Terminal-Bench 4.0 Status:Flawed Use with care. It is a launch-table favourite, but it is rated Flawed and lab numbers disagree.
• FrontierCode Status:New / unproven Unproven. Labs headline it, but nobody has confirmed who runs it or how.
• CursorBench Status:Still useful Yes, as one signal. Remember that a product company owns it.

Where it appears in the report

Chapter 1, Benchmarks

• Figure 1. Each New Benchmark Fills Up Faster Than the Last
• Figure 5. Thirteen Flawed verdicts in ten weeks
• Table 2. First model to cross each threshold in “Go deeper: The full saturation timeline (14 benchmarks)”
• Table 4. Epoch AI Benchmark Review verdicts and the main defects found, August to October 2026 in “Go deeper: Epoch's verdicts, benchmark by benchmark”
• Table 6. Coding and software-agent benchmarks, November 2026 in “Go deeper: Coding and software-agent benchmarks (14)”
• Figure 14. OpenAI and Anthropic share only five headline benchmarks

Chapter 2, Evaluation science

• Open problems in “Go deeper: Grading: key numbers and open problems”
• Open problems in “Go deeper: Statistics: key numbers and open problems”
• Table 4. Key statistics and reliability numbers in “Go deeper: Statistics: key numbers and open problems”
• Passing a benchmark overstates real work by about two times
• Table 5. Key validity and contamination numbers in “Go deeper: Validity: key numbers and open problems”