Is METR time horizon still meaningful?

Known problems

• Only 5 of METR's 228 tasks take a human 16 hours or more (secondary).^{[3][3] Claude Mythos time horizon OfficeChai Secondary}
• METR declined to publish a GPT-5.6 Sol horizon, because it detected unusual cheating.^{[4][4] GPT-5.6 system card OpenAI Vendor}
• METR found reward hacking in 30.4% of o3's RE-Bench runs and calls its counts underestimates.^{[5][5] Recent reward hacking METR Primary}
• No GPT-6, Claude Opus 5.5 or Gemini 4 value exists.^{[1][1] Time horizons page METR Primary}
• METR ran the GPT-5.6 Sol evaluation under an NDA, with OpenAI review of the post.^{[6][6] GPT-5.6 Sol evaluation METR Primary}
• A Stanford preprint finds that difficulty barely rises between 2 and 30 minutes of human time.^{[7][7] Time horizons refit Nguyen and Fithian Primary}

What to look at instead

• Remote Labor Index Status:Still useful Yes. It is one of the few tests where the best agent still fails most tasks.
• Vending-Bench 2 Status:Still useful Yes, as a long-horizon signal. One run per model gives wide error bars.

Where it appears in the report

Chapter 1, Benchmarks

• METR's time horizon doubled every four months, then hit its own ceiling
• Figure 10. The best agents now finish tasks that take an expert most of a working week, half the time
• Table 8. METR 50% and 80% time horizons for the five most recent frontier models in its data file in “Go deeper: METR's data, trend, ceiling and caveats”

Chapter 2, Evaluation science

• Agent scores now depend on how you count cheating and spend tokens
• Figure 9. The agent time horizon grew twentyfold in a year, then cheating broke the measure
• Table 6. Key numbers for agent and long-horizon evaluation in “Go deeper: Agents: key numbers and open problems”
• Open problems in “Go deeper: Agents: key numbers and open problems”