Is OTIS Mock AIME 2024-25 still meaningful?

Known problems

• It has too few items to separate models.^{[1][1] Benchmark data bundle Epoch AI Primary}

What to look at instead

• MathArena Status:Still useful Yes. Fresh contests limit contamination, but small contests give wide error bars.
• FrontierMath Tiers 1–3 Status:Near ceiling Fading. The top score is 93.7%, and the funder can see part of the set.

Where it appears in the report

Chapter 1, Benchmarks

• Figure 1. Each New Benchmark Fills Up Faster Than the Last
• Figure 2. Recent benchmarks went from 50% to 90% in under a year
• Table 2. First model to cross each threshold in “Go deeper: The full saturation timeline (14 benchmarks)”
• Table 5. Reasoning and knowledge benchmarks: what they measure, who maintains them, the top score on 5 November 2026 and the known flaw in “Go deeper: Reasoning and knowledge benchmarks (17)”