Is Vending-Bench 2 still meaningful?
Status:Still useful Checked against the report, 5 November 2026
Verdict
Yes, as a long-horizon signal. One run per model gives wide error bars.
Why still useful: The chapter lists it as active. Andon Labs fits a frontier trend of about $854 more per month.[1][1] Vending-Bench 2 Andon Labs Primary
What “Still useful” meansKnown problems
- One long run per model gives wide error bars.[1][1] Vending-Bench 2 Andon Labs Primary
What to look at instead
The report names no direct successor.
Where it appears in the report
Chapter 1, Benchmarks
- The new frontier prices tasks in money and hours
- Table 3. How each area replaced its saturated benchmarks, oldest first in “Go deeper: Lineage of replacements, by area”
- Table 7. General and computer-use agent benchmarks, November 2026 in “Go deeper: General and computer-use agent benchmarks (18)”
Sources
Copied from the chapter source lists. Each number links back to the source in its chapter.
- [1] Vending-Bench 2, Andon Labs. It gives final balances and the frontier trend. Primary andonlabs.com Chapter 1, source 29