Is Chatbot Arena (LMArena) still meaningful?

Known problems

• A chat-tuned experimental Llama 4 model ranked second in April 2025.^{[3][3] Llama 4 on LMArena Simon Willison Secondary} The public release model ranked 32nd.^{[4][4] Unmodified Llama 4 Maverick ranking Neowin Secondary}
• Extra Arena data gave relative gains of up to 112% on the Arena distribution.^{[1][1] The Leaderboard Illusion (Singh et al.) Primary}
• LMArena replied that any provider may test variants when it has capacity.^{[5][5] Our response LMArena Primary} Arvind Narayanan found that reply unconvincing.^{[6][6] Criticism of Chatbot Arena Simon Willison Secondary} The report could not verify later policy changes.
• Arena passed $100M in annualised revenue in October 2026, and most of it likely comes from the labs it ranks.^{[2][2] Series B announcement Arena Vendor}
• Labs pre-test unreleased models on it under codenames. OpenAI tested GPT-5 as "summit".^{[7][7] LMArena Series A SiliconANGLE Secondary}
• On 19 October 2026, the top ten on Arena's vision board sat within 16 Elo points, with overlapping intervals.^{[8][8] Arena, vision leaderboard Primary}
• The two video arenas disagree. Gemini Omni 1.1 Flash ranks first on Arena but tenth on Artificial Analysis (Sep–Nov 2026).^{[9][9] Arena, text-to-video leaderboard Primary[10][10] Artificial Analysis, text-to-video leaderboard Primary}

What to look at instead

The report names no direct successor.

Where it appears in the report

Chapter 2, Evaluation science

• AI judges match humans on easy tasks and fall near chance on hard ones
• Figure 2. Five AI Judges, and None Agrees Well With Human Voters
• Table 2. Judge agreement only means something next to the human baseline
• Passing a benchmark overstates real work by about two times
• Figure 7. Two labs received two-fifths of all Chatbot Arena data
• Table 5. Key validity and contamination numbers in “Go deeper: Validity: key numbers and open problems”

Chapter 4, Market and companies

• Leaderboards became businesses, and every business model carries a conflict of interest
• Figure 12. Only the Buyer-Paid Benchmark Avoids Both Conflicts
• Table 15. Leaderboards and independent evaluators in “Go deeper: Leaderboard companies, conflict table and more detail”

Chapter 7, Multimodal, voice and embodied

• Table 1. Each modality has one number that tells its story
• Figure 3. The top ten vision models are statistically tied
• Arena details in “Go deeper: Arena and GUI grounding detail”
• Table 5. The two arenas agree on images and split on video
• Table 7. Board sizes and leaders as read in September to November 2026 in “Go deeper: Generation arenas, and how Artificial Analysis runs its board”