Is GenEval still meaningful?

Known problems

• Its automatic judge errs by up to 17.7% against human judgment (Dec 2025).^{[1][1] GenEval 2, arXiv 2512.16853 Primary}
• A frozen judge falls behind the models it grades, so it needs regular re-calibration against fresh human labels.^{[1][1] GenEval 2, arXiv 2512.16853 Primary}

What to look at instead

Blind human votes on Arena and Artificial Analysis now rank image generation.^{[3][3] Arena, text-to-image leaderboard Primary[4][4] Artificial Analysis, text-to-image leaderboard Primary}

• GenEval 2 Status:New / unproven Yes, tentatively. The best model reaches only 35.8%, but outside use is still thin.
• Chatbot Arena (LMArena) Status:Contested Partly. It still shows crowd preference, but private testing and unequal data access distort the ranking.

Where it appears in the report

Chapter 7, Multimodal, voice and embodied

• Table 1. Each modality has one number that tells its story
• Blind human votes now rank generation, because automatic judges drift
• Figure 4. A harder test drops the best image score from 97% to 36%
• Table 6. Automatic metrics and learned scorers for image, video and music generation, read November 2026 in “Go deeper: Automated generation metrics”