Is Deepfake-Eval-2024 still meaningful?
Status:Still useful Checked against the report, 5 November 2026
Verdict
Yes, as a reality check. Open-source detectors lose about half their skill on its real fakes.
Why still useful: Open-source state-of-the-art detectors lost 45–50% of their AUC on its real fakes, so it still exposes large gaps.[1][1] Deepfake-Eval-2024, arXiv 2503.02857 Primary It is an academic snapshot, not a maintained leaderboard.
What “Still useful” meansKnown problems
- It is an academic snapshot from early 2025, not a maintained leaderboard.[1][1] Deepfake-Eval-2024, arXiv 2503.02857 Primary
- Commercial and fine-tuned detectors did better, but they "do not yet reach the accuracy of deepfake forensic analysts".[1][1] Deepfake-Eval-2024, arXiv 2503.02857 Primary
- The chapter found no standing, independent leaderboard for deepfake detection.[1][1] Deepfake-Eval-2024, arXiv 2503.02857 Primary
What to look at instead
The report names no direct successor.
Where it appears in the report
Chapter 7, Multimodal, voice and embodied
- Table 1. Each modality has one number that tells its story
- Deepfake detectors lose about half their skill on real fakes
- Figure 5. Open-source deepfake detectors lose about half their AUC in the wild
- Table 8. Share of AUC lost on in-the-wild 2024 deepfakes compared with academic benchmarks in “Go deeper: Deepfake detection: detail and gaps”
- Figure 11. Robotics, GUI agents and deepfake detection have the weakest independent checks
Sources
Copied from the chapter source lists. Each number links back to the source in its chapter.
- [1]