Is LIBERO still meaningful?

Known problems

• Policies that scored above 90% collapsed to 0% under changed objects, start states, instructions or scenes. The authors call this "rote memorization of action sequences".^{[1][1] LIBERO-PRO, arXiv 2510.03827 Primary}
• Modest camera and start-state shifts cut success from 95% to below 30%.^{[2][2] LIBERO-Plus, arXiv 2510.13626 Primary}
• Most robot papers run too few rollouts. Seventy rollouts at 90% success leave a band about 15 points wide.^{[4][4] The Robot Report on NVIDIA robot evaluation Secondary}
• Its code last changed in March 2025.^{[5][5] GitHub, LIBERO Primary}

What to look at instead

LIBERO-PRO and LIBERO-Plus re-test LIBERO policies under perturbations.^{[1][1] LIBERO-PRO, arXiv 2510.03827 Primary[2][2] LIBERO-Plus, arXiv 2510.13626 Primary}

• BEHAVIOR-1K Status:Still useful Yes. The best challenge entry succeeds on only 12.4% of tasks, which leaves much room.

Where it appears in the report

Chapter 7, Multimodal, voice and embodied

• Figure 1. Tests Beyond Text Fill Up in Months, Too
• Table 1. Each modality has one number that tells its story
• Robot scores collapse under small changes, and labs mostly grade themselves
• Figure 8. Robot policies that ace LIBERO fail when the scene shifts
• Table 12. Simulation benchmarks for robot manipulation and household tasks, read November 2026 in “Go deeper: Simulation benchmarks”