RoboDojo: When Generalist Robot Policies Meet Reality
This presentation examines RoboDojo, a unified benchmark that exposes the harsh reality of current generalist robot manipulation policies through integrated simulation and real-world testing. We explore how 30 state-of-the-art policies achieve barely 9% success in simulation and 13% in the real world, revealing catastrophic gaps in generalization, precision, and skill composition that human operators solve effortlessly at 76% success rates.Script
The best robot manipulation policies today achieve 9% success in controlled simulation and 13% in the real world, while human experts succeed 76% of the time. RoboDojo reveals why this chasm exists and what it tells us about the limits of current embodied intelligence.
The authors designed 42 simulation tasks and 18 real-world tasks organized across five capability dimensions. Each dimension probes a distinct failure mode: can policies handle visual variation, remember context across long sequences, chain multi-step workflows, execute contact-rich precision maneuvers, or interpret open language instructions they have never seen before?
Domain randomization systematically stresses policies by varying backgrounds, lighting, object appearance, and clutter at evaluation time. Top models suffer catastrophic collapse, with performance dropping over 90% when exposed to these variations. Even minor distributional shifts break what appears robust in carefully controlled settings.
No policy exceeded 9% average success across all simulation tasks, and the best real-world performer managed just 12.8%. Precision tasks remain a critical bottleneck: policies fail at state-conditioned correction, contact awareness, and action smoothness. Semantically open tasks, where language alone specifies unseen task compositions, are essentially unsolved.
RoboDojo provides integrated infrastructure for seamless policy deployment across simulation and reality. Policies integrate once, then evaluate across all tasks via a unified interface. Remote access to real hardware and heterogeneous parallel simulation enable rapid iteration, while strict reproducibility protocols and hidden verification layouts prevent task-specific overfitting and metric gaming.
Current policies cannot yet generalize, remember, compose skills, or safely recover from errors. The benchmark will expand to dexterous hands, humanoid embodiments, tactile sensing, and mobile manipulation. Discover more research like this and generate your own video explainers at EmergentMind.com.
