A Balanced Data Diet: Addressing the Exploration Bottleneck in Mega-Scale RL for Robot Control
This presentation examines a critical scaling failure in massively parallel robot reinforcement learning: simply adding more simulated environments doesn't guarantee more useful learning signal. The paper introduces Success Guided Sampling, an adaptive method that reallocates training experience toward configurations at the policy's current capability frontier. Through experiments spanning multi-terrain quadruped locomotion and contact-rich assembly manipulation at scales up to one million parallel environments, the authors demonstrate that intelligent allocation of rollouts is essential for converting simulation throughput into actual learning progress, ultimately enabling zero-shot transfer of complex assembly behaviors to physical hardware.Script
At one million parallel robot simulations, uniform sampling achieved just 6 percent success on nut-and-bolt assembly, while an adaptive sampler reached 70 percent using the exact same computational resources. The difference wasn't how many environments ran in parallel, but which configurations those environments practiced.
When task configurations are sampled uniformly, the policy wastes rollouts on configurations it has already mastered and configurations currently beyond its reach. The paper argues that the learning frontier, where success rates hover between total failure and reliable mastery, is where gradient information is richest.
Success Guided Sampling maintains a rolling window of recent outcomes for each configuration and assigns high probability to those whose empirical success rates fall near a target value. The weighting is derived from a Beta-shaped kernel, ensuring that easy and unsolved configurations retain nonzero probability while the bulk of experience concentrates at the capability frontier.
On the demanding Franka nut-and-bolt assembly task, uniform sampling and prioritized level replay both failed at 32 thousand environments and remained near failure even at one million. Success Guided Sampling learned at the smaller scale and continued improving monotonically, demonstrating that adaptive allocation converts additional parallelism into denser learning signal rather than merely larger, redundant batches.
The authors distilled state-based teacher policies into RGB students using online imitation and transferred them zero-shot to a physical UR5e arm. Gear-mesh insertion achieved 94 percent real-world success, while nut-and-bolt assembly transferred at 37.5 percent, revealing that sim-to-real gaps remain task dependent and that physical realism trade-offs, such as gripper friction tuning, directly shape transfer quality.
This work demonstrates that massively parallel simulation is only as effective as the allocation strategy guiding it: scaling throughput without focusing experience on the learnable frontier leaves the hardest, most valuable behaviors out of reach. Visit EmergentMind.com to explore the full paper and create your own videos on the research shaping the future of robotic learning.
