A Balanced Data Diet: Addressing the Exploration Bottleneck in Mega-Scale RL for Robot Control

Abstract: General-purpose robots must perform a wide range of tasks from agile locomotion to dexterous manipulation. While sim-to-real reinforcement learning (RL) has proven to be a useful tool for this goal, current RL pipelines depend on engineering-heavy, per-task structural priors such as shaped rewards and demonstrations. Recent work has shown that diverse simulator resets, combined with massively parallel simulation, can alleviate much of this engineering burden on several manipulation problems. However, we find that naively scaling this paradigm to more precise or dynamic problems remains non-trivial. While simulator resets can help with exploration, uniformly sampling over this distribution wastes a growing fraction of learning experience on task configurations the policy has already mastered or cannot yet attempt. This makes it challenging to see the expected benefits of scaling parallel environments for RL, since much of the learning signal in a batch is wasted during learning. To mitigate this, we introduce Success Guided Sampling (SGS), a simple adaptive sampler that concentrates RL training on task configurations around the frontier of the policy's capabilities. Doing so allows large-scale simulated RL to make the most out of the experience in a batch, enabling much more effective scaling to large-scale parallel simulation. Across experiments using up to $2<sup>{20}$ (over one million) parallel environments, SGS enables RL to solve challenging multi-terrain quadruped locomotion and contact-rich assembly tasks that prior methods fail to solve. Finally, we distill the learned manipulation policies into RGB-based policies and demonstrate zero-shot transfer to several challenging assembly tasks on real hardware. Project website: https://sgs-rl.github.io/.

• The paper introduces a Success Guided Sampling (SGS) mechanism to reallocate experience toward task configurations with intermediate success rates in mega-scale reinforcement learning (RL) for robot control configs.
• SGS achieves 73% mean success on multi-terrain locomotion and 70% success on Franka nut-and-bolt assembly at one million environments, significantly outperforming uniform sampling and Prioritized Level Replay (PLR).
• The method demonstrates that adaptive allocation of parallel rollouts is crucial for effective learning, particularly in challenging, long-horizon manipulation tasks, and supports zero-shot RGB sim-to-real transfer.

The paper addresses a specific scaling failure in massively parallel robot reinforcement learning: increasing the number of simulated environments does not necessarily increase the amount of useful learning signal. When task configurations are sampled uniformly, configurations that are already solved and configurations that are currently beyond the policy’s capabilities consume rollout capacity despite contributing little actionable gradient information. “A Balanced Data Diet: Addressing the Exploration Bottleneck in Mega-Scale RL for Robot Control” (2610.12465) proposes Success Guided Sampling (SGS), an adaptive reset sampler that reallocates experience toward configurations with intermediate empirical success rates.

Problem formulation and central claim

The paper studies goal-conditioned RL for two substantially different domains: multi-terrain quadruped locomotion and contact-rich assembly manipulation. A task configuration $\tau$ comprises the initial state, goal, and environment parameters. For locomotion, this includes terrain type, terrain geometry, robot pose, and target pose. For manipulation, it includes the assembly task, object and robot states, and receptacle pose.

The training pipeline deliberately minimizes task-specific engineering. Locomotion uses one shared reward across all terrains, and manipulation uses one shared reward across all assembly tasks. RL training uses no demonstrations. Exploration is instead supported by a fixed, diverse set of simulator reset configurations, including reaching, stable-grasp, and near-goal states for manipulation. This design builds on the premise that reset diversity can expose the policy to useful intermediate states, but argues that reset diversity alone is insufficient: the learner must also allocate experience non-uniformly across those states.

The paper’s central claim is that adaptive allocation of parallel rollouts is an essential complement to massively parallel simulation. In particular, the authors argue that a simple success-based sampler can outperform uniform sampling, hand-designed curricula, and Prioritized Level Replay (PLR) when the task distribution contains both mastered and currently inaccessible configurations.

Success Guided Sampling

SGS begins with a fixed finite set of task configurations. For each configuration $\tau_i$, it maintains a rolling window of the most recent $H$ binary episode outcomes and estimates the empirical success rate $p_i$. The sampler assigns high probability to configurations whose success rates are near a target value $t$, while retaining a nonzero probability for configurations that are either very easy or currently unsolved.

The weighting function is derived from a Beta-shaped kernel:

$$w_i=(p_i+\epsilon)^{\kappa t}(1-p_i+\epsilon)^{\kappa(1-t)}.$$

The normalized sampling distribution is obtained by applying a temperature-controlled softmax to the logarithm of these weights. The target success rate determines the region of the capability frontier that receives the greatest sampling mass; $\kappa$ controls concentration around that target; and the temperature controls the greediness of configuration selection. The small floor $\epsilon$ ensures that every configuration remains sampleable, which is important because otherwise unsolved configurations could become permanently excluded and their success estimates could never recover.

The method is intentionally not presented as a fundamentally new curriculum principle. It is closely related to learnability-based task selection, including Sampling for Learnability (SFL), which prioritizes configurations according to $p(1-p)$. The contribution is instead algorithmic and empirical: SGS provides a simple, domain-independent implementation of moderate-success sampling and demonstrates that it remains useful when combined with very large-scale PPO training.

For locomotion, the authors use 104,000 task configurations, a target success rate of $0.66$, $\tau_i$0, and a history window of 100 outcomes. For manipulation, they use 32,768 configurations, a target of $\tau_i$1, and $\tau_i$2. The same qualitative mechanism is applied in both domains, although the hyperparameters differ.

Early in training, SGS behaves approximately like broad coverage sampling because success estimates are uninformative. As the policy improves, probability mass moves toward configurations at the current learning frontier. In the multi-terrain locomotion experiment, the sampler first emphasizes relatively accessible transitions, later concentrates on jumping between similarly elevated islands, and eventually shifts toward configurations involving a substantially elevated central pillar. This behavior is not determined by a manually specified ordering of terrain difficulty.

Figure 1: SGS shifts sampling mass from broad initial coverage toward configurations at the policy’s evolving capability frontier.

Experimental design

The experiments evaluate three questions: whether SGS scales more effectively with the number of parallel environments, whether it unlocks capabilities at smaller scales, and whether policies trained with SGS can be transferred to physical hardware.

The main baselines are uniform sampling and PLR. Uniform sampling draws configurations directly from the reset distribution. PLR uses a learnability score based on the absolute generalized advantage estimate, together with a staleness bonus. The paper also compares SGS with SFL in selected locomotion experiments.

The scaling studies hold PPO and optimizer hyperparameters fixed while varying the number of parallel environments. This is a stringent test of data allocation because the batch size increases with environment count while the number of PPO mini-batches remains constant. The experiments reach approximately one million parallel environments. Locomotion runs use up to 1,048,576 environments, while manipulation experiments also evaluate at million-environment scale.

The benchmark suite includes 13 locomotion terrain types and six UR5e NIST Assembly Task-Board tasks, together with Franka nut-and-bolt assembly. The assembly tasks require grasping, object transport, precise alignment, insertion, threading, or gear meshing. Nut-and-bolt assembly is particularly demanding because the robot must thread an M16 nut through approximately 6.5 revolutions.

Figure 2: The benchmark combines six precision assembly tasks with multi-terrain quadruped locomotion requiring sequential agile maneuvers and accurate contacts.

Scaling behavior in locomotion and manipulation

The strongest evidence for SGS concerns the interaction between adaptive sampling and parallelism. At one million parallel environments, SGS obtains 73% mean success on multi-terrain locomotion, compared with 54% for PLR. On Franka nut-and-bolt assembly, SGS reaches 70% success, whereas uniform sampling reaches only 6% and PLR 5%.

Task and scale	Uniform sampling	PLR	SGS
Multi-terrain locomotion, 1M environments	—	54%	73%
Franka nut-and-bolt, 1M environments	6%	5%	70%
UR5e rod-in-hole, 32K environments	0%	0%	Successful learning
UR5e rod-in-hole, 256K environments	Approximately 98%	Approximately 98%	Approximately 98%

The Franka result is particularly consequential because it contradicts the expectation that merely increasing simulator throughput will make uniform sampling adequate. Uniform sampling remains near failure even at one million environments, while SGS produces a high-performing policy under the same nominal scale. The implication is that the bottleneck is not only the total number of transitions but the fraction of transitions that occur near the policy’s learnable frontier.

The authors report monotonic improvement for SGS as the number of environments increases, while competing methods either improve weakly or collapse at larger scales. This supports the paper’s claim that adaptive sampling converts additional parallelism into denser learning signal rather than simply producing larger batches containing redundant or uninformative trajectories.

Figure 3: SGS exhibits stronger scaling on multi-terrain locomotion, Franka nut-and-bolt assembly, and UR5e rod-in-hole insertion.

SGS also provides gains below mega-scale. With only 4,096 parallel environments, it achieves nontrivial multi-terrain locomotion success while uniform sampling and other baselines fail to learn. On UR5e rod-in-hole insertion, SGS learns at 32,000 environments, whereas uniform sampling and PLR obtain zero success at that scale. At 256,000 environments, all methods reach approximately 98%, showing that SGS primarily improves sample efficiency and access to difficult behaviors rather than changing the ultimate ceiling on this simpler task.

The comparison with SFL is also informative. At 4,096 environments, SFL obtains $\tau_i$3 final success and $\tau_i$4 best-checkpoint success, compared with $\tau_i$5 for SGS. At 32,000 environments, SFL’s best checkpoint reaches $\tau_i$6 and its final performance declines to $\tau_i$7, whereas SGS reaches $\tau_i$8. Thus, the paper reports not merely a performance difference but a stability difference: SFL can achieve useful intermediate performance and then deteriorate, while SGS continues improving in the reported runs.

The largest manipulation advantage occurs on nut-and-bolt assembly. Across the additional UR5e experiments, SGS reaches 90.04% simulation success from the reaching reset distribution, compared with 48.83% for uniform sampling and 59.77% for PLR. On five simpler insertion and connector tasks, however, SGS is mostly comparable to the baselines. This asymmetry is important. SGS is not uniformly superior; its largest benefit appears on long-horizon, contact-sensitive tasks in which success depends on chaining grasping, reorientation, alignment, and insertion.

Figure 4: The SGS pipeline targets difficult multi-terrain locomotion and contact-rich manipulation and subsequently distills manipulation policies for RGB-based hardware deployment.

Reset diversity and the role of the capability frontier

SGS does not generate new configurations. It samples from a preconstructed reset distribution. This distinction matters because the method’s effectiveness depends on whether the reset set contains states that connect the current policy to the desired behavior.

The reset ablation on UR5e rod-in-hole insertion demonstrates this dependence. With all three reset families, the policy obtains 99.5% success from stable-grasp states, 96.0% from reaching states, and 100.0% from near-goal states. Removing near-goal resets causes complete failure on both stable-grasp and reaching evaluation: success falls to 0.0% in both cases. Removing reaching resets has almost no effect on the retained distributions, while removing stable-grasp resets reduces performance and causes one of three seeds to collapse.

These results qualify the paper’s claim that SGS reduces engineering. The method substantially reduces the need for manually ordered curricula and task-specific reward design, but it does not eliminate the requirement for a sufficiently informative reset distribution. In particular, near-goal states appear necessary for bootstrapping the insertion behavior in this experiment. SGS determines which available configurations to practice; it does not solve the separate problem of deciding which configurations should exist.

The configuration-density study further shows that diversity is beneficial only up to a point. Increasing the number of locomotion configurations from 52,000 to 104,000 raises success from 0.38 to 0.49, whereas doubling it again to 208,000 produces 0.48. The result suggests that configuration coverage and sampler estimation interact: more configurations can improve generalization and frontier resolution, but excessive discretization may not provide additional benefit under the available rollout budget.

Zero-shot RGB sim-to-real transfer

The paper extends the simulation results to physical UR5e assembly. A state- and point-cloud-based teacher is trained with SGS and then distilled into an RGB student using online DAgger. The student observes third-person and wrist-mounted RGB images, robot state variables, and the previous action. No real-world data or real-world fine-tuning is used.

The transfer tasks are nut-and-bolt assembly, rod-in-hole insertion, and gear mesh insertion. Simulation performance of the RGB students remains close to that of the teachers: 89.05% versus 90.04% for nut-and-bolt, 93.36% versus 95.70% for rod insertion, and 96.42% versus 96.09% for gear meshing.

Real-world performance is substantially lower but nontrivial:

Task	Simulation RGB student	Real success	First-try success	Throughput
Nut-and-bolt assembly	89.05%	37.5% (18/48)	12.5%	0.23/min
Rod-in-hole insertion	93.36%	61.0% (30/49)	28.57%	0.71/min
Gear mesh insertion	96.42%	94.0% (47/50)	76.0%	3.08/min

Figure 5: Zero-shot RGB policies complete nut-and-bolt, rod-in-hole, and gear-mesh assembly trajectories on a UR5e system.

The gear-mesh result is the strongest transfer result, with 94% overall success and 76% first-try success. Rod insertion transfers less reliably, while nut-and-bolt assembly exhibits a large sim-to-real gap and frequent retries. The results therefore support the claim that SGS-trained manipulation policies can transfer zero-shot, but they do not establish uniform transfer robustness across assembly types.

Several implementation choices are directly relevant to this gap. The authors reduce simulated gripper-pad friction from the value used in prior work, $\tau_i$9, to $H$0 because high friction permits physically implausible grasps. At $H$1, $H$2, and $H$3, training from scratch fails, while $H$4 is the lowest tested value that trains stably. This creates a clear trade-off between physical realism and learnability. The simulation remains an approximation, and the selected friction value is an engineering compromise rather than a validated physical estimate.

The distillation experiments also show that observability and reward symmetry are critical. On an auxiliary peg-insertion task, a pose-based teacher yields approximately 55% RGB-student success, whereas an eight-orientation symmetry-aware point-cloud teacher yields 98–99%. The improvement follows from aligning reset symmetries and success criteria with what the camera can distinguish. If the teacher is rewarded for a preferred orientation that is physically equivalent to other orientations but visually ambiguous to the student, distillation introduces an avoidable observability mismatch.

Figure 6: Held-out RGB-student success lags training success during DAgger, demonstrating that online updates can make training performance an unreliable measure of autonomous student competence.

Limitations and open questions

SGS maintains empirical success estimates over a discrete configuration set. The reported configuration counts—32,768 for manipulation and 104,000 for locomotion—are substantial but finite. Continuous or much larger task spaces would require a learned, hierarchical, or function-approximation-based scoring mechanism. The current nonparametric estimator may become statistically inefficient when each configuration receives too few outcomes.

The method also assumes that a meaningful reset distribution has already been constructed. The paper relies on programmatic reset generation and diverse intermediate states; SGS only reweights those states. The complete closed-loop problem of jointly generating configurations and allocating sampling mass remains unresolved.

The real-world experiments leave a substantial sim-to-real gap, especially for nut-and-bolt assembly. The authors modify the simulator, use domain randomization, adjust gripper friction, add observation noise in some settings, and alter symmetry-aware rewards, but the resulting policies still require retries and show task-dependent transfer quality. The hardware results therefore validate feasibility rather than equivalence between simulation and deployment.

Finally, the comparisons do not isolate every interaction among SGS, PPO batch scaling, reset distributions, control interfaces, and auxiliary curricula. The main Franka scaling study uses a gravity curriculum, although a one-seed ablation indicates that it is unnecessary: success is 70.2% with the curriculum and 69.4% without it. The UR5e and Franka results also use different control interfaces, frequencies, and embodiments, so cross-embodiment performance comparisons must be interpreted cautiously.

Conclusion

The paper identifies configuration allocation as a distinct exploration bottleneck in large-scale robot RL. Its empirical contribution is that SGS, a simple success-rate-weighted sampler, can substantially improve both low-scale learning and million-environment scaling when combined with diverse resets, shared domain-level rewards, and PPO. The clearest gains occur on multi-terrain locomotion and long-horizon contact-rich assembly: 73% versus 54% for locomotion at one million environments, and 70% versus 6% for Franka nut-and-bolt assembly against uniform sampling.

The results support a narrower but important conclusion: increasing simulation throughput is effective only when rollout allocation preserves a high density of configurations that are difficult but currently learnable. SGS does not remove the need for reset-distribution design, physical modeling, or sim-to-real validation, but it provides a simple mechanism for making those resources more productive during training.