Success Guided Sampling in RL

• Success Guided Sampling (SGS) is an adaptive reset-sampling method in reinforcement learning that targets task configurations near the current capability frontier to maximize learning efficiency.
• Key applications of SGS include multi-terrain quadruped locomotion and contact-rich assembly tasks, achieving significant improvements in performance and efficiency over uniform sampling methods.
• The approach uses a beta-shaped weighted mechanism to efficiently sample configurations with a specified target success probability, essentially creating a dynamic curriculum for learning.

Success Guided Sampling (SGS) is an adaptive reset-sampling method for massively parallel reinforcement learning that concentrates experience on task configurations near the current capability frontier of a policy. Rather than sampling simulator resets uniformly, SGS estimates the recent success probability of each configuration and preferentially selects configurations that are neither already mastered nor effectively unreachable. The method was introduced for large-scale robot-control training, where it enabled multi-terrain quadruped locomotion and contact-rich assembly with up to $2^{20}$ parallel environments, and later supported zero-shot transfer of distilled RGB policies to real hardware (Zhang et al., 8 Oct 2026). The term is also used more broadly for feedback-driven sampling schemes, including reward-guided, variance-guided, energy-guided, and geometry-aware procedures, but these related methods do not all implement the specific SGS reset sampler.

1. Concept and motivation

SGS addresses an exploration bottleneck caused by uniform sampling over a broad distribution of initial states, goals, terrain configurations, object poses, or task parameters. Uniform sampling allocates approximately equal probability to configurations regardless of the policy’s current competence. This produces two inefficient regimes. Configurations with success probability close to one are already mastered and generate limited new learning signal, whereas configurations with success probability close to zero may produce almost exclusively failures and little directly useful gradient in sparse-reward, long-horizon tasks.

The useful region lies between these extremes. A configuration with an intermediate success probability produces both successful and failed trajectories. Successful rollouts provide positive experience, while failures identify current limitations. For a binary outcome $Y$ with success probability $p$, the outcome variance is

$\operatorname{Var}(Y)=p(1-p),$

which is maximized at $p=0.5$. SGS generalizes this intuition by specifying a target success probability $t$ that need not equal $0.5$. Locomotion experiments use $t=0.66$, whereas manipulation experiments use $t=0.50$ (Zhang et al., 8 Oct 2026).

The resulting sampler implements an automatically moving curriculum. Early in training, many configurations have low success rates and the distribution remains comparatively broad. As the policy masters easier configurations, their success estimates increase and their sampling weights decrease. Probability mass consequently shifts toward more difficult configurations whose success rates remain intermediate. The curriculum is based on measured capability rather than a manually specified difficulty ordering.

SGS is distinct from selecting successful trajectories solely because they succeeded. Its target is the capability frontier: configurations that the policy can solve sometimes, but not reliably. This distinction is important because repeatedly sampling only successful configurations can reduce exposure to failure modes, while repeatedly sampling nearly impossible configurations can yield little positive learning signal.

2. Formal formulation

The robot-learning problem is modeled as a goal-conditioned Markov decision process

$$\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{G},P,r,\gamma,\rho),$$

where $Y$0 is the state space, $Y$1 the action space, $Y$2 the goal space, $Y$3 the transition dynamics, $Y$4 the reward function, $Y$5 the discount factor, and $Y$6 the distribution over initial-state/goal pairs. A task configuration is represented as

$Y$7

with initial state $Y$8, goal $Y$9, and environment configuration $p$0. SGS samples among a fixed set of configurations obtained from the task distribution; it does not generate new configurations during training.

For configuration $p$1, the policy’s success probability is

$p$2

Because the exact probability is unavailable, SGS estimates it from the most recent $p$3 binary outcomes:

$p$4

The implementation uses a circular buffer, replacing the oldest outcome when the buffer is full. Consequently, $p$5 is a rolling estimate rather than a lifetime average. This allows the sampler to respond to changing policy competence: configurations that were previously impossible may become useful as transferable skills develop, while formerly difficult configurations may become mastered.

The unnormalized configuration weight is a Beta-shaped function:

$p$6

where $p$7 prevents numerical singularities, $p$8 is the target success probability, and $p$9 controls concentration around the target. The weighting corresponds to the kernel

$\operatorname{Var}(Y)=p(1-p),$0

The resulting log-weight is

$\operatorname{Var}(Y)=p(1-p),$1

Configurations are sampled according to a temperature-controlled categorical distribution:

$\operatorname{Var}(Y)=p(1-p),$2

Lower $\operatorname{Var}(Y)=p(1-p),$3 produces greedier selection of high-weight configurations, whereas higher $\operatorname{Var}(Y)=p(1-p),$4 smooths exploration across the configuration set. The reported experiments use $\operatorname{Var}(Y)=p(1-p),$5 and retain nonzero selection probability for every configuration.

For locomotion, the configuration distribution includes terrain type, terrain level or seed, start state, and goal state:

$\operatorname{Var}(Y)=p(1-p),$6

The locomotion set contains 13 terrain types, two seeds per terrain type, 10 levels, 20 start states, and 20 goal states, yielding 104,000 configurations. For manipulation,

$\operatorname{Var}(Y)=p(1-p),$7

where $\operatorname{Var}(Y)=p(1-p),$8 denotes a reset strategy or reset configuration. The reported manipulation buffer contains 32,768 collision-checked configurations distributed across Near-Goal, Stable Grasp, and Reaching reset families.

3. Algorithm and implementation

SGS begins by sampling a fixed configuration set $\operatorname{Var}(Y)=p(1-p),$9 from $p=0.5$0 and associating each configuration with a buffer of its most recent $p=0.5$1 outcomes. The reported value is $p=0.5$2. At each episode reset, the sampler computes the current rolling success estimate, evaluates the Beta-shaped weight, applies the temperature-scaled categorical distribution, and selects a configuration.

After the episode, SGS determines whether the task succeeded, appends the binary outcome to the selected configuration’s buffer, discards the oldest outcome if necessary, and updates the rolling estimate. The process closes the loop:

$p=0.5$3

The method is compatible with PPO and shared task rewards. In massively parallel training, multiple configurations are selected across environments, and assignment statistics are updated as episode outcomes become available. The sampler is applied throughout training rather than imposing a hand-designed sequence of task difficulties.

The principal hyperparameters are the target success probability $p=0.5$4, concentration parameter $p=0.5$5, history length $p=0.5$6, numerical floor $p=0.5$7, and sampling temperature $p=0.5$8. The reported settings are:

Domain	Configurations	$p=0.5$9	$t$0	$t$1	$t$2
Locomotion	$t$3	100	0.66	5	2
Manipulation	$t$4	100	0.50	1	2

The method requires a discrete configuration table containing a current score for every configuration. This makes the implementation straightforward for finite reset sets but limits direct application to continuous or extremely large configuration spaces. Every configuration must also retain a nonzero probability of selection, since a configuration with a low current success estimate may be unreachable because of inadequate training rather than intrinsic difficulty.

SGS is an allocation mechanism rather than a reset-generation method. Its efficacy depends on the initial reset distribution containing sufficiently diverse and meaningful configurations. In manipulation, the reported reset families expose different capability levels: Near-Goal states bootstrap precise contact behavior, Stable Grasp states require transport and alignment, and Reaching states require object acquisition before assembly. Removing Near-Goal resets prevented learning in the tested rod-in-hole runs, demonstrating that reweighting cannot compensate for absent reset coverage (Zhang et al., 8 Oct 2026).

4. Relation to adaptive sampling methods

SGS belongs to a broader family of feedback-driven sampling methods but differs from several related approaches in both signal and operating level.

Uniform reset sampling preserves the original task distribution but wastes experience on solved or currently inaccessible configurations. Hand-designed curricula impose a human-defined ordering of difficulty and therefore require task-specific engineering. Sampling for Learnability (SFL) prioritizes configurations using $t$5, corresponding to a fixed target of $t$6. SGS instead uses a Beta-shaped weighting function with an adjustable target $t$7, rolling estimates, temperature-controlled sampling, and explicit nonzero support.

Prioritized Level Replay (PLR) uses temporal-difference or generalized-advantage-related signals and staleness. SGS uses direct task success, which is particularly interpretable in sparse-reward robotics. The distinction is not that SGS introduces the general principle of sampling learnable states: it operationalizes that principle through a target success probability, rolling statistics, and integration with large-scale parallel simulation.

SGS also differs from success-biased replay in which successful episodes or trajectories are preferentially replayed. SGS samples task configurations, such as terrain/start/goal combinations or object-reset states, rather than selecting individual successful trajectories. A configuration can receive high probability despite containing failures, provided its recent success rate is near the target. Conversely, a configuration with perfect success can be down-weighted because it no longer lies on the capability frontier.

The method is conceptually distinct from Adaptive Weighted SGD, which learns a sampling distribution to minimize importance-weighted gradient variance and whose unrestricted optimum is proportional to gradient norm (Bouchard et al., 2015). It is also distinct from Adaptive Sample with Reward, which treats sampling-distribution adjustment as an RL policy optimized for cumulative validation improvements in representation learning (Dou et al., 2022). Those methods adapt sampling based on gradient variance or downstream reward, respectively; SGS uses rolling task success at the reset-configuration level.

Energy-guided diffusion samplers provide another related but different mechanism. Inference-time adaptation for combinatorial optimization can use an objective and constraint energy to modify a pretrained diffusion prior, repeatedly noising and denoising candidate solutions (Lei et al., 15 Feb 2025). Such methods guide the generation trajectory toward high-quality solutions but do not use the Beta-shaped reset sampler or a rolling success estimate. Similarly, DPM-Solver++ accelerates numerical integration of strongly guided diffusion trajectories but is a solver rather than a success-selection policy (Lu et al., 2022).

5. Empirical applications

Locomotion

The locomotion system trains an Anymal-D quadruped jointly across 13 terrain types, including slopes, pits, gaps, stairs, beams, balance beams, stepping stones, navigation mazes, contoured terrains, and floating-island environments requiring jumping and precise foothold placement. The configuration set contains 104,000 combinations of terrain, level, start state, and goal state.

The policy observes base linear and angular velocity, projected gravity, joint positions and velocities, previous action, goal command, and a local terrain-height scan. Actions are 12-dimensional joint-position offsets. The shared reward combines terminal success and failure terms with regularizers for work, posture, touchdown velocity, undesired contacts, and forward motion. Terminal success and failure have coefficients of $t$8 and $t$9.

At approximately one million parallel environments, SGS reaches a reported multi-terrain success rate of $0.5$0, compared with $0.5$1 for PLR. SGS improves with simulation scale, whereas competing methods may saturate or collapse. In a floating-island example, the sampling distribution initially covers configurations broadly, later concentrates on jumps between islands of similar height, and eventually shifts toward configurations involving a central pillar with a substantial height difference. This behavior illustrates a capability-driven curriculum without manually ordering terrain difficulty.

Compared with SFL, SGS reports the following results:

Parallel environments	SFL final	SFL best	SGS final
4K	$0.5$2	$0.5$3	$0.5$4
32K	$0.5$5	$0.5$6	$0.5$7

SFL’s best checkpoint approaches SGS at 32K environments, but its performance declines substantially later in training, whereas SGS continues improving.

Manipulation

The manipulation experiments use UR5e and Franka systems with tasks drawn from the NIST Assembly Task Board. These include nut-and-bolt threading, rod-in-hole insertion, gear mesh insertion, waterproof connector insertion, BNC assembly, and rectangular peg insertion. The reset distribution contains Near-Goal, Stable Grasp, and Reaching configurations.

At one million environments, Franka nut-and-bolt assembly achieves $0.5$8 success with SGS, compared with $0.5$9 for uniform sampling and $t=0.66$0 for PLR. The corresponding ablation reports $t=0.66$1 with a gravity curriculum and $t=0.66$2 without it, indicating that the gravity curriculum is not essential.

For UR5e rod-in-hole insertion, SGS succeeds at 32K environments, while uniform sampling and PLR obtain zero success at that scale. At 256K environments, all methods reach approximately $t=0.66$3, showing that sufficiently large brute-force parallelism can eventually make simpler tasks learnable.

Across six UR5e tasks evaluated from the Reaching reset distribution, SGS obtains the strongest reported result on nut-and-bolt assembly, with $t=0.66$4 success compared with $t=0.66$5 for PLR and $t=0.66$6 for uniform sampling. On simpler insertion tasks, all methods perform strongly once adequate scale is available.

6. Scaling, transfer, limitations, and interpretation

SGS is designed to make large batches more useful rather than merely larger. The reported training used up to approximately one million parallel environments, with up to 32,768 locomotion environments or 16,384 manipulation environments per GPU on L40S and H200 hardware. Approximate training times were 36 hours for locomotion and 128 hours for Franka nut-and-bolt assembly.

The learned manipulation policies were distilled into RGB-based policies using DAgger. The student observes third-person and wrist-camera RGB images, joint positions, end-effector pose, and previous action. The teacher provides target actions in simulation. No real-world data or real-world fine-tuning is used. Symmetry-aware training randomizes equivalent object orientations and accepts equivalent successful goal orientations, improving simulation performance for an OmniReset peg-insertion task from approximately $t=0.66$7 for a pose-based baseline to about $t=0.66$8 with an eight-orientation, point-cloud teacher and symmetry-aware rewards and resets.

The resulting RGB policies transfer zero-shot to real UR5e hardware:

Task	Teacher simulation	RGB simulation	Real success
Nut-and-bolt	90.04%	89.05%	37.5%
Rod-in-hole	95.70%	93.36%	61%
Gear mesh	96.09%	96.42%	94%

The principal limitations are the discrete configuration table, dependence on reset coverage, the need for reliable binary success labels, and the residual simulation-to-real gap. Continuous or extremely large task spaces may require learned, hierarchical, or clustered configuration representations. Rolling estimates can also be noisy when configurations are visited infrequently, and a configuration with low success may be either genuinely difficult or simply undertrained. The exploration floor is therefore essential.

SGS is sensitive to the history length, concentration, temperature, target success rate, and configuration-set size. In locomotion, reported success at 32K environments varied from approximately $t=0.66$9 to $t=0.50$0 across tested settings, compared with $t=0.50$1 for the nominal configuration. Increasing the configuration set from 52,000 to 104,000 helped, while increasing it further to 208,000 produced little additional improvement.

The method should not be interpreted as a universal guarantee of faster learning or higher final performance. Its benefits depend on the reset distribution containing configurations that expose a meaningful progression of capabilities, and on the success signal accurately reflecting task completion. It does not replace reward design, reset generation, simulator realism, or coverage of essential capability levels. Its specific contribution is to adaptively allocate experience among available configurations according to recent success.

In its narrow technical sense, SGS is therefore a success-probability-guided reset sampler:

$t=0.50$2

Its broader significance is methodological. It treats data allocation as an active component of RL optimization, using observed capability to control future exposure. In this respect, SGS connects task curriculum design, adaptive experience allocation, sparse-reward exploration, and large-scale simulation into a single feedback loop.