Success Visitation Matching in Sparse-Reward RL

• Success Visitation Matching (SVM) is a reward-shaping method that redefines binary success signals as dense, step-level rewards based on state-action visitation patterns.
• It uses a discriminator trained on successful versus unsuccessful episodes to compute a log density ratio that guides the agent’s progress.
• Empirical results in robotic finetuning show SVM reduces training steps by up to 50% while achieving success rates consistently above 80%.

to=arxiv_search 红鼎json {"query":"\"Success Visitation Matching\" reinforcement learning sparse reward 2026", "max_results": 10} to=arxiv_search 天天中彩票微信json {"query":"(Bolland et al., 2024)"} Success Visitation Matching (SVM) is a reward-shaping method for sparse-reward reinforcement learning that turns a binary success signal into a dense step-level training signal by learning to distinguish transitions from successful versus unsuccessful episodes, then rewarding the policy for matching the visitation patterns of successful episodes and avoiding those of unsuccessful ones (Tsao et al., 22 Jun 2026). In the formulation introduced under this name, the natural task reward is an outcome reward that is $0$ everywhere and $+1$ only when the task is completed, while the shaped reward is defined over state-action visitations at each timestep. Closely related precursor work did not use the term “Success Visitation Matching,” but studied conditional discounted future visitation distributions and KL-based target matching over future states, actions, or features (Bolland et al., 2024, Bolland et al., 19 Mar 2026).

1. Sparse outcome rewards and the SVM objective

The motivating setting is sparse-reward RL in which the MDP reward is binary success reward $r:S\to\{0,1\}$, and the policy objective is

$$J(\pi) := E^\pi\!\left[\sum_{h=1}^{H} r(s_h)\right].$$

Because the environment transitions to a terminal state after success, this is effectively the success probability of the policy (Tsao et al., 22 Jun 2026).

SVM distinguishes between a sparse outcome reward and a dense process reward. The sparse outcome reward is the binary success/completion reward $r(s)$, only nonzero at successful states. The dense process reward is a step-level reward that provides feedback during the trajectory about whether the agent is making progress toward success. The central intuition is that if a state-action pair tends to appear in trajectories that eventually succeed, then visiting it is evidence of progress, whereas if a state-action pair appears mostly in failed trajectories, it is evidence of poor progress (Tsao et al., 22 Jun 2026).

Given a collected dataset of episodes $D$, SVM partitions it into successful episodes $D^+$ and unsuccessful episodes $D^-$. For each timestep $h$, it defines visitation estimates $w_h^+(s,a)$ and $+1$0 as estimates of the state-action visitation density at step $+1$1 for successful and unsuccessful episodes respectively. The exact SVM reward is

$+1$2

with

$+1$3

The paper defines $+1$4 and $+1$5 (Tsao et al., 22 Jun 2026).

This definition is explicitly timestep-indexed and state-action based. It does not directly match whole trajectory distributions in the main definition; it is closest to matching stepwise state-action occupancy or visitation distributions (Tsao et al., 22 Jun 2026). A plausible implication is that SVM targets local progress structure throughout the episode rather than only terminal success states.

2. Visitation densities, discriminators, and the learned process reward

For the theoretical result, SVM specializes to empirical count-based visitation estimates. For successful episodes,

$+1$6

where $+1$7 is all observations at step $+1$8 from $+1$9, and $r:S\to\{0,1\}$0 is defined analogously from $r:S\to\{0,1\}$1 (Tsao et al., 22 Jun 2026).

The policy-induced state-action visitation distribution is

$r:S\to\{0,1\}$2

This quantity supports the paper’s KL-regularized interpretation of the shaped objective (Tsao et al., 22 Jun 2026).

To scale beyond explicit density estimation, SVM replaces direct estimation of $r:S\to\{0,1\}$3 and $r:S\to\{0,1\}$4 with binary classification. For each timestep $r:S\to\{0,1\}$5, it fits a discriminator

$r:S\to\{0,1\}$6

Assuming sufficient expressivity,

$r:S\to\{0,1\}$7

Applied to SVM,

$r:S\to\{0,1\}$8

The implementable reward is therefore

$r:S\to\{0,1\}$9

This reward is used inside RL (Tsao et al., 22 Jun 2026).

The resulting signal is dense because the discriminator can score every visited state-action, not just the final success state. Positive shaped reward is produced when $$J(\pi) := E^\pi\!\left[\sum_{h=1}^{H} r(s_h)\right].$$0 is high, meaning the transition looks more like success data than failure data, and negative shaped reward is produced when it looks more like failure data (Tsao et al., 22 Jun 2026). The method uses online experience only: successful episodes become positives, unsuccessful ones negatives.

3. Occupancy-matching and optimality preservation

Without clipping, the expected shaped objective satisfies

$$J(\pi) := E^\pi\!\left[\sum_{h=1}^{H} r(s_h)\right].$$1

By inserting $$J(\pi) := E^\pi\!\left[\sum_{h=1}^{H} r(s_h)\right].$$2, the paper rewrites this as

$$J(\pi) := E^\pi\!\left[\sum_{h=1}^{H} r(s_h)\right].$$3

which yields the KL form

$$J(\pi) := E^\pi\!\left[\sum_{h=1}^{H} r(s_h)\right].$$4

Under this interpretation, the policy is encouraged to reduce divergence to successful visitations and increase divergence from unsuccessful visitations (Tsao et al., 22 Jun 2026).

The main theoretical guarantee is Theorem 1. Under deterministic transitions, possibly stochastic initial state distribution, countable $$J(\pi) := E^\pi\!\left[\sum_{h=1}^{H} r(s_h)\right].$$5 and $$J(\pi) := E^\pi\!\left[\sum_{h=1}^{H} r(s_h)\right].$$6, empirical count-based visitation estimates from any dataset $$J(\pi) := E^\pi\!\left[\sum_{h=1}^{H} r(s_h)\right].$$7, and the assumption that success states transition to a terminal state thereafter, any policy maximizing $$J(\pi) := E^\pi\!\left[\sum_{h=1}^{H} r(s_h)\right].$$8 also maximizes $$J(\pi) := E^\pi\!\left[\sum_{h=1}^{H} r(s_h)\right].$$9 (Tsao et al., 22 Jun 2026). The authors emphasize that the guarantee holds for any previous set of episodes $r(s)$0, regardless of which policy generated them, and does not require oracle visitation estimates.

The proof divides states into not-success-reachable and success-reachable classes. For not-success-reachable states, no action can improve success. For success-reachable states, backward induction is used to show that any trajectory maximizing the SVM reward must continue choosing actions that keep success reachable; otherwise it would incur lower shaped return than a path that preserves reachability and eventually receives the true success reward (Tsao et al., 22 Jun 2026). This distinguishes SVM from arbitrary heuristic shaping: the shaping term is derived from relative visitation statistics of successful versus unsuccessful behavior, while the original sparse reward remains present.

A common misconception is to treat SVM as merely a classifier-derived bonus. The KL identity shows a more precise characterization: it is regularized RL that pushes policy occupancy toward successful occupancy and away from unsuccessful occupancy (Tsao et al., 22 Jun 2026).

4. Algorithmic procedure and integration with off-policy RL

The practical algorithm is given as “Reinforcement Learning with SVM Process Reward.” Its steps are: collect initialization data, split episodes into $r(s)$1 and $r(s)$2, train the discriminator, initialize the policy, and then iterate by rolling out the current policy, relabeling the episode by success or failure, updating the discriminator on $r(s)$3 and $r(s)$4, and updating the policy with RL on

$r(s)$5

(Tsao et al., 22 Jun 2026).

The framework itself is generic, but the practical instantiations are mostly off-policy RL. The paper lists DSRL-SAC for diffusion-policy finetuning, Residual RL with SAC, and RLPD in the demonstrations setting (Tsao et al., 22 Jun 2026). Because replay buffers are maintained and the policy and discriminator are repeatedly updated from accumulated data, the practical setup is largely off-policy.

Although the theory uses timestep-conditioned discriminators $r(s)$6, the experiments state that “we train a single discriminator for all steps $r(s)$7 together rather than an individual discriminator for each step,” and an ablation further shows timestep conditioning is not important on average (Tsao et al., 22 Jun 2026). This suggests that the step-indexed theory is compatible with a simpler implementation in which a single state-action classifier provides the shaped reward.

The paper also reports that using RL on the SVM reward is much better than directly maximizing $r(s)$8, action filtering or sampling by $r(s)$9, or adding behavior cloning on successful episodes (Tsao et al., 22 Jun 2026). Different monotone transforms of $D$0 help, but

$D$1

generally works best. Negative feedback is important: penalizing similarity to unsuccessful episodes matters, and mixing successful examples into the negative set degrades performance (Tsao et al., 22 Jun 2026).

5. Empirical behavior in robotic finetuning

The main evaluation domain is robotic manipulation finetuning with sparse binary success reward. The environments are LIBERO-90, RoboCasa, the real-world WidowX 250 6-DoF robot arm, Robomimic, and $D$2 VLA finetuning on LIBERO tasks (Tsao et al., 22 Jun 2026).

Across all three LIBERO kitchen scenes and both DSRL and Residual RL, SVM improves both sample efficiency and final success rate over the baselines listed in the paper. With DSRL, SVM reaches significantly higher final success after $D$3 timesteps and converges much faster. With Residual RL, SVM often needs $D$4 or more fewer samples to converge, or reaches substantially higher success than baselines. Final performance with SVM consistently exceeds $D$5 (Tsao et al., 22 Jun 2026).

Across all three RoboCasa tasks, SVM quickly reaches around $D$6 success, while other methods often fail to improve RL at all. On the real-world WidowX tasks, SVM substantially improves finetuning speed, reaches over $D$7 success with many fewer environment steps, and on Pick and Place and Open Drawer, outcome-only RL fails to learn while SVM succeeds (Tsao et al., 22 Jun 2026). In $D$8 VLA finetuning on four LIBERO tasks, SVM requires roughly $D$9 fewer environment steps to reach $D^+$0 success than sparse reward alone. In RL from demonstrations on Robomimic Can and Square with 200 demonstrations, SVM clearly improves over outcome-only RL and performs approximately equal to standard GAIL (Tsao et al., 22 Jun 2026).

Several ablations clarify the method’s behavior. Timestep conditioning is not necessary on average. Symmetric positive/negative sampling is generally better than asymmetric sampling. Heatmaps show that SVM quickly concentrates visitation around successful regions, while alternatives are more diffuse or focus on failure regions (Tsao et al., 22 Jun 2026).

These results position SVM primarily as a finetuning method for sparse-reward control. The paper’s strongest empirical emphasis is not on outperforming imitation methods under demonstrations, but on transforming sparse outcome rewards into dense process rewards during online RL.

6. Relation to future-visitation methods, related paradigms, and limitations

SVM is closely related to a line of work on conditional discounted future visitation distributions. “Off-Policy Maximum Entropy RL with Future State and Action Visitation Measures” defines

$D^+$1

and uses the intrinsic reward

$D^+$2

where $D^+$3 is the discounted future visitation distribution of features generated from downstream state-action pairs (Bolland et al., 2024). The later paper “Maximum-Entropy Exploration with Future State-Action Visitation Measures” studies the same family of conditional discounted future feature distributions and proves contraction-based fixed-point results for their Bellman-style operators (Bolland et al., 19 Mar 2026). Neither paper explicitly introduces “Success Visitation Matching,” but both provide a learned object that is a conditional successor visitation distribution over future features, together with KL-based target matching (Bolland et al., 2024, Bolland et al., 19 Mar 2026).

The difference in emphasis is substantial. In SVM proper, the target signal is derived from successful and unsuccessful episodes collected online, and the learned reward is

$D^+$4

or, equivalently in the exact formulation, a timestep-wise log density ratio between successful and unsuccessful visitation estimates (Tsao et al., 22 Jun 2026). In the future-visitation papers, the target is a reference distribution $D^+$5, often uniform, and the objective is exploration through future visitation entropy or relative entropy (Bolland et al., 2024, Bolland et al., 19 Mar 2026). A plausible implication is that SVM occupies the success-conditioned reward-shaping end of a broader visitation-matching family, whereas the future-visitation papers occupy the exploration-oriented target-matching end.

The relation to adversarial imitation is explicit. SVM is very close to adversarial inverse RL or GAIL because it uses a discriminator and the reward

$D^+$6

The difference is that in GAIL, positives are fixed expert demonstrations and negatives are policy rollouts, whereas in SVM, positives are successful episodes, negatives are unsuccessful episodes, and both are updated online using the observed outcome reward (Tsao et al., 22 Jun 2026). The relation to preference-based reward learning is also explicit: compared with methods such as SORS, SVM does not learn from pairwise trajectory rankings; it learns a binary density-ratio signal at the state-action level (Tsao et al., 22 Jun 2026).

The main limitation explicitly discussed for SVM is the theory’s reliance on deterministic transitions. Extending the guarantee to stochastic settings is left open. Other practical caveats stated in the paper are that if there are too few successes, the discriminator may be weak early on; if success and failure trajectories are not separable in the chosen representation, reward quality may suffer; and in tasks where fixed demonstrations already fully define success patterns, SVM may reduce to something close to GAIL (Tsao et al., 22 Jun 2026). These caveats delimit the regime in which success visitation statistics provide reliable dense process rewards.