---
title: Success Visitation Matching in Sparse-Reward RL
url: https://www.emergentmind.com/topics/success-visitation-matching-svm
type: topic
---

# Success Visitation Matching in Sparse-Reward RL

to=arxiv_search  红鼎json
{"query":"\"Success Visitation Matching\" reinforcement learning sparse reward 2026", "max_results": 10}
to=arxiv_search  天天中彩票微信json
{"query":"2412.06655"}
Success Visitation Matching (SVM) is a reward-shaping method for sparse-reward reinforcement learning that turns a binary success signal into a dense step-level training signal by learning to distinguish transitions from successful versus unsuccessful episodes, then rewarding the policy for matching the visitation patterns of successful episodes and avoiding those of unsuccessful ones [2606.23640]. In the formulation introduced under this name, the natural task reward is an outcome reward that is \(0\) everywhere and \(+1\) only when the task is completed, while the shaped reward is defined over state-action visitations at each timestep. Closely related precursor work did not use the term “Success Visitation Matching,” but studied conditional discounted future visitation distributions and KL-based target matching over future states, actions, or features [2412.06655][2603.18965].

## 1. Sparse outcome rewards and the SVM objective

The motivating setting is sparse-reward RL in which the MDP reward is binary success reward \(r:S\to\{0,1\}\), and the policy objective is
\[
J(\pi) := E^\pi\!\left[\sum_{h=1}^{H} r(s_h)\right].
\]
Because the environment transitions to a terminal state after success, this is effectively the success probability of the policy [2606.23640].

SVM distinguishes between a **sparse outcome reward** and a **dense process reward**. The sparse outcome reward is the binary success/completion reward \(r(s)\), only nonzero at successful states. The dense process reward is a step-level reward that provides feedback during the trajectory about whether the agent is making progress toward success. The central intuition is that if a state-action pair tends to appear in trajectories that eventually succeed, then visiting it is evidence of progress, whereas if a state-action pair appears mostly in failed trajectories, it is evidence of poor progress [2606.23640].

Given a collected dataset of episodes \(D\), SVM partitions it into successful episodes \(D^+\) and unsuccessful episodes \(D^-\). For each timestep \(h\), it defines visitation estimates \(w_h^+(s,a)\) and \(w_h^-(s,a)\) as estimates of the state-action visitation density at step \(h\) for successful and unsuccessful episodes respectively. The exact SVM reward is
\[
\hat r_h(s,a) := r(s) + \lambda \cdot clip_\beta \left ( \log \frac{w_h^+(s,a)}{w_h^-(s,a)} \right ),
\]
with
\[
clip_\beta(x) := \begin{cases}
 \beta \cdot \mathrm{sign}(x) & |x| > \beta \\
 x & |x| \le \beta .
\end{cases}
\]
The paper defines \(\log 0 := -\infty\) and \(\log 0/0 := -\infty\) [2606.23640].

This definition is explicitly timestep-indexed and state-action based. It does not directly match whole trajectory distributions in the main definition; it is closest to matching stepwise state-action occupancy or visitation distributions [2606.23640]. A plausible implication is that SVM targets local progress structure throughout the episode rather than only terminal success states.

## 2. Visitation densities, discriminators, and the learned process reward

For the theoretical result, SVM specializes to empirical count-based visitation estimates. For successful episodes,
\[
w_h^+(s,a) := \frac{1}{|D^+|} \cdot \sum_{(s',a') \in D_h^+} I \{ (s',a') = (s,a) \},
\]
where \(D_h^+\) is all observations at step \(h\) from \(D^+\), and \(w_h^-\) is defined analogously from \(D^-\) [2606.23640].

The policy-induced state-action visitation distribution is
\[
w_h^\pi(s,a) = \mathbb{P}^\pi[s_h = s, a_h = a].
\]
This quantity supports the paper’s KL-regularized interpretation of the shaped objective [2606.23640].

To scale beyond explicit density estimation, SVM replaces direct estimation of \(w_h^+\) and \(w_h^-\) with binary classification. For each timestep \(h\), it fits a discriminator
\[
f_h^* := {\textstyle \argmin_{f} \ E_{(s,a) \sim D_h^+}[\log f(s,a)]  + E_{(s,a) \sim D_h^-}[\log (1 - f(s,a))]}.
\]
Assuming sufficient expressivity,
\[
f_h^*(x) = \frac{P(x)}{P(x)+Q(x)}, \qquad \frac{f_h^*(x)}{1-f_h^*(x)} = \frac{P(x)}{Q(x)}.
\]
Applied to SVM,
\[
\frac{f_h(s,a)}{1-f_h(s,a)} \approx \frac{w_h^+(s,a)}{w_h^-(s,a)}.
\]
The implementable reward is therefore
\[
\hat r_h(s,a) = r(s) + \lambda \cdot clip_\beta \left (  \log \tfrac{f_h(s,a)}{1-f_h(s,a)} \right ).
\]
This reward is used inside RL [2606.23640].

The resulting signal is dense because the discriminator can score every visited state-action, not just the final success state. Positive shaped reward is produced when \(f_h(s,a)\) is high, meaning the transition looks more like success data than failure data, and negative shaped reward is produced when it looks more like failure data [2606.23640]. The method uses online experience only: successful episodes become positives, unsuccessful ones negatives.

## 3. Occupancy-matching and optimality preservation

Without clipping, the expected shaped objective satisfies
\[
E^\pi\!\left[\sum_{h=1}^H \hat r_h(s_h,a_h)\right] = J(\pi) + \lambda \cdot \sum_{h=1}^H E^\pi \left[ \log \frac{w_h^+(s_h,a_h)}{w_h^-(s_h,a_h)} \right].
\]
By inserting \(w_h^\pi\), the paper rewrites this as
\[
\textstyle E^\pi[\sum_{h=1}^H \hat r_h(s_h,a_h)]
= J(\pi) + \lambda \cdot \sum_{h=1}^H E^\pi \left[ \log \frac{w^\pi_h(s_h,a_h)}{w_h^+(s_h,a_h)} - \log \frac{w^\pi_h(s_h,a_h)}{w_h^-(s_h,a_h)} \right],
\]
which yields the KL form
\[
\textstyle E^\pi[\sum_{h=1}^H \hat r_h(s_h,a_h)] = J(\pi) + \lambda \cdot \sum_{h=1}^H \left[ KL(w^\pi_h \,\|\, w_h^-) - KL(w^\pi_h \,\|\, w_h^+) \right].
\]
Under this interpretation, the policy is encouraged to reduce divergence to successful visitations and increase divergence from unsuccessful visitations [2606.23640].

The main theoretical guarantee is Theorem 1. Under deterministic transitions, possibly stochastic initial state distribution, countable \(S\) and \(A\), empirical count-based visitation estimates from any dataset \(D\), and the assumption that success states transition to a terminal state thereafter, any policy maximizing \(\hat r_h(s,a)\) also maximizes \(r(s)\) [2606.23640]. The authors emphasize that the guarantee holds for any previous set of episodes \(D\), regardless of which policy generated them, and does not require oracle visitation estimates.

The proof divides states into not-success-reachable and success-reachable classes. For not-success-reachable states, no action can improve success. For success-reachable states, backward induction is used to show that any trajectory maximizing the SVM reward must continue choosing actions that keep success reachable; otherwise it would incur lower shaped return than a path that preserves reachability and eventually receives the true success reward [2606.23640]. This distinguishes SVM from arbitrary heuristic shaping: the shaping term is derived from relative visitation statistics of successful versus unsuccessful behavior, while the original sparse reward remains present.

A common misconception is to treat SVM as merely a classifier-derived bonus. The KL identity shows a more precise characterization: it is regularized RL that pushes policy occupancy toward successful occupancy and away from unsuccessful occupancy [2606.23640].

## 4. Algorithmic procedure and integration with off-policy RL

The practical algorithm is given as “Reinforcement Learning with SVM Process Reward.” Its steps are: collect initialization data, split episodes into \(D^+\) and \(D^-\), train the discriminator, initialize the policy, and then iterate by rolling out the current policy, relabeling the episode by success or failure, updating the discriminator on \(D^+\) and \(D^-\), and updating the policy with RL on
\[
\hat r_h(s,a) = r(s) + \lambda \cdot clip_\beta \left (  \log \tfrac{f_h(s,a)}{1-f_h(s,a)} \right )
\]
[2606.23640].

The framework itself is generic, but the practical instantiations are mostly off-policy RL. The paper lists **DSRL-SAC** for diffusion-policy finetuning, **Residual RL with SAC**, and **RLPD** in the demonstrations setting [2606.23640]. Because replay buffers are maintained and the policy and discriminator are repeatedly updated from accumulated data, the practical setup is largely off-policy.

Although the theory uses timestep-conditioned discriminators \(f_h\), the experiments state that “we train a single discriminator for all steps \(h\) together rather than an individual discriminator for each step,” and an ablation further shows timestep conditioning is not important on average [2606.23640]. This suggests that the step-indexed theory is compatible with a simpler implementation in which a single state-action classifier provides the shaped reward.

The paper also reports that using RL on the SVM reward is much better than directly maximizing \(f\), action filtering or sampling by \(f\), or adding behavior cloning on successful episodes [2606.23640]. Different monotone transforms of \(f\) help, but
\[
\log \frac{f}{1-f}
\]
generally works best. Negative feedback is important: penalizing similarity to unsuccessful episodes matters, and mixing successful examples into the negative set degrades performance [2606.23640].

## 5. Empirical behavior in robotic finetuning

The main evaluation domain is robotic manipulation finetuning with sparse binary success reward. The environments are **LIBERO-90**, **RoboCasa**, the real-world **WidowX 250 6-DoF robot arm**, **Robomimic**, and \(\pi_0\) VLA finetuning on LIBERO tasks [2606.23640].

Across all three LIBERO kitchen scenes and both DSRL and Residual RL, SVM improves both sample efficiency and final success rate over the baselines listed in the paper. With DSRL, SVM reaches significantly higher final success after \(2 \cdot 10^5\) timesteps and converges much faster. With Residual RL, SVM often needs \(2\times\) or more fewer samples to converge, or reaches substantially higher success than baselines. Final performance with SVM consistently exceeds \(80\%\) [2606.23640].

Across all three RoboCasa tasks, SVM quickly reaches around \(90\%\) success, while other methods often fail to improve RL at all. On the real-world WidowX tasks, SVM substantially improves finetuning speed, reaches over \(80\%\) success with many fewer environment steps, and on Pick and Place and Open Drawer, outcome-only RL fails to learn while SVM succeeds [2606.23640]. In \(\pi_0\) VLA finetuning on four LIBERO tasks, SVM requires roughly \(2\times\) fewer environment steps to reach \(90\%\) success than sparse reward alone. In RL from demonstrations on Robomimic Can and Square with 200 demonstrations, SVM clearly improves over outcome-only RL and performs approximately equal to standard GAIL [2606.23640].

Several ablations clarify the method’s behavior. Timestep conditioning is not necessary on average. Symmetric positive/negative sampling is generally better than asymmetric sampling. Heatmaps show that SVM quickly concentrates visitation around successful regions, while alternatives are more diffuse or focus on failure regions [2606.23640].

These results position SVM primarily as a finetuning method for sparse-reward control. The paper’s strongest empirical emphasis is not on outperforming imitation methods under demonstrations, but on transforming sparse outcome rewards into dense process rewards during online RL.

## 6. Relation to future-visitation methods, related paradigms, and limitations

SVM is closely related to a line of work on conditional discounted future visitation distributions. “Off-Policy Maximum Entropy RL with Future State and Action Visitation Measures” defines
\[
d^{\pi,\gamma}(\bar s,\bar a \mid s,a) = (1-\gamma)\,\pi(\bar a\mid \bar s)\sum_{\Delta=1}^\infty \gamma^\Delta p^\pi_\Delta(\bar s\mid s,a)
\]
and uses the intrinsic reward
\[
R^{int}(s,a) = - KL_z\!\left[q^\pi(z\mid s,a)\,\|\, q^*(z)\right],
\]
where \(q^\pi(z\mid s,a)\) is the discounted future visitation distribution of features generated from downstream state-action pairs [2412.06655]. The later paper “Maximum-Entropy Exploration with Future State-Action Visitation Measures” studies the same family of conditional discounted future feature distributions and proves contraction-based fixed-point results for their Bellman-style operators [2603.18965]. Neither paper explicitly introduces “Success Visitation Matching,” but both provide a learned object that is a conditional successor visitation distribution over future features, together with KL-based target matching [2412.06655][2603.18965].

The difference in emphasis is substantial. In SVM proper, the target signal is derived from successful and unsuccessful episodes collected online, and the learned reward is
\[
\log \frac{f(s,a)}{1-f(s,a)}
\]
or, equivalently in the exact formulation, a timestep-wise log density ratio between successful and unsuccessful visitation estimates [2606.23640]. In the future-visitation papers, the target is a reference distribution \(q^*\), often uniform, and the objective is exploration through future visitation entropy or relative entropy [2412.06655][2603.18965]. A plausible implication is that SVM occupies the success-conditioned reward-shaping end of a broader visitation-matching family, whereas the future-visitation papers occupy the exploration-oriented target-matching end.

The relation to adversarial imitation is explicit. SVM is very close to adversarial inverse RL or GAIL because it uses a discriminator and the reward
\[
\log \frac{f(s,a)}{1-f(s,a)}.
\]
The difference is that in GAIL, positives are fixed expert demonstrations and negatives are policy rollouts, whereas in SVM, positives are successful episodes, negatives are unsuccessful episodes, and both are updated online using the observed outcome reward [2606.23640]. The relation to preference-based reward learning is also explicit: compared with methods such as SORS, SVM does not learn from pairwise trajectory rankings; it learns a binary density-ratio signal at the state-action level [2606.23640].

The main limitation explicitly discussed for SVM is the theory’s reliance on deterministic transitions. Extending the guarantee to stochastic settings is left open. Other practical caveats stated in the paper are that if there are too few successes, the discriminator may be weak early on; if success and failure trajectories are not separable in the chosen representation, reward quality may suffer; and in tasks where fixed demonstrations already fully define success patterns, SVM may reduce to something close to GAIL [2606.23640]. These caveats delimit the regime in which success visitation statistics provide reliable dense process rewards.

Source: https://www.emergentmind.com/topics/success-visitation-matching-svm