Learning Process Rewards via Success Visitation Matching for Efficient RL

Abstract: In many modern applications of reinforcement learning (RL), the natural reward for a task of interest is inherently sparse: a reward of 0 is given everywhere except when the task is completed, when a reward of +1 is given. Training a policy to maximize such a sparse reward requires solving a challenging credit assignment problem, leading to slow or ineffective RL improvement. We propose a simple approach to transform a sparse outcome reward into a dense process reward. Our approach relies on training a discriminator to distinguish between previous successful and unsuccessful episodes, and using this discriminator to incentivize the RL-learned policy to match the state-action visitations of successful episodes, while avoiding those of unsuccessful episodes. By incentivizing the policy to match the visitations over all states, not just those that correspond to task success, this reward provides dense feedback on whether progress is being made towards task completion, and, we show, provably achieves this without changing the optimal policy. Focusing on finetuning of robotic control policies, we demonstrate that our approach leads to significantly faster RL finetuning performance on both simulated and real-world manipulation tasks, as compared to simply maximizing the sparse outcome reward.

• The paper introduces Svm, which constructs dense process rewards by using discriminator-based success visitation matching to enhance sparse-reward RL.
• The paper proves that in deterministic settings, policies maximizing Svm rewards are optimal, with experiments showing faster convergence and higher success rates in robotic tasks.
• The paper demonstrates that Svm scales to high-dimensional environments through classification, outperforming alternative methods with approximately 2× faster RL convergence.

Learning Process Rewards via Success Visitation Matching for Efficient RL

Motivation and Problem Formulation

The paper "Learning Process Rewards via Success Visitation Matching for Efficient RL" (2606.23640) addresses a longstanding challenge in reinforcement learning: efficiently training policies with inherently sparse outcome rewards, particularly in robotic settings. The outcome reward signals are only received upon successful task completion, resulting in a severe credit assignment problem and suboptimal sample efficiency for RL. Existing reward shaping approaches often depend on additional supervision or task-specific side information, limiting their applicability in domains such as robotics where such resources are scarce.

Success Visitation Matching: Methodology and Theoretical Properties

The authors propose Success Visitation Matching (Svm), an automated, provably-correct method for constructing dense process rewards from observed outcome rewards. Svm utilizes episodic trajectories labeled by success or failure (Figure 1), trains a discriminator to distinguish state-action pairs in successful versus unsuccessful episodes, and uses the resulting log-ratio as a process reward:

$$r_h^{\text{svm}}(s, a) := r^{\text{out}}(s) + \lambda \cdot \text{clip}_\beta \left( \log \frac{w_h^+(s, a)}{w_h^-(s, a)} \right)$$

where $w_h^+$ and $w_h^-$ denote estimated state-action visitation densities at step $h$ for successful and unsuccessful episodes, respectively.

The reward is dense: it incentivizes visiting states and actions associated with prior successful trajectories, providing step-level progress signals even before full task completion. Importantly, the authors prove that, in deterministic environments, policies maximizing this Svm process reward are also optimal for the original sparse outcome reward. This is achieved by leveraging the empirical visitation densities, without requiring oracle information, and by regularizing the process reward appropriately.

Figure 1: Svm framework—trajectories labeled with sparse outcome reward yield a discriminator, producing dense process reward and enabling efficient policy finetuning.

Svm shows a close connection to KL-regularized RL: the Svm reward maximization objective encourages the learned policy's visitation distribution to have low KL-divergence to the distribution of successful episodes and high divergence to unsuccessful ones, thus guiding learning toward task success while preserving optimality.

Implementation: Scalable Reward Estimation

To scale Svm to large or continuous state spaces, the paper replaces explicit density estimation with classification. A discriminator $f_{h}$ is trained to distinguish between state-action pairs from successful and unsuccessful episodes; its score is used to estimate the visitation ratio via:

$$\frac{f_h(s, a)}{1 - f_h(s, a)} \approx \frac{w_h^+(s, a)}{w_h^-(s, a)}$$

This approach sidesteps explicit density computations, extending Svm to high-dimensional, real-world robotic control.

Experimental Results: RL Finetuning and Sample Efficiency

The empirical evaluation focuses on RL finetuning of robotic control policies using Svm process rewards, across simulated (LIBERO-90, RoboCasa) and real-world (WidowX 250 6-DoF robot arm) benchmarks. The experiments target two main questions: whether Svm enables faster RL improvement versus sparse outcome reward RL, and whether the optimal policy under Svm achieves high outcome reward.

Aggregated results demonstrate that Svm process rewards significantly accelerate convergence and improve final success rates compared to both outcome-reward-only RL and other process reward shaping baselines. Notably, Svm rewards enable approximately $2\times$ faster convergence in RL finetuning of vision-language-action models ($\pi_{0}$), and outperform methods relying on preference modeling, kernel density estimation, and vision-LLMs.

Figure 2: Aggregated RL finetuning results with Dsrl and Svm on LIBERO Scenes 1–3 (16 tasks); Svm achieves superior sample efficiency and final performance.

Svm also scales effectively to Residual RL in RoboCasa kitchen tasks (Figure 3), consistently providing robust learning where outcome-reward-only RL or alternatives fail.

Figure 3: RL finetuning with Svm process reward on real-world WidowX robot tasks; Svm enables efficient learning in practical, non-deterministic environments.

Policy Extraction, Reward Form and Ablations

Ablation studies systematically analyze Svm’s components:

• Policy extraction from the discriminator: RL training using the Svm reward performs better than maximizing the discriminator score directly or filtered action sampling.
• Functional form of process reward: $\log f/(1-f)$ as the reward yields best results; monotonic alternatives offer gains but are less effective.
• Negative feedback: Penalizing visits to unsuccessful regions, via explicit negative feedback, accelerates learning and improves success rates.
• Sampling: Symmetric sampling from positive and negative episodes for discriminator training is critical; asymmetric approaches degrade performance.

State visitation heatmaps reveal that Svm rapidly concentrates trajectory density in successful regions (Figure 4), providing a robust mechanism for task progress signaling.

Figure 4: State visitation heatmaps for different reward shaping methods at step 20,000; Svm shapes trajectories toward task-relevant regions.

Implications and Future Directions

This work has broad theoretical and practical implications. Svm enables efficient RL training in sparse-reward, real-world domains without needing expert demonstrations or task-specific engineering. The approach is valid under standard RL assumptions and is scalable via discriminator-based estimation, making it applicable to diverse robotic manipulation and potentially to LLM process reward design paradigms.

Theoretically, Svm provides a policy-invariant reward transformation, aligning with classic reward shaping results [ng1999policy] but generalized via visitation densities. A primary limitation is the deterministic environment assumption in proofs. However, experimental results indicate scalability to non-deterministic settings.

Future directions include extending theory to stochastic environments, applying Svm in RL with verifiable rewards for LLMs, and integrating more sophisticated state-action embeddings for highly complex domains.

Conclusion

"Learning Process Rewards via Success Visitation Matching for Efficient RL" introduces an automated, theoretically sound, and empirically validated method for constructing dense process rewards from sparse outcome rewards in RL. Svm leverages state-action visitation distributions and discriminator-based estimation to provide informative progress signals, accelerating RL convergence and achieving robust performance across simulated and real robotic tasks. The approach holds considerable promise for both robotics and broader RL domains seeking efficient learning from sparse signals.