Pathwise KL Accumulation in RL

• Pathwise KL accumulation is a reinforcement learning concept that cumulatively applies per-step KL penalties to constrain policy drift.
• It enables error averaging along trajectories, improving stability and sample efficiency in algorithms like TRPO, MPO, and SAC.
• By controlling policy divergence, it offers robust error propagation and convergence properties via a dual-averaging approach.

Pathwise KL accumulation is a core concept in modern reinforcement learning (RL), describing the cumulative application of Kullback–Leibler (KL) regularization across the state or trajectory level to control policy updates. Fundamentally, it refers both to the formulation of regularized RL objectives that penalize per-step policy divergence, and to the analysis of how such penalties accumulate along trajectories or across iterations, yielding favorable error-propagation properties and robustness in learning. This mechanism underlies the stability and performance of a broad class of algorithms that control policy drift by accumulating discounted KL-divergences, either explicitly as part of optimization constraints or implicitly within regularized Bellman operators (Voelcker et al., 15 Jul 2025, Vieillard et al., 2020).

1. Formal Definitions and Mathematical Framework

Given an old (behavior) policy $\pi_{θ′}$ and a candidate new policy $\pi_{θ}$, the per-step divergence at state $s_t$ is

$$D_{KL}\bigl[\pi_{θ′}(\cdot\mid s_t)\,\|\;\pi_{θ}(\cdot\mid s_t)\bigr] = \int_a \pi_{θ′}(a\mid s_t)\,\log\frac{\pi_{θ′}(a\mid s_t)}{\pi_{θ}(a\mid s_t)}\,da.$$

To constrain policy drift, these local divergences are accumulated along a trajectory of length $T$: $$\mathbb{E}_{s_{0:T}\sim\pi_{θ′}}\Bigl[\sum_{t=0}^{T-1} \gamma^t D_{KL}\bigl[\pi_{θ′}(\cdot\mid s_t)\,\|\;\pi_{θ}(\cdot\mid s_t)\bigr] \Bigr].$$ The total pathwise KL cost thus quantifies, in the occupancy measure of the old policy, the discounted sum of per-step divergences from $\pi_{θ′}$ to $\pi_{θ}$ (Voelcker et al., 15 Jul 2025).

Within value-iteration schemes, this framework generalizes further. Given regularization scales $\lambda, \tau \ge 0$, KL- and entropy-regularized Bellman operators and greedy-steps are defined as (Vieillard et al., 2020): $$G^{\lambda,\tau}_\mu(q) = \arg\max_{\pi}\langle \pi, q \rangle - \lambda\, KL(\pi\|\mu) + \tau H(\pi)$$ and

$\pi_{θ}$0

2. Optimization Objective and Algorithmic Realization

Policy learning with pathwise KL accumulation is naturally posed as a constrained optimization problem:

• Objective: Maximize the entropy-augmented expected return,

$\pi_{θ}$1

• Constraint: Bound the cumulative KL,

$\pi_{θ}$2

Introducing a Lagrange multiplier $\pi_{θ}$3, the problem is relaxed to the saddle point: $\pi_{θ}$4 In practice, intractable trajectory expectations are estimated using on-policy rollouts and a learned critic $\pi_{θ}$5, yielding the differentiable actor loss over batch $\pi_{θ}$6: $\pi_{θ}$7 The stochastic gradient step $\pi_{θ}$8 implements pathwise KL accumulation, with each state's divergence directly shaping the update (Voelcker et al., 15 Jul 2025).

3. Error Propagation and Theoretical Advantages

A salient feature of pathwise KL accumulation is its effect on error propagation in approximate dynamic programming. For standard approximate value iteration (AVI) without KL, the bound on the sub-optimality of the final policy scales quadratically with the effective horizon as $\pi_{θ}$9, and the per-iteration Bellman errors $s_t$0 enter via their maximum: $s_t$1 In contrast, adding per-step KL penalties transforms this accumulation into a pathwise average, reducing the horizon dependence to linear: $s_t$2 Thus, approximation errors propagate only through their average, not their worst-case value (Vieillard et al., 2020). The heart of this result is a dual-averaging argument, whereby the pathwise KL regularizer yields a telescoping (averaging) effect across iterations or along the trajectory.

In the more general case with both KL and entropy regularizers, each $s_t$3 enters via a moving average with decay $s_t$4, offering a trade-off between convergence and variance reduction.

4. Empirical Consequences and Algorithmic Stability

Pathwise KL regularization has several important empirical consequences:

• Stability of Policy Improvement: By tightly controlling the per-step drift from the behavior policy, methods such as Relative Entropy Pathwise Policy Optimization (REPPO) achieve significantly more stable policy improvement, as measured by aggregate returns and the rate of reliably converged runs, than standard score-based approaches (e.g., PPO) (Voelcker et al., 15 Jul 2025).
• Sample Efficiency: The KL constraint permits larger, yet safe, update steps when the critic is accurate. This yields improved sample efficiency, allowing REPPO to reach high performance in both locomotion and manipulation tasks with fewer environment interactions than PPO, and comparable performance to off-policy methods such as SAC or FastTD3, with orders-of-magnitude less replay memory required.
• Robustness: Joint tuning of dual variables $s_t$5 keeps the balance between exploration (entropy) and update conservatism (KL penalty), supporting robust performance across a diverse set of tasks with a single hyperparameter configuration (Voelcker et al., 15 Jul 2025).
• Error Compensation: For many practical algorithms, the incremental Bellman errors $s_t$6 have approximately zero mean when averaged over time. Pathwise KL accumulation ensures that only this average (rather than maximum or sum) impacts convergence, conferring robustness to stochastic approximation (Vieillard et al., 2020).

5. Design Principles and Generalization

Pathwise KL accumulation crystallizes into a general design principle for RL algorithms: Any method that (i) adds a local KL-penalty $s_t$7 at each update step, and (ii) incorporates this penalty both in policy improvement and value evaluation, will inherit the linear-horizon, pathwise-averaging error bound. Crucially, omitting the KL penalty from the evaluation step forfeits the error-averaging property. This mechanism underpins the stability of trust region and regularized policy optimization methods (e.g., TRPO, MPO, SAC).

In summary, the per-step KL-penalty accumulates sublinearly along the trajectory, owing to the implicit averaging mechanism, and not as a discounted sum or worst-case accumulation. As a result, RL algorithms featuring pathwise KL accumulation achieve more favorable stability, sample efficiency, and error resilience than their unregularized counterparts (Vieillard et al., 2020).

6. Constraints, Hyperparameterization, and Practical Considerations

The efficacy of pathwise KL accumulation hinges on appropriate regularization hyperparameters. The KL penalty coefficient $s_t$8 (or Lagrange multiplier $s_t$9 in dual formulations) must be tuned to balance linear horizon scaling and sufficient error averaging. In mixed regularization with entropy ($$D_{KL}\bigl[\pi_{θ′}(\cdot\mid s_t)\,\|\;\pi_{θ}(\cdot\mid s_t)\bigr] = \int_a \pi_{θ′}(a\mid s_t)\,\log\frac{\pi_{θ′}(a\mid s_t)}{\pi_{θ}(a\mid s_t)}\,da.$$0), the moving average decay $$D_{KL}\bigl[\pi_{θ′}(\cdot\mid s_t)\,\|\;\pi_{θ}(\cdot\mid s_t)\bigr] = \int_a \pi_{θ′}(a\mid s_t)\,\log\frac{\pi_{θ′}(a\mid s_t)}{\pi_{θ}(a\mid s_t)}\,da.$$1 governs the trade-off between convergence speed and variance reduction. Excessive KL regularization may impede policy improvement, while insufficient regularization amplifies instability (Vieillard et al., 2020).

Modern RL implementations typically employ dual variable updates in log-space for both entropy and KL constraints, automatically steering the system toward user-specified entropy and KL targets (Voelcker et al., 15 Jul 2025). This mechanism allows a single hyperparameter set to yield robust training behavior across a wide range of domains.

7. Implications for Algorithmic Families and Theoretical Understanding

Pathwise KL accumulation offers a principled mechanism that explains the empirical success of KL-regularized methods, including TRPO, MPO, and SAC. Its key impact is the replacement of worst-case or cumulative error propagation with an averaging effect. This both reduces the theoretical worst-case gap and confers robustness in settings where approximation errors are stochastic or ergodic.

A plausible implication is that the dual-averaging perspective may inform further regularization schemes, potentially extending error-averaging properties to more general nonlinear or nonstationary optimization settings. The explicit pathwise perspective clarifies how to systematically design algorithms that automatically balance exploration and conservatism via local, accumulated divergences.

References: (Voelcker et al., 15 Jul 2025, Vieillard et al., 2020)