Almost Sure Convergence Rates of Stochastic Approximation and Reinforcement Learning via a Poisson-Moreau Drift

Abstract: Establishing almost sure convergence rates for stochastic approximation and reinforcement learning under Markovian noise is a fundamental theoretical challenge. We make progress towards this challenge for a class of stochastic approximation algorithms whose expected updates are contractive, a setting that arises in many reinforcement learning algorithms such as $Q$-learning and linear temporal difference learning. Specifically, for a power-law learning rate $O(n^{-η})$ with $η\in (1/2, 1)$, we obtain an almost sure convergence rate arbitrarily close to $o(n^{1 - 2η})$. For a harmonic learning rate $O(n^{-1})$, we obtain an almost sure convergence rate arbitrarily close to $o(n^{-1})$, which we argue is a strong result because it is close to the optimal rate $O(n^{-1}\log\log n)$ given by the law of the iterated logarithm (for a special case of i.i.d. noise). Key to our analysis is a novel Lyapunov drift construction that applies a Poisson-equation based correction for Markovian noise to the well-established Moreau-envelope smoothing for the contractive mapping.

• The paper introduces a Poisson-Moreau drift-based Lyapunov function that cancels Markovian bias for almost sure convergence.
• It establishes sharp convergence rates for SA and RL with contractive mappings under power-law learning rates.
• The analysis unifies and improves upon prior results for i.i.d. and Markovian noise models, enhancing practical RL robustness.

Almost Sure Convergence Rates in Stochastic Approximation and RL: Poisson-Moreau Drift

Introduction and Problem Setting

The analysis of almost sure (a.s.) convergence rates for stochastic approximation (SA) algorithms, especially under Markovian noise, remains a central challenge in the theoretical study of reinforcement learning (RL) and related online, data-driven algorithms. The canonical SA recursion,

$$\theta_{n+1} = \theta_n + \alpha_n F(\theta_n, Y_n),$$

serves as the backbone of stochastic gradient descent (SGD), temporal-difference (TD) learning, and $Q$-learning. The solution trajectory $\{\theta_n\}$ is typically analyzed for convergence in mean, $L^p$, or probability; however, the a.s. mode is fundamentally stronger, yielding trajectory-level guarantees.

Prior results often confine themselves to weaker noise models, such as i.i.d. sequences, or impose strong mixing assumptions that do not hold for general Markov chains. Additionally, even when a.s. convergence is established, quantitative convergence rates are elusive, especially for nonlinear contractive mappings or non-Euclidean contraction norms, which are ubiquitous in modern RL (e.g., $Q$-learning with $\ell_\infty$ contractivity).

This work introduces a novel Lyapunov drift argument based on a Poisson-equation correction to the Moreau-envelope energy function, leading to sharp a.s. convergence rates for contractive SA under Markovian noise (2605.07104). The construction achieves rates close to the best known for the i.i.d. case and covers general nonlinear, norm-contracting mappings.

Main Results: Convergence Rates and Regimes

Consider SA with Markovian noise $Y_n$ driven by a Markov kernel $P$ with invariant distribution $\pi$. The mean update $f(\theta) = \int F(\theta, y) \pi(dy)$ defines a contractive mapping $Q$0 under norm $Q$1, with contractivity parameter $Q$2. The general setting includes RL algorithms such as $Q$3-learning and linear TD with Hurwitz matrix.

Learning Rate and Convergence

For power-law learning rates $Q$4 with $Q$5, the key result establishes:

• If $Q$6, $Q$7 a.s. for any $Q$8,
• If $Q$9, $\{\theta_n\}$0 a.s. for any $\{\theta_n\}$1, with the rate arbitrarily close to $\{\theta_n\}$2, matching the iterated logarithm rate in the i.i.d. case (see LIL).

These rates are sharper than those obtainable by previous Markovian-noise analyses, especially for linear/nonlinear contractive operators and non-Euclidean (e.g., $\{\theta_n\}$3) norms, which are standard outside of i.i.d. algorithms. The analysis also delivers $\{\theta_n\}$4 convergence rates for all $\{\theta_n\}$5.

Poisson-Moreau Drift Construction: Technical Innovations

Classical Lyapunov methods (e.g., squared Euclidean norm) are insufficient when contraction is not in the Euclidean norm or when the noise is non-i.i.d. The Moreau envelope, $\{\theta_n\}$6, is a smoothing of $\{\theta_n\}$7 that preserves contractivity in an equivalent norm and offers smoothness for analysis. However, under Markovian noise, naively using $\{\theta_n\}$8 as a Lyapunov function results in additional uncontrolled bias terms that cannot be handled pathwise.

The innovation here is the construction of a corrected Lyapunov function ("Poisson-Moreau drift"): $\{\theta_n\}$9 where $L^p$0 and $L^p$1 solves the Poisson equation $L^p$2. The Poisson correction cancels the leading-order Markovian bias in the conditional drift and preserves a pathwise (almost) supermartingale property, critical for the application of a.s. rate theorems.

This Lyapunov construction requires delicate control of the regularization parameter $L^p$3 in the Moreau envelope and a suitable choice of $L^p$4 to ensure coercivity. The proof leverages norm equivalence in finite dimensions and bounds the remaining bias and martingale difference terms using properties of the Markov chain and Lipschitz continuity.

Applications: Generalized Temporal Difference Learning

The methodology is demonstrated for a generalized class of tabular TD algorithms, covering on- and off-policy $L^p$5-step TD, Retrace, Tree-Backup, $L^p$6-trace, and $L^p$7. For the linear SA formulation,

$L^p$8

where $L^p$9 summarizes the TD increment structure (with general importance sampling factors and eligibility traces), the analysis confirms that the algorithmic assumptions are satisfied under mild mixing and coverage (irreducible and aperiodic Markov chain), and that the Hurwitz condition for the mean matrix ensures contractivity.

The a.s. convergence rates then follow directly for these general TD algorithms, establishing that policy evaluation via single-trajectory sampling attains pathwise sharp rates under minimal structural and noise assumptions, thus justifying the practical robustness of such methods for RL.

Comparison to Prior Work

Earlier works either restrict to i.i.d. noise, linear operators, or require mixing rates that are summable (decay to zero), which is not the case for general Markov chains. Approaches based on skeleton iterates or expectation-only Lyapunov recursions cannot deliver the same class of a.s. rates or do so under narrower conditions and slower rates. This analysis closes the gap for general contractive, possibly nonlinear SA with Markovian dynamics and non-Euclidean contractivity.

Theoretical and Practical Implications

The main consequences are:

• Pathwise guarantees: Practitioners can expect every real algorithm trajectory (beyond in-distribution or expectation) to converge at a rate essentially matching the optimal in the i.i.d. limit, even in the presence of Markovian temporal correlations.
• Generality: The results unify and strengthen previous analyses for RL algorithms, including complex, off-policy, or non-Euclidean settings.
• Modularity of analysis: The Poisson-Moreau construction can, in principle, be adapted to broader classes of SA and RL, including two-timescale or averaged algorithms, and can be combined with high-probability and $Q$0 results.

Potential extensions may address open issues such as a.s. rates for $Q$1, matching the iterated logarithm rates, or the synthesis of tail bounds with exponential decay under Markovian noise.

Conclusion

This work develops a Poisson-equation-corrected Lyapunov drift, culminating in explicit and sharp almost sure convergence rates for contractive stochastic approximation (including RL algorithms) under Markovian noise (2605.07104). The construction addresses limitations of prior Lyapunov-based and expectation-based analyses, delivers non-asymptotic, trajectory-wise guarantees, and can accommodate contractive mappings in arbitrary norms. The theoretical advances support both the robustness of practical RL algorithms under realistic data-generating processes and future generalizations in the theory of SA and RL.