Contractive Markovian Stochastic Approximation

• Contractive Markovian stochastic approximation is a framework where iterative updates use contractive operators under Markovian noise to guarantee unique convergence in algorithms like Q-learning.
• It integrates ODE limits and Poisson-equation corrections to manage temporal dependencies and derive finite-time error bounds and concentration results.
• The method combines deterministic contraction properties with stochastic approximation techniques to provide robust theoretical guarantees in reinforcement learning applications.

Contractive Markovian stochastic approximation denotes a class of stochastic recursive schemes in which the update noise is generated by a Markov process while the deterministic mean update is governed by a contractive operator. In the reinforcement-learning setting emphasized by "Reinforcement Learning: Stochastic Approximation Algorithms for Markov Decision Processes" (Krishnamurthy, 2015), the basic template is stochastic approximation with Markovian noise, $\theta_{k+1}=\theta_k+\alpha_k H(\theta_k,x_k)$ or, in Robbins–Monro form, $\theta_{k+1}=\theta_k+\alpha_k c(x_k,\theta_k)$, where the driving process is geometrically ergodic and the mean drift is $h(\theta)=\mathbb{E}_{\pi_\theta}[H(\theta,x)]$. The qualifier “contractive” refers to the fact that, in discounted Markov decision processes, Bellman operators are $\gamma$-contractions under $\|\cdot\|_\infty$, so the associated fixed points are unique and stochastic recursions such as value iteration and Q-learning inherit convergence guarantees from that deterministic contraction structure (Krishnamurthy, 2015).

1. Formal framework and basic definitions

The foundational stochastic-approximation recursion is either written with constant step size,

$\theta_{k+1}=\theta_k+\epsilon H(\theta_k,x_k),$

or with decreasing step size,

$$\theta_{k+1}=\theta_k-\alpha_k \widehat{\nabla}_\theta C_k(\theta_k),$$

where the standard Robbins–Monro conditions are

$$\sum_k \alpha_k=\infty,\qquad \sum_k \alpha_k^2<\infty.$$

Typical schedules include $\alpha_k=\epsilon/(k+s)^\zeta$ with $0.5<\zeta\le 1$ (Krishnamurthy, 2015). In the Markovian setting, the noise process is not i.i.d.; rather, it is modeled as a geometrically ergodic Markov process with transition kernel $\theta_{k+1}=\theta_k+\alpha_k c(x_k,\theta_k)$0 and stationary distribution $\theta_{k+1}=\theta_k+\alpha_k c(x_k,\theta_k)$1, so the mean field is parameter dependent and the limiting ODE takes the form $\theta_{k+1}=\theta_k+\alpha_k c(x_k,\theta_k)$2 (Krishnamurthy, 2015).

A complementary operator-theoretic formulation studies iterated random operators on a Polish space. One fixes a contraction $\theta_{k+1}=\theta_k+\alpha_k c(x_k,\theta_k)$3 with modulus $\theta_{k+1}=\theta_k+\alpha_k c(x_k,\theta_k)$4 and unique fixed point $\theta_{k+1}=\theta_k+\alpha_k c(x_k,\theta_k)$5, and defines a random recursion

$\theta_{k+1}=\theta_k+\alpha_k c(x_k,\theta_k)$6

where, for each $\theta_{k+1}=\theta_k+\alpha_k c(x_k,\theta_k)$7, the random maps $\theta_{k+1}=\theta_k+\alpha_k c(x_k,\theta_k)$8 are i.i.d. across time. Under uniform-on-compacts convergence in probability of $\theta_{k+1}=\theta_k+\alpha_k c(x_k,\theta_k)$9 to $h(\theta)=\mathbb{E}_{\pi_\theta}[H(\theta,x)]$0, the induced trajectory laws converge weakly to the deterministic trajectory generated by repeated application of $h(\theta)=\mathbb{E}_{\pi_\theta}[H(\theta,x)]$1 (Gupta et al., 2019). This formulation makes precise a recurrent theme in contractive Markovian stochastic approximation: randomness perturbs an underlying fixed-point iteration rather than replacing it.

The term therefore encompasses two closely related viewpoints. One is classical stochastic approximation with Markovian dependence and diminishing or constant step sizes. The other is constant-step or minibatch recursion viewed as a Markov chain induced by random approximations of a deterministic contraction. Both viewpoints place the contractive mean operator at the center of the theory (Krishnamurthy, 2015).

2. Contractive operators in discounted Markov decision processes

In discounted MDPs, contractivity enters through Bellman operators. For a policy $h(\theta)=\mathbb{E}_{\pi_\theta}[H(\theta,x)]$2, the policy-evaluation Bellman operator is

$h(\theta)=\mathbb{E}_{\pi_\theta}[H(\theta,x)]$3

while the optimality operator is

$h(\theta)=\mathbb{E}_{\pi_\theta}[H(\theta,x)]$4

The discounted-cost Bellman equation may also be written as

$h(\theta)=\mathbb{E}_{\pi_\theta}[H(\theta,x)]$5

and the associated Q-factor recursion is

$h(\theta)=\mathbb{E}_{\pi_\theta}[H(\theta,x)]$6

For $h(\theta)=\mathbb{E}_{\pi_\theta}[H(\theta,x)]$7, the standard property used throughout this literature is that both $h(\theta)=\mathbb{E}_{\pi_\theta}[H(\theta,x)]$8 and $h(\theta)=\mathbb{E}_{\pi_\theta}[H(\theta,x)]$9 are $\gamma$0-contractions under $\gamma$1, with consequence that $\gamma$2 and $\gamma$3 are unique fixed points and that value iteration converges geometrically to the appropriate fixed point (Krishnamurthy, 2015).

This contractive structure extends beyond the ordinary sup-norm formulation. "Stochastic approximation with cone-contractive operators: Sharp $\gamma$4-bounds for $\gamma$5-learning" (Wainwright, 2019) studies monotonicity and quasi-contractivity with respect to a cone-induced gauge norm. In the orthant cone on $\gamma$6 with $\gamma$7, the gauge norm coincides with $\gamma$8, so discounted Q-learning becomes a special case of a more general cone-contractive framework. The paper’s sandwich relation yields pathwise upper and lower bounds on the iterate error and leads to non-asymptotic $\gamma$9 bounds for synchronous Q-learning (Wainwright, 2019).

The central implication is structural rather than merely analytical. Contractivity supplies uniqueness of the deterministic target, global stability of the corresponding mean dynamics, and a norm in which stochastic errors can be controlled. In discounted control, this is why value iteration is a fixed-point iteration, Q-learning is a Robbins–Monro recursion toward a unique $\|\cdot\|_\infty$0, and empirical Bellman updates can be analyzed as random approximations of a contraction (Krishnamurthy, 2015).

3. Markovian noise, ODE limits, and Poisson-equation corrections

The ODE method remains the basic asymptotic device for stochastic approximation under Markovian dependence. Under the assumptions that $\|\cdot\|_\infty$1 is uniformly bounded, that a weak law of large numbers holds for conditional block averages of $\|\cdot\|_\infty$2, and that the ODE $\|\cdot\|_\infty$3 has a unique solution for each initial condition, the interpolated constant-step trajectory converges weakly on finite horizons to the ODE solution. In the Markovian specialization, the drift is exactly the stationary expectation under the parameterized chain,

$\|\cdot\|_\infty$4

which makes the mean field an averaged update under the invariant law of the Markov process (Krishnamurthy, 2015).

The same framework yields diffusion approximations. If

$\|\cdot\|_\infty$5

then, under regularity conditions,

$\|\cdot\|_\infty$6

with

$\|\cdot\|_\infty$7

Near an equilibrium $\|\cdot\|_\infty$8, the asymptotic covariance solves the Lyapunov equation

$\|\cdot\|_\infty$9

When $\theta_{k+1}=\theta_k+\epsilon H(\theta_k,x_k),$0 for a contraction $\theta_{k+1}=\theta_k+\epsilon H(\theta_k,x_k),$1, the associated ODE inherits a unique globally asymptotically stable equilibrium, so contractivity becomes a Lyapunov mechanism for stochastic approximation itself (Krishnamurthy, 2015).

Recent work has sharpened this picture in genuinely Markovian settings. "The ODE Method for Stochastic Approximation and Reinforcement Learning with Markovian Noise" (Liu et al., 2024) extends the Borkar–Meyn stability theorem from martingale-difference noise to Markovian noise. "Almost Sure Convergence Rates and Concentration of Stochastic Approximation and Reinforcement Learning with Markovian Noise" (Qian et al., 2024) introduces a diminishing-interval discretization of the mean ODE, producing a skeleton recursion with diminishing effective step sizes and an averaged noise term. Under geometric mixing, contractive $\theta_{k+1}=\theta_k+\epsilon H(\theta_k,x_k),$2, and step sizes $\theta_{k+1}=\theta_k+\epsilon H(\theta_k,x_k),$3 with $\theta_{k+1}=\theta_k+\epsilon H(\theta_k,x_k),$4, it proves the first almost sure convergence rate for general contractive stochastic approximation with Markovian noise; under $\theta_{k+1}=\theta_k+\epsilon H(\theta_k,x_k),$5, it also proves the first maximal concentration bound with exponential tails for this class (Qian et al., 2024).

A different but closely related correction principle appears in "Almost Sure Convergence Rates of Stochastic Approximation and Reinforcement Learning via a Poisson-Moreau Drift" (Liu et al., 8 May 2026). There the Markovian noise is handled by solving a Poisson equation

$\theta_{k+1}=\theta_k+\epsilon H(\theta_k,x_k),$6

and injecting the corresponding correction into a Moreau-envelope Lyapunov function. For step sizes $\theta_{k+1}=\theta_k+\epsilon H(\theta_k,x_k),$7 with $\theta_{k+1}=\theta_k+\epsilon H(\theta_k,x_k),$8, this yields almost sure rates arbitrarily close to $\theta_{k+1}=\theta_k+\epsilon H(\theta_k,x_k),$9 when $$\theta_{k+1}=\theta_k-\alpha_k \widehat{\nabla}_\theta C_k(\theta_k),$$0, and arbitrarily close to $$\theta_{k+1}=\theta_k-\alpha_k \widehat{\nabla}_\theta C_k(\theta_k),$$1 under harmonic steps. The construction is specifically designed for contractive mean maps, including tabular Q-learning and linear TD methods (Liu et al., 8 May 2026).

4. Canonical algorithms and their stochastic-approximation interpretations

Value iteration is the deterministic prototype: $$\theta_{k+1}=\theta_k-\alpha_k \widehat{\nabla}_\theta C_k(\theta_k),$$2 Its convergence follows directly from Bellman contraction. Q-learning is the stochastic counterpart: $$\theta_{k+1}=\theta_k-\alpha_k \widehat{\nabla}_\theta C_k(\theta_k),$$3 with per-state-action step size

$$\theta_{k+1}=\theta_k-\alpha_k \widehat{\nabla}_\theta C_k(\theta_k),$$4

satisfying the Robbins–Monro conditions componentwise. For finite MDPs, bounded rewards, $$\theta_{k+1}=\theta_k-\alpha_k \widehat{\nabla}_\theta C_k(\theta_k),$$5, sufficient exploration, and infinitely many visits to each state-action pair, Q-learning converges almost surely to $$\theta_{k+1}=\theta_k-\alpha_k \widehat{\nabla}_\theta C_k(\theta_k),$$6, precisely because the Bellman Q-operator has a unique fixed point and the stochastic recursion tracks it (Krishnamurthy, 2015).

The same chapter presents a two-time-scale implementation of Q-learning. The fast time scale updates $$\theta_{k+1}=\theta_k-\alpha_k \widehat{\nabla}_\theta C_k(\theta_k),$$7 over intervals of length $$\theta_{k+1}=\theta_k-\alpha_k \widehat{\nabla}_\theta C_k(\theta_k),$$8 under a fixed policy $$\theta_{k+1}=\theta_k-\alpha_k \widehat{\nabla}_\theta C_k(\theta_k),$$9, while the slow time scale updates the policy greedily according to

$$\sum_k \alpha_k=\infty,\qquad \sum_k \alpha_k^2<\infty.$$0

This already exhibits the actor–critic pattern later abstracted as

$$\sum_k \alpha_k=\infty,\qquad \sum_k \alpha_k^2<\infty.$$1

with fast critic tracking and slow actor adaptation (Krishnamurthy, 2015).

Policy-gradient methods belong to the same stochastic-approximation family but do not rely on Bellman contraction. The paper uses parameterized policies such as the exponential softmax

$$\sum_k \alpha_k=\infty,\qquad \sum_k \alpha_k^2<\infty.$$2

or spherical-coordinate parameterizations, and minimizes the average cost

$$\sum_k \alpha_k=\infty,\qquad \sum_k \alpha_k^2<\infty.$$3

through

$$\sum_k \alpha_k=\infty,\qquad \sum_k \alpha_k^2<\infty.$$4

For Markov processes, the score-function estimator has bias $$\sum_k \alpha_k=\infty,\qquad \sum_k \alpha_k^2<\infty.$$5 and variance $$\sum_k \alpha_k=\infty,\qquad \sum_k \alpha_k^2<\infty.$$6, whereas the weak-derivative estimator has bias $$\sum_k \alpha_k=\infty,\qquad \sum_k \alpha_k^2<\infty.$$7 and variance $$\sum_k \alpha_k=\infty,\qquad \sum_k \alpha_k^2<\infty.$$8; the paper also gives a parameter-free “cut-and-paste” coupling implementation for weak derivatives (Krishnamurthy, 2015). This variance contrast is a recurrent practical reason to separate contractive Bellman-based methods from policy-gradient methods: both are stochastic approximation, but only the former are driven by fixed-point contraction.

The same stochastic-approximation perspective extends to empirical value iteration and empirical Q-value iteration. In the finite-state finite-action case, empirical Bellman operators are random maps that converge uniformly on compacts to the exact Bellman operator, and the resulting Markov chains on value functions satisfy trajectory-level weak convergence to the deterministic contraction trajectory as the batch size grows (Gupta et al., 2019).

5. Finite-time bounds, concentration, steady state, and inference

A first layer of finite-sample control concerns averages of a Markov chain. For a regular transition matrix with Dobrushin coefficient

$$\sum_k \alpha_k=\infty,\qquad \sum_k \alpha_k^2<\infty.$$9

"Theorem 3.1" in (Krishnamurthy, 2015) gives a bias bound, an $\alpha_k=\epsilon/(k+s)^\zeta$0 mean-square-deviation bound, and an exponential concentration inequality for time-average estimators. This quantifies the basic simulation question “how long to simulate a Markov chain?” before stochastic approximation or policy-gradient estimates stabilize (Krishnamurthy, 2015).

A second layer concerns the iterates themselves. "Concentration of Contractive Stochastic Approximation and Reinforcement Learning" (Chandak et al., 2021) proves “from time $\alpha_k=\epsilon/(k+s)^\zeta$1 on” concentration bounds for contractive stochastic approximation with Markov noise and applies them to asynchronous Q-learning and TD(0). For general nonlinear contractive SA with Markovian noise, (Qian et al., 2024) proves maximal concentration with exponential tails and derives $\alpha_k=\epsilon/(k+s)^\zeta$2 rates as a corollary. For two-time-scale stochastic approximation with arbitrary norm contractions and Markovian noise, "Finite-Time Bounds for Two-Time-Scale Stochastic Approximation with Arbitrary Norm Contractions and Markovian Noise" (Chandak et al., 24 Mar 2025) gives the first mean-square bound in the nonlinear setting, with rate $\alpha_k=\epsilon/(k+s)^\zeta$3 in the general case and $\alpha_k=\epsilon/(k+s)^\zeta$4 when the slower timescale is noiseless; it applies these results to SSP Q-learning, Polyak-averaged Q-learning, and learning generalized Nash equilibria (Chandak et al., 24 Mar 2025).

Constant-step regimes require a different asymptotic object: not pointwise convergence, but a stationary distribution. "Steady-State Behavior of Constant-Stepsize Stochastic Approximation: Gaussian Approximation and Tail Bounds" (Wang et al., 15 Feb 2026) studies

$\alpha_k=\epsilon/(k+s)^\zeta$5

under contractive nonlinear drift and Markovian noise. For the centered-and-scaled steady state

$\alpha_k=\epsilon/(k+s)^\zeta$6

it proves a Wasserstein-1 Gaussian approximation bound of order $\alpha_k=\epsilon/(k+s)^\zeta$7, where the Gaussian covariance solves a Lyapunov equation driven by the long-run covariance $\alpha_k=\epsilon/(k+s)^\zeta$8 of the Markovian noise. It also derives non-uniform Berry–Esseen-type tail bounds for one-dimensional projections (Wang et al., 15 Feb 2026).

Inference for averaged stochastic approximation under Markovian dependence has also become explicit. "Statistical inference for Linear Stochastic Approximation with Markovian Noise" (Samsonov et al., 25 May 2025) studies linear SA

$\alpha_k=\epsilon/(k+s)^\zeta$9

under a Hurwitz mean matrix and uniformly geometrically ergodic Markov noise. For Polyak–Ruppert averages, it proves non-asymptotic Berry–Esseen bounds with leading rate $0.5<\zeta\le 1$0 in Kolmogorov distance, establishes a multiplier block bootstrap with error of order $0.5<\zeta\le 1$1 up to logarithmic factors for suitable tuning, and obtains variance-estimation accuracy of order $0.5<\zeta\le 1$2 up to logarithmic factors (Samsonov et al., 25 May 2025).

6. Extensions, misconceptions, and limitations

One common misconception is that contraction is synonymous with stochastic approximation. It is not. In the Bellman-based setting, contraction of $0.5<\zeta\le 1$3 or $0.5<\zeta\le 1$4 supplies existence, uniqueness, and global stability of the target fixed point. In policy-gradient stochastic approximation, by contrast, convergence is organized around stability of the mean drift rather than Bellman contraction, and the central technical issue is often variance and bias of gradient estimators under Markovian dependence rather than fixed-point uniqueness (Krishnamurthy, 2015).

A second misconception is that Markovian noise is merely a mild variant of i.i.d. noise. The more recent literature treats its dependence structure as a first-order analytical obstacle. The weak convergence of whole path measures for iterated random contractions, the Poisson-equation corrections used in ODE and Lyapunov methods, and the block-based bootstrap required for inference all reflect the fact that the noise is temporally dependent and often state dependent (Gupta et al., 2019).

The framework also extends beyond discounted tabular control. The 2015 survey repeatedly notes that Q-learning and policy-gradient schemes can be used as suboptimal methods for partially observed Markov decision processes, and the discounted belief-MDP formulation preserves Bellman contractivity on bounded belief-value functions (Krishnamurthy, 2015). At the same time, several limitations recur across the literature: independence or identical-distribution assumptions on random maps in the iterated-operator setting, geometric ergodicity or uniform geometric ergodicity of the Markov chain, boundedness or projection requirements, and the restriction to settings where the mean operator is contractive or pseudo-contractive (Gupta et al., 2019).

Finally, contractivity does not erase algorithmic trade-offs. In synchronous discounted Q-learning, the non-asymptotic $0.5<\zeta\le 1$5 analysis of (Wainwright, 2019) shows that the worst-case sample complexity in $0.5<\zeta\le 1$6 is worse than that of model-based Q-iteration: model-free synchronous Q-learning scales as $0.5<\zeta\le 1$7 to $0.5<\zeta\le 1$8, whereas model-based Q-iteration achieves the minimax-optimal $0.5<\zeta\le 1$9 dependence. This result does not contradict contraction; it clarifies that a contractive fixed point guarantees identifiability and stability, not optimal statistical efficiency (Wainwright, 2019).

In that sense, contractive Markovian stochastic approximation is best understood as an overview of three ingredients. The first is a Markovian noise model, typically controlled through ergodicity, Poisson equations, or operator-level weak convergence. The second is a contractive mean operator, most familiarly a Bellman operator in a discounted MDP, but more generally a contraction in an arbitrary norm or cone-induced gauge. The third is a stochastic-approximation analysis—ODE, diffusion, martingale, or steady-state—that converts those two structural properties into convergence, concentration, and inference statements.