A Diffusion Approximation for Temporal-Difference Learning with Linear Features under Markovian Noise

Abstract: Temporal difference (TD) learning with linear function approximation is a core method for policy evaluation. Its classical continuous-time description is an ordinary differential equation (ODE), which captures the asymptotic mean dynamics but neglects stochastic fluctuations determining the error floor. We introduce a stochastic differential equation (SDE) approximation for linear TD(0) under Markovian noise. The resulting model distinguishes the contraction dynamics governed by the projected Bellman operator from the influence of Markovian sampling. As a consequence, the model explains the constant-stepsize error floor through the interaction between Markovian long-run covariance and the contraction geometry of the projected Bellman operator.

• The paper introduces a diffusion approximation framework using SDEs to capture both mean behavior and stochastic fluctuations in TD learning under Markovian noise.
• It employs Poisson equation techniques to decompose the update noise, linking Markov chain mixing properties to the resultant error floor and variance structure.
• Simulations validate that the SDE closely tracks discrete TD iterates, offering practical guidelines for algorithm design and stepsize selection.

Diffusion Approximation for TD Learning with Linear Features under Markovian Noise

Introduction

This work offers a rigorous stochastic differential equation (SDE) framework for the analysis of temporal difference (TD) learning with linear function approximation under Markovian noise. The classical ODE-based analysis of TD learning characterizes only the asymptotic mean behavior and is incapable of describing stationary error floors and the influence of temporal correlations. By constructing a weak SDE approximation, this paper unveils both the contraction dynamics dictated by the projected Bellman operator and the stochastic fluctuations determined by Markovian sampling. The resultant framework not only yields a continuous-time proxy for TD iterates but captures the main sources of error and articulates the intricate interplay between algorithmic noise and the geometry of TD learning.

Problem Setting and Methodology

The analysis focuses on policy evaluation for finite-state irreducible, aperiodic Markov chains with fixed policies, bounded linear features, and rewards. The conventional continuous-time limit is the ODE:

$\dot{\theta} = b - A \theta,$

which is only descriptive of the population drift and masks stochastic fluctuations. Practically, with constant stepsizes, the mean squared error plateaus at an $O(\alpha)$ floor due to persistent stochasticity even with exact features. Existing non-asymptotic discrete-time analyses provide finite-time error bounds that capture this phenomenon, but their form is typically an uninformative sum of a decaying term plus a variance floor.

This paper's contribution is to elevate the analysis to a non-asymptotic, weak SDE approximation of TD(0) with constant stepsize $\alpha$:

$$d\Theta_t = (b - A\Theta_t)dt + \sqrt{\alpha} B(\Theta_t) dW_t,$$

where $B(\theta)B(\theta)^\top = \Gamma(\theta)$ for a covariance $\Gamma(\theta)$ that captures the long-run, temporally correlated TD noise. This approximation is established on the basis of one-step moment matching and the machinery of stochastic modified equations.

A significant obstacle is the non-i.i.d. nature of Markovian noise. To surmount this, the authors employ Poisson equation techniques to decompose the TD update noise into a sum of a martingale increment and a coboundary term, identifying the relevant long-run covariance as an explicit function of the Markov chain's mixing properties and the TD geometry.

Characterization of Markovian Noise and Effective Covariance

Let $g_\theta(z)$ be the centered TD increment at parameter $\theta$ and Markov chain observation $z$. Instead of a direct i.i.d.-based variance, the effective covariance for the SDE is:

$$\Gamma(\theta) = \mathbb{E}_\nu\left[ \left(u_\theta(Z_1) - P_Z u_\theta(Z_0)\right) \left(u_\theta(Z_1) - P_Z u_\theta(Z_0)\right)^\top \right],$$

where $O(\alpha)$0 solves the Poisson equation $O(\alpha)$1, and $O(\alpha)$2 is the kernel of the observation Markov chain.

Equivalently, the covariance admits a long-run representation:

$O(\alpha)$3

with $O(\alpha)$4 being the lag-$O(\alpha)$5 autocovariance under stationarity. Thus, $O(\alpha)$6 aggregates all temporal correlations, rather than just the instantaneous variance.

Markov chain mixing properties directly control the size of the effective covariance. The paper provides explicit bounds showing that $O(\alpha)$7 for any positive semidefinite $O(\alpha)$8 is up to a constant factor controlled by the mixing time and the parameter norm. This highlights how slow mixing (i.e., high autocorrelation) amplifies stochastic fluctuations in TD iterates.

Figure 1 Reference

The empirical mean parameter trajectory for the TD and corresponding SDE iterates, as demonstrated in simulation, matches the theoretical assertion that the SDE closely tracks the ensemble mean of the discrete TD algorithm.

Figure 1: Empirical mean parameter trajectory for both TD and diffusion approximation, confirming the fidelity of the SDE as a law-approximating model.

Construction of the SDE and Error Analysis

An affine, globally Lipschitz factor $O(\alpha)$9 is explicitly constructed such that $\alpha$0, with a dimension bounded by the size of state-space features and the sparsity of the transition structure. This regularization is mandatory, given that the principal square root of a possibly degenerate noise covariance would generally lack Lipschitz continuity.

The main technical theorem formally establishes that, for sufficiently regular test functions, the law of the discrete TD iterates and the law of the SDE iterates at matched time scales differ by at most $\alpha$1 over any finite horizon. The SDE thus serves as a robust weak approximation of TD learning under constant stepsize.

In terms of stability, the Lyapunov analysis of the SDE recovers a quantitative expression for the stationary error floor and the rate of contractivity. Explicitly, for the mean squared error,

$\alpha$2

where $\alpha$3 is a function of the chain's mixing time and $\alpha$4 reflects the effective dimension of the noise.

Local Gaussian Structure and Covariance Dynamics

A refined analysis produces a local central limit result: in the neighborhood of the fixed point $\alpha$5, the properly rescaled error process converges in distribution to the solution of a linear Ornstein-Uhlenbeck SDE. The covariance evolves by

$\alpha$6

and the stationary covariance solves the associated Lyapunov equation. This provides complete characterization of the noise geometry: the error variance in any direction is weighted by how both the Markovian noise and the drift $\alpha$7 interact.

Directional variances, error anisotropy, and contraction-weighted effective noise dimensions are thus precisely quantifiable. These results provide diagnostics for the effects of feature map design, policy choice, and stepsize setting.

Implications and Future Directions

The SDE modeling framework developed transcends the mean-field view of ODE methods, allowing both qualitative and quantitative insight into TD's stochastic behavior under general Markovian noise. Practically, it offers rigorous criteria for stepsize selection, quantifies mixing-dependent variance amplification, and exposes how algorithmic design decisions impact error floors. The explicit construction of affine diffusion factors and the recognition of pathwise non-explosion address practical implementation concerns in the analysis of TD and similar stochastic approximation algorithms.

On the theoretical front, this approach bridges finite-time analysis and diffusion limits, delivers continuous-time tools for stability and variance diagnostics, and aligns with modern stochastic approximation analysis—including central limit theory for discrete Markovian recursions.

Future avenues include the extension to unbounded rewards and features, the adaptation to more general stochastic approximation algorithms, and the use of these SDE techniques for the design and principled tuning of RL algorithms in the presence of temporally correlated data.

Conclusion

This paper advances the theory of TD learning with linear features by replacing the traditional ODE perspective with a diffusion approximation that captures both mean and stochastic effects. By systematically deriving the SDE proxy and quantifying how Markovian noise and mixing influence stationary variance, the framework provides new analytical and practical tools for the analysis and diagnosis of TD and related stochastic approximation methods.