Constant-Stepsize Stochastic Approximation

• Constant-Stepsize SA is a recursive algorithm that uses a fixed gain to update parameters, producing a stationary fluctuation regime around the target under noisy observations.
• It leverages detailed analysis of finite-time behavior via Lyapunov drift and Markovian mixing conditions to control bias and preserve exponential convergence properties.
• Applications span least-mean-squares, temporal-difference learning, Q-learning, and reinforcement learning, where methods such as averaging and extrapolation manage inherent biases.

Constant-stepsize stochastic approximation (SA) is the class of recursive procedures that update a parameter with a fixed gain, typically in the form

$\theta_{k+1}=\theta_k+\alpha\,H_k,$

where $\alpha>0$ does not vanish with $k$. In root-finding form, the objective is to solve $F(\theta^\ast)=0$ or $\bar F(\theta^\ast)=0$ from noisy observations, and the constant-gain regime replaces asymptotic point convergence by a stationary or near-stationary fluctuation regime around the target. Modern analyses treat linear and nonlinear SA, smooth and nonsmooth operators, i.i.d., Markovian, and decision-dependent noise, and applications ranging from least-mean-squares and temporal-difference learning to Q-learning, tracking, two-timescale recursion, and approximate posterior sampling (Chen et al., 2019, Chen et al., 2021, Zhang et al., 2024, Hadavi et al., 15 Apr 2026).

1. Core formulation and mean-field structure

A canonical SA recursion studied in recent work is

$$\theta_{k+1}=\theta_k+\alpha_k\bigl(F(X_k,\theta_k)+w_k\bigr),$$

with stepsizes $\{\alpha_k\}$, a noise process $\{X_k\}$, and a martingale-difference term $\{w_k\}$. The associated mean field is

$\bar F(\theta)=\mathbb{E}_{\mu_X}[F(X,\theta)],$

and the corresponding ODE is

$\alpha>0$0

In the constant-stepsize regime, $\alpha>0$1, the central object is no longer almost-sure convergence of $\alpha>0$2, but the finite-time and stationary behavior of $\alpha>0$3 or of the invariant law of the Markov chain induced by the recursion (Chen et al., 2019).

The standard structural assumptions in the modern literature are global Lipschitz continuity or affine growth of $\alpha>0$4, together with a stability condition on the mean dynamics. In nonlinear Markovian SA this appears as global strong monotonicity or dissipativity,

$\alpha>0$5

which yields exponential stability of the ODE. In linear SA the analogous condition is that the mean matrix is Hurwitz. In contractive SA, one writes $\alpha>0$6, with $\alpha>0$7 a contraction in a suitable norm, so that the Jacobian linearization $\alpha>0$8 is Hurwitz at the fixed point (Chen et al., 2019, Chen et al., 2021).

Markovian noise enters through a chain $\alpha>0$9 with stationary law $k$0. A recurring assumption is uniform geometric ergodicity, which implies a mixing-time bound

$k$1

This is the mechanism by which dependence is controlled: over windows of length $k$2, the empirical sampling operator becomes close to its stationary counterpart, and the SA recursion behaves like a discretization of the mean ODE plus perturbations whose magnitude depends on $k$3 (Chen et al., 2019).

These formulations place constant-stepsize SA at the intersection of ODE stability, Markov-chain mixing, and invariant-measure analysis. A plausible implication is that the fixed-gain regime should be understood less as asymptotic root-finding and more as controlled stochastic perturbation of a stable flow.

2. Finite-time behavior under fixed gain

For nonlinear SA under Markovian noise, a sharp finite-sample result is available under the conditions above. If $k$4 and

$k$5

then for all $k$6,

$k$7

Since $k$8, the bias term decays geometrically while the limiting neighborhood has radius $k$9. This is the precise sense in which constant-step SA achieves exponential convergence to a nonzero neighborhood under Markovian noise (Chen et al., 2019).

The proof strategy in that setting combines Lyapunov drift with explicit control of the Markovian bias. Writing $F(\theta^\ast)=0$0, the one-step increment is decomposed into deterministic drift, martingale noise, a Markovian sampling-bias term, and an $F(\theta^\ast)=0$1 discretization term. The technical heart is to show that the Markovian bias is of order $F(\theta^\ast)=0$2, so that it does not overwhelm the negative drift once $F(\theta^\ast)=0$3 is small (Chen et al., 2019).

A distinct finite-time line of work studies biased SA under very general stochastic perturbations by replacing the usual one-step Lyapunov function with a multistep Lyapunov functional that sums over future iterates. In that framework, the mean-square error for constant stepsize admits a bound with four components: geometric decay of initialization error, an $F(\theta^\ast)=0$4 term, an $F(\theta^\ast)=0$5 steady-state term, and a pre-mixing transient term controlled by the bias-decay function of the underlying ergodic process. The construction was motivated by temporal-difference and Q-learning algorithms observed along Markov trajectories (Wang et al., 2019).

Constant-gain SA also appears in nonstationary tracking. For the recursion

$F(\theta^\ast)=0$6

where $F(\theta^\ast)=0$7 samples a slowly moving target, the tracking error satisfies an all-time root-mean-square bound of the form

$F(\theta^\ast)=0$8

Here the error decomposes into contributions from bounded additive error, target motion, noise and discretization, and initialization (Kumar et al., 2018).

Taken together, these results fix a characteristic constant-stepsize pattern: fast transient contraction at rate $F(\theta^\ast)=0$9, followed by a nonvanishing error floor whose scale is dictated by mixing, bias, or tracking.

3. Stationary laws, Gaussian limits, and departures from Gaussianity

In the smooth and contractive settings, constant-stepsize SA is naturally analyzed through its stationary distribution. For the recursion

$\bar F(\theta^\ast)=0$0

with i.i.d. mean-zero noise of covariance $\bar F(\theta^\ast)=0$1, one studies the centered and scaled stationary variable

$\bar F(\theta^\ast)=0$2

For three benchmark cases—SGD with a smooth strongly convex objective, linear SA with Hurwitz matrix $\bar F(\theta^\ast)=0$3, and nonlinear SA with a contractive operator—the correct scaling is $\bar F(\theta^\ast)=0$4, and every weak limit of $\bar F(\theta^\ast)=0$5 solves an integral equation. Under a uniqueness assumption, the limit is Gaussian and its covariance is the unique solution of a Lyapunov equation, such as

$\bar F(\theta^\ast)=0$6

in the linear case (Chen et al., 2021).

This stationary viewpoint differs from classical diminishing-stepsize central limit theory. The order of limits is

$\bar F(\theta^\ast)=0$7

so the Gaussian law describes steady-state fluctuations of a fixed-gain algorithm as the gain tends to zero, rather than transient fluctuations of a vanishing-gain scheme (Chen et al., 2021).

Finite-time normal approximation has also been quantified. For constant stepsize, the rescaled error $\bar F(\theta^\ast)=0$8 satisfies a Wasserstein bound

$\bar F(\theta^\ast)=0$9

where $$\theta_{k+1}=\theta_k+\alpha_k\bigl(F(X_k,\theta_k)+w_k\bigr),$$0 solves the limiting Lyapunov equation and $$\theta_{k+1}=\theta_k+\alpha_k\bigl(F(X_k,\theta_k)+w_k\bigr),$$1. The last term is the transient contribution, while the first two quantify steady-state non-Gaussianity and discretization error. This gives an explicit pre-asymptotic rate to Gaussianity in the constant-gain regime (Haque et al., 14 Feb 2026).

The Gaussian picture is not universal. In the quartic example

$$\theta_{k+1}=\theta_k+\alpha_k\bigl(F(X_k,\theta_k)+w_k\bigr),$$2

the correct scaling is $$\theta_{k+1}=\theta_k+\alpha_k\bigl(F(X_k,\theta_k)+w_k\bigr),$$3, not $$\theta_{k+1}=\theta_k+\alpha_k\bigl(F(X_k,\theta_k)+w_k\bigr),$$4, and the limiting law is non-Gaussian. More broadly, nonsmooth contractive SA admits stationary weak limits in Wasserstein distance, but its asymptotic bias is proportional to $$\theta_{k+1}=\theta_k+\alpha_k\bigl(F(X_k,\theta_k)+w_k\bigr),$$5, in sharp contrast to smooth SA (Chen et al., 2021, Zhang et al., 2024).

These results suggest that the stationary measure of constant-stepsize SA is governed by local geometry near the root: linearization and quadratic curvature yield Gaussian $$\theta_{k+1}=\theta_k+\alpha_k\bigl(F(X_k,\theta_k)+w_k\bigr),$$6-fluctuations, whereas kinks, flat directions, or higher-order local structure can force different scaling and different invariant laws.

4. Bias, averaging, and statistical inference

In i.i.d. linear SA, constant stepsize becomes substantially more effective when combined with Polyak–Ruppert averaging. For the linear recursion

$$\theta_{k+1}=\theta_k+\alpha_k\bigl(F(X_k,\theta_k)+w_k\bigr),$$7

there exists an admissible interval of constant stepsizes such that

$$\theta_{k+1}=\theta_k+\alpha_k\bigl(F(X_k,\theta_k)+w_k\bigr),$$8

with a bias term of order $$\theta_{k+1}=\theta_k+\alpha_k\bigl(F(X_k,\theta_k)+w_k\bigr),$$9 and a variance term of order $\{\alpha_k\}$0. The admissible range is characterized by positivity of a stochastic stability functional $\{\alpha_k\}$1. The same work shows that no universal constant stepsize exists for all bounded positive-definite data distributions, whereas PSD-structured classes admit a uniform stabilizing range (Lakshminarayanan et al., 2017).

In least-squares regression, averaged constant-step-size least-mean-squares admits a sharper bias–variance decomposition: $\{\alpha_k\}$2 The leading variance term is asymptotically independent of $\{\alpha_k\}$3, while the bias decays faster in $\{\alpha_k\}$4 but is highly sensitive to the chosen constant stepsize. This explains why large stable gains are beneficial in the bias-dominated regime but mostly irrelevant once the variance term dominates (Défossez et al., 2014).

Under Markovian disturbance, however, averaging does not in general remove bias. The “curse of memory” analysis shows that for constant-step Markovian SA the effective target is shifted: the target bias is $\{\alpha_k\}$5 in nonlinear SA, and in linear SA

$\{\alpha_k\}$6

For averaged iterates, the asymptotic covariance remains within $\{\alpha_k\}$7 of the optimal covariance, but the $\{\alpha_k\}$8 correction may be large when the mean dynamics are ill-conditioned or the Markov chain has strong memory (Lauand et al., 2023).

A related step-size analysis for $\{\alpha_k\}$9 clarifies the boundary between diminishing and nearly constant regimes. In linear SA, averaging yields optimal asymptotic covariance for every $\{X_k\}$0, but the MSE can still be dominated by bias: $\{X_k\}$1 The paper’s major conclusion is that $\{X_k\}$2 or even $\{X_k\}$3 is justified only in select settings, because in general the bias may preclude fast convergence (Lauand et al., 2024).

For statistical inference, averaged constant-stepsize Markovian LSA satisfies a CLT centered at the stationary mean,

$\{X_k\}$4

and batch-means covariance estimation benefits from the geometric ergodicity of the constant-step chain. Richardson–Romberg extrapolation is then used to cancel the leading $\{X_k\}$5 bias. The same study identifies important zero-bias settings, including independent multiplicative noise, linear regression with independent additive noise, and realizable linear TD learning (Huo et al., 2023).

5. Reinforcement learning and adjacent application domains

Reinforcement learning has been a primary driver of constant-stepsize SA theory. In Q-learning with linear function approximation, defining

$\{X_k\}$6

the recursion

$\{X_k\}$7

is exactly Markovian SA with $\{X_k\}$8. Under a behavior-policy condition ensuring global stability of the projected Bellman equation, the constant-step bound specializes to

$\{X_k\}$9

with $\{w_k\}$0. The analysis requires neither i.i.d. samples nor an artificial projection step (Chen et al., 2019).

Biased SA analysis has also been pushed directly at TD and Q-learning under practical Markov chain observation models. The multistep Lyapunov framework yields non-asymptotic mean-square bounds for unmodified TD- and Q-learning updates under general mixing conditions and arbitrary initial distribution, a regime where at least one simplifying assumption used in prior work must be violated (Wang et al., 2019).

For nonsmooth RL dynamics, synchronous and asynchronous Q-learning fit into contractive SA with additive and multiplicative noise. In that framework, the iterates converge in Wasserstein distance to a stationary limit law. The scaled stationary distributions converge as $\{w_k\}$1, and the asymptotic bias is proportional to $\{w_k\}$2. The paper further characterizes when the leading $\{w_k\}$3 term is present: for Q-learning, tied optimal actions in reachable states generate the nonsmooth bias, whereas in the complementary regime the leading term vanishes (Zhang et al., 2024).

Constant stepsizes also matter in multi-timescale RL-style architectures. For linear two-timescale SA with Markovian noise and constant stepsizes $\{w_k\}$4, the joint iterate converges geometrically in Wasserstein distance to a unique stationary distribution. The stationary biases of both timescales are $\{w_k\}$5 up to higher-order terms, while the variance of the slow iterate is $\{w_k\}$6 and that of the fast iterate is $\{w_k\}$7. Tail-averaging and Richardson–Romberg extrapolation then improve the MSE of both iterates to $\{w_k\}$8 (Kwon et al., 2024).

Outside RL proper, constant-step SA has been used as the averaging mechanism inside approximate Thompson sampling. In TS-SA, a fixed SA stepsize and a stationary Langevin interpretation replace round-specific posterior sampling schedules; the resulting algorithm admits near-optimal regret and a unified convergence analysis through a stationary SGLD process (Wang et al., 6 Oct 2025).

6. Extensions, limitations, and current frontiers

Recent work extends constant-stepsize SA to decision-dependent Markovian noise, where the transition kernel itself depends on the current decision. In that setting, the analysis begins from finite-time $\{w_k\}$9-th moment bounds and then uses a local regularity condition termed Poisson–Gateaux differentiability to show that the stationary bias is $\bar F(\theta)=\mathbb{E}_{\mu_X}[F(X,\theta)],$0 for a broad class of models. The same work establishes geometric weak convergence of the joint SA process to a unique stationary distribution and a functional central limit theorem, while covering non-smooth kernels such as acceptance–rejection mechanisms, projected Langevin dynamics, and clipped state dynamics (Hadavi et al., 15 Apr 2026).

Another frontier concerns absorbing-state systems. For SA on the simplex with constant step $\bar F(\theta)=\mathbb{E}_{\mu_X}[F(X,\theta)],$1,

$\bar F(\theta)=\mathbb{E}_{\mu_X}[F(X,\theta)],$2

the boundary is absorbing and the relevant invariant object is a quasi-stationary distribution rather than an ordinary stationary law. Under large-deviation assumptions and the existence of an interior attractor for the ODE $\bar F(\theta)=\mathbb{E}_{\mu_X}[F(X,\theta)],$3, weak-$\bar F(\theta)=\mathbb{E}_{\mu_X}[F(X,\theta)],$4 limit points of the quasi-stationary distributions are invariant measures of the ODE supported on interior quasi-attractors, and the expected absorption time grows at least like $\bar F(\theta)=\mathbb{E}_{\mu_X}[F(X,\theta)],$5 (Marmet, 2013).

The main unresolved structural divide is now clear. In smooth linearized settings, the long-run bias is typically $\bar F(\theta)=\mathbb{E}_{\mu_X}[F(X,\theta)],$6 and the scaled stationary law is asymptotically Gaussian (Chen et al., 2021, Haque et al., 14 Feb 2026). In nonsmooth contractive settings, the leading bias can instead be $\bar F(\theta)=\mathbb{E}_{\mu_X}[F(X,\theta)],$7, and the stationary law may no longer be Gaussian (Zhang et al., 2024). In Markovian settings, memory can generate an $\bar F(\theta)=\mathbb{E}_{\mu_X}[F(X,\theta)],$8 target bias that persists under averaging (Lauand et al., 2023). Taken together, these results suggest that constant-stepsize SA is best interpreted as an ergodic numerical method whose accuracy is determined jointly by local operator geometry, the dependence structure of the noise, and the way bias is managed—by smaller gain, averaging, extrapolation, or problem structure.