---
title: Constant-Stepsize Q-Learning
url: https://www.emergentmind.com/topics/constant-stepsize-q-learning
type: topic
---

# Constant-Stepsize Q-Learning

{} to=arxiv_search  玩北京赛车 鲁夜夜啪കം  天天种彩票json
{"query":"Constant Stepsize Q-learning Distributional Convergence Bias and Extrapolation 2401.13884", "max_results": 5}
{} to=arxiv_search 
Constant-stepsize Q-learning is the variant of Q-learning in which the learning rate is fixed, $\alpha_k \equiv \alpha > 0$, rather than diminished over time. In discounted finite Markov decision processes, it is widely used because it delivers fast transient progress, but its asymptotic behavior differs fundamentally from diminishing-stepsize schemes: instead of almost sure point convergence to $Q^*$, the iterates typically converge in law to a stationary distribution concentrated around the optimal action-value function. Recent work has analyzed this regime through stochastic approximation, Markov-chain, cone-contractive, and switching-system formalisms, yielding results on exponential convergence in Wasserstein distance, central limit theorems, steady-state bias, Richardson–Romberg extrapolation, and finite-time error envelopes [2401.13884], [2404.06023], [2102.08583].

## 1. Definition and canonical update

In a finite discounted MDP $(S,A,T,r,\gamma)$ with bounded rewards and discount factor $\gamma \in (0,1)$, asynchronous constant-stepsize Q-learning updates only the visited state-action pair at time $t$:
\[
Q_{t+1}(s_t,a_t) = (1-\alpha)\, Q_t(s_t,a_t) + \alpha\, \big(r_t + \gamma \max_{a'} Q_t(s_{t+1}, a')\big),
\]
while $Q_t(s,a)$ is unchanged for $(s,a)\neq (s_t,a_t)$ [2401.13884]. In vector form, with $q_k \in \mathbb{R}^{|S||A|}$ and transition tuple $x_k=(s_k,a_k,s_{k+1})$, the recursion is written as
\[
q_{k+1} = q_k + \alpha_k F(x_k,q_k),
\]
with $\alpha_k \equiv \alpha$ and $F(x,q)$ zeroing out all entries except the visited coordinate, where it inserts the temporal-difference target minus the current value [2401.13884].

The constant-stepsize regime has been studied under several sampling models. In the Markovian off-policy setting, the behavior policy induces a time-homogeneous Markov chain on $X := S\times A\times S$, assumed irreducible and aperiodic with stationary distribution $\mu_X$; the associated mixing time is
\[
t_\delta := \min \left\{ k \ge 0: \max_{x\in X} \| p^k(x,\cdot)-\mu_X(\cdot)\|_{TV}\le \delta \right\}
\]
[2401.13884]. Other analyses consider i.i.d. sampling of $(s,a)$ with full coverage $d(s,a)>0$ for all state-action pairs, particularly in switching-system and direct Lyapunov treatments [2102.08583], [2604.19569].

The expected update operator is central. Under stationarity,
\[
\bar F(q) := \mathbb{E}_{x\sim \mu_X}[F(x,q)],
\]
and $\bar F(q)+q$ is a contraction in $\|\cdot\|_\infty$ [2401.13884]. The Bellman optimality operator is
\[
(T^*Q)(s,a)=\mathbb{E}\big[r(s,a)+\gamma \max_{a'} Q(s',a')\mid s,a\big], \qquad Q^*=T^*Q^*,
\]
so constant-stepsize Q-learning can be viewed as a noisy contractive recursion around the fixed point $Q^*$ [2401.13884].

## 2. Stochastic-approximation and Markov-chain formulations

A defining feature of the constant-stepsize setting is that the iterate process alone need not be Markov under Markovian data, but the joint process does become a time-homogeneous Markov chain. Specifically, while $\{q_k\}$ is not Markov because updates depend on the data chain $\{x_k\}$, the augmented chain $\{(x_k,q_k)\}$ is time-homogeneous on $X\times \mathbb{R}^{|S||A|}$ when $\alpha$ is constant [2401.13884]. This observation makes distributional convergence and stationary-law analysis possible.

The main small-stepsize regime used to guarantee ergodicity is
\[
\alpha t_\alpha \le c_0 \frac{(1-\beta)^2}{\log(|S||A|)},
\]
for some constant $c_0$, where $\beta<1$ is the contraction coefficient of the expected operator [2401.13884]. Under the finite-state irreducible and aperiodic assumption, this condition holds for sufficiently small $\alpha$ because the mixing time grows only logarithmically.

This Markov-chain viewpoint is compatible with a broader stochastic-approximation perspective. General nonlinear SA results under Markovian noise show that when $\alpha_k\equiv \alpha$, one obtains exponential convergence to a neighborhood whose radius scales as $O(\alpha \log(1/\alpha))$ in the mean-square sense, with the logarithmic factor arising from mixing-time control [1905.11425]. In the Q-learning literature, this general paradigm underlies both tabular constant-stepsize analyses and linear-function-approximation results under explicit stability conditions on the behavior policy [1905.11425].

The same recursion has also been represented as a stochastic switching system. In one formulation, asynchronous constant-stepsize Q-learning can be written as a stochastic affine switching system
\[
Q_{t+1}-Q^* = A_{Q_t}(Q_t-Q^*) + b_{Q_t} + \alpha w_t,
\]
where the switching signal is the greedy policy induced by $Q_t$ [2102.08583]. A later direct switching representation removes the affine comparison term by introducing a stochastic policy $\mu_{Q_t}$ such that
\[
V_{Q_t}-V^* = \Pi^{\mu_{Q_t}}(Q_t-Q^*),
\]
which yields the switched linear conditional-mean recursion
\[
e_{t+1}=M_{\mu_t}e_t+\alpha w_t
\]
for the error $e_t=Q_t-Q^*$ [2604.19569]. These reformulations expose instance-dependent drift rates that are sharper than row-sum arguments.

## 3. Distributional convergence and stationary behavior

The central asymptotic result for tabular asynchronous constant-stepsize Q-learning is distributional convergence of the iterates to a unique stationary law. Under the Markovian-data assumptions and the small-stepsize condition above, the joint chain $\{(x_k,q_k)\}$ converges in an extended Wasserstein-2 metric to a unique stationary distribution $\bar\mu$, and the marginal law of $q_k$ converges exponentially fast in $W_2$ to the law $\mu$ of a stationary random vector $q_\infty$ [2401.13884]. The rate is
\[
W_2^2\big(\mathsf{Law}(q_k),\mu\big)
\le
24\,\eta^{k-t_\alpha}
\left(
\mathbb{E}\|q_0\|_\infty^2+\mathbb{E}\|q_\infty\|_\infty^2
\right),
\qquad
\eta=1-\frac{(1-\beta)\alpha}{2},
\]
for all $k\ge t_\alpha$ [2401.13884].

The stationary fluctuations are nonvanishing. The same result bounds the stationary variance as
\[
\mathrm{Var}(q_\infty)\le
c_Q \frac{\log(|S||A|)}{(1-\beta)^2}\,\alpha t_\alpha,
\]
which formalizes the standard constant-stepsize trade-off: faster forgetting of initialization at larger $\alpha$, but a wider stationary neighborhood [2401.13884]. Corresponding moment convergence results show exponential decay of the first two moments of $q_k$ toward those of $q_\infty$ [2401.13884].

For nonsmooth contractive SA motivated by Q-learning, a related Wasserstein result establishes geometric ergodicity of the stationary law under i.i.d. sampling. For synchronous and asynchronous Q-learning, there exist constants $c>0$ and $\bar\alpha_0'>0$ such that, for $\alpha\le \bar\alpha_0'$,
\[
W_2^2\big(\mathrm{Law}(q_t),\mathrm{Law}(q^{(\alpha)})\big)
\le
c\,(1-\alpha(1-\sqrt{\gamma_0}))^t,
\]
and the stationary mean-square error satisfies
\[
\mathbb{E}\|q^{(\alpha)}-q^*\|_2^2 \in O(\alpha),
\]
where $\gamma_0=1-(1-\gamma)\min_i D_{ii}$ encodes asynchronous coverage [2404.06023]. This result emphasizes that geometric convergence to a stationary distribution is not restricted to the smooth setting.

A further refinement is the Gaussian approximation of the centered-and-scaled steady state. For contractive nonlinear SA, later work gives explicit non-asymptotic Wasserstein bounds of order $\sqrt{\alpha}\log(1/\alpha)$ between the stationary law of $(Q_\infty-Q_*)/\sqrt{\alpha}$ and a Gaussian law, and specializes this framework to tabular Q-learning under differentiability conditions near $Q_*$ [2602.13960]. The covariance matrix is characterized by a Lyapunov equation involving the linearized drift and the long-run covariance of the Markovian noise [2602.13960].

## 4. Averaging, central limit theory, and bias

A major advance in the constant-stepsize theory is the separation between variance reduction by averaging and bias reduction by extrapolation. For the averaged iterate
\[
\bar Q_n = \frac{1}{n}\sum_{t=1}^n Q_t,
\]
the centered partial sums
\[
S_n=\sum_{k=0}^{n-1}\big(q_k-\mathbb{E}[q_\infty]\big)
\]
satisfy a central limit theorem and a functional CLT:
\[
n^{-1/2}S_n \Rightarrow \mathcal N(0,\Sigma),
\]
and the interpolated process converges to Brownian motion with covariance $\Sigma$ [2401.13884]. This yields asymptotic normality of long time averages centered at the stationary mean $\mathbb{E}[q_\infty]$.

The same paper gives a non-asymptotic analysis of Polyak–Ruppert tail averaging. For burn-in $k_0$,
\[
\bar q_{k_0,k} := \frac{1}{k-k_0}\sum_{t=k_0}^{k-1} q_t,
\]
and under the small-stepsize regime with a unique optimal policy separated by a positive action-gap, the mean error obeys
\[
\mathbb{E}[\bar q_{k_0,k}] - q^*
=
\alpha B + \mathcal O(\alpha^2+\alpha^2 t_{\alpha^2}^2)
+
\mathcal O\!\left(
\frac{1}{\alpha(k-k_0)}
\exp\!\left(-\frac{\alpha(1-\beta)k_0}{4}\right)
\right),
\]
while the variance scales as $O(1/((k-k_0)\alpha))$ and the asymptotic squared bias is of order $\alpha^2$ [2401.13884]. The linear coefficient $B$ is explicit and depends on $(r,\gamma,P)$.

This $O(\alpha)$ bias expansion enables Richardson–Romberg extrapolation. With stepsizes $\alpha$ and $2\alpha$, the extrapolated estimator
\[
\widetilde q^{(\alpha)}_{k_0,k}
=
2\bar q^{(\alpha)}_{k_0,k} - \bar q^{(2\alpha)}_{k_0,k}
\]
satisfies
\[
\mathbb{E}[\widetilde q^{(\alpha)}_{k_0,k}] - q^*
=
\mathcal O(\alpha^2+\alpha^2 t_{\alpha^2}^2)
+
\mathcal O\!\left(
\frac{1}{\alpha(k-k_0)}
\exp\!\left(-\frac{\alpha(1-\beta)k_0}{4}\right)
\right),
\]
so the leading mean bias drops from $O(\alpha)$ to $O(\alpha^2)$ and the squared bias from $O(\alpha^2)$ to $O(\alpha^4)$, while the variance and optimization-error terms remain of the same order [2401.13884].

A key qualification is that the bias order depends on local smoothness of the Bellman max. In nonsmooth contractive SA, and specifically in Q-learning, the small-stepsize stationary bias can instead scale as $\Theta(\sqrt{\alpha})$. The decisive distinction is whether the MDP has reachable tied optimal actions. When there exists a tied state that is not rooted (Type A), one has
\[
\mathbb{E}[q^{(\alpha)}]-q^*=\Theta(\sqrt{\alpha});
\]
when there are no such effective ties (Type B), the leading $\sqrt{\alpha}$ term vanishes, and under sufficient moments
\[
\mathbb{E}[q^{(\alpha)}] = q^* + O(\alpha^n)
\]
for any $n\ge 2$ covered by the moment assumptions [2404.06023]. This resolves an apparent contradiction in the literature: the $O(\alpha)$ expansion is a local-linearization result under a positive action-gap, whereas the $\Theta(\sqrt{\alpha})$ bias is a genuinely nonsmooth phenomenon caused by the kink of the max operator at reachable ties.

## 5. Finite-time bounds and control-theoretic viewpoints

Parallel to the stationary-law literature, several works analyze constant-stepsize Q-learning through finite-time envelopes. For synchronous Q-learning, cone-contractive analysis yields a deterministic sandwich relation on the error and, under constant $\alpha$, leads to an expected $\ell_\infty$ bound of the form
\[
\mathbb{E}\|Q_t-Q^*\|_\infty
\le
\beta^{t-1}\|Q_0-Q^*\|_\infty
+
C\!\left(
\sqrt{\alpha}\,\|\sigma(Q^*)\|_\infty \sqrt{\log(2D)}
+
\alpha \gamma\,\mathrm{span}(Q^*)\log(2D)
\right),
\]
with $\beta=1-(1-\gamma)\alpha$ [1905.06265]. The transient decays geometrically, while the steady-state floor is variance-dominated at order $\sqrt{\alpha}$ in $\ell_\infty$.

For asynchronous Q-learning under i.i.d. visitation, a switching-system analysis constructs lower and upper comparison processes and proves a finite-time bound for the running average
\[
\tilde Q_N := \frac{1}{N}\sum_{t=0}^{N-1} Q_t
\]
of the form
\[
\mathbb{E}\|\tilde Q_N-Q^*\|_\infty
\le
C(|S|,|A|,\gamma,d_{\min},d_{\max})
\sqrt{32\alpha+\frac{4}{N\alpha}},
\]
thereby making the classical constant-stepsize bias-variance trade-off explicit in the averaged-iterate error [2102.08583]. The same framework explains overestimation through the nonnegative contribution induced by the Bellman maximum [2102.08583].

A sharper direct-switching theory represents the error recursion as
\[
e_{t+1}=M_{\mu_t}e_t+\alpha w_t,
\qquad
M_\mu = I-\alpha D + \alpha\gamma D P \Pi^\mu,
\]
and identifies the intrinsic drift rate with the joint spectral radius
\[
\rho_\alpha^{\mathrm{dir}} := \rho(\{M_\pi:\pi\in\Theta\}),
\]
which satisfies
\[
\rho_\alpha^{\mathrm{dir}} \le \rho_{\mathrm{row}} := 1-\alpha d_{\min}(1-\gamma)
\]
and can be strictly smaller [2604.19569]. This yields finite-time last-iterate bounds such as
\[
\mathbb{E}\|Q_k-Q^*\|_\infty
\le
\sqrt{nC_\varepsilon}\Big(B_0+\frac{R_{\max}}{1-\gamma}\Big)\beta_\varepsilon^k
+
\alpha C_\varepsilon \sqrt{\frac{W_{\max}}{1-\beta_\varepsilon^2}},
\]
as well as a computable quadratic-certificate version based on LMIs and common quadratic Lyapunov functions [2604.19569].

A sign-separated refinement decomposes the error into positive and negative parts, $e_k=e_k^+-e_k^-$. The negative part is dominated by a lower comparison LTI system associated with a fixed optimal policy, while the positive part is controlled by a switching family over all deterministic policies. The resulting certificates satisfy
\[
\rho_-^\star \le \rho_- \le \rho_+,
\]
so the negative-side envelope is no slower and may be faster than the positive-side envelope [2605.16103]. This formalizes a max-induced asymmetry: positive errors can be selected and propagated by the Bellman maximum, whereas negative errors admit an optimal-policy lower comparison, directly connecting constant-stepsize dynamics to overestimation [2605.16103].

## 6. Extensions, applications, and scope

The tabular theory has motivated several extensions. In restless-bandit index learning, constant-stepsize Q-learning appears as the fast timescale of a two-timescale stochastic approximation scheme, with the slow timescale updating a subsidy or Whittle index. Under Lipschitz, contraction, bounded-noise, and coverage assumptions, the steady-state neighborhood scales as
\[
O(\alpha) + O(\gamma/\alpha),
\]
where $\alpha$ is the Q-learning stepsize and $\gamma$ is the slower index-update stepsize [2409.04605]. This makes constant-stepsize Q-learning a component in broader adaptive-control procedures rather than merely a standalone RL algorithm.

With linear function approximation, finite-sample guarantees exist under explicit stability assumptions on the behavior policy. For off-policy linear Q-learning with geometrically mixing Markovian samples, constant stepsize yields
\[
\mathbb{E}\|\theta_k-\theta^*\|^2
\le
\eta_1 \Big(1-\frac{\kappa}{2}\alpha\Big)^{k-t_\alpha}
+
\frac{2\eta_2}{\kappa}\,\alpha t_\alpha,
\]
so the asymptotic mean-square neighborhood is $O(\alpha\log(1/\alpha))$ [1905.11425]. The same paper emphasizes that no projection step is needed and that the samples need not be i.i.d. [1905.11425].

Other work addresses the instability of constant-step Q-learning with function approximation by modifying the algorithm itself. A stabilized linear-architecture variant uses a second-order streaming update, target networks, and a replay-like policy-replay mechanism; within each epoch, the targets are fixed, the data distribution is stationary, and the update is exactly equivalent to regularized least squares [2206.00796]. An implicit variant reformulates the update as a fixed-point equation, producing the adaptive normalization
\[
\alpha_{\mathrm{eff}}=\frac{\alpha}{1+\alpha\|\phi_t\|^2},
\]
which substantially enlarges the range of stable constant stepsizes in both Q-learning and SARSA under the paper’s assumptions [2601.18907].

Empirically, the tabular theory of constant-stepsize asynchronous Q-learning has been validated on a $1\times 3$ Gridworld and a $4\times 4$ slippery Frozen-Lake setting. In those experiments, larger $\alpha$ led to faster convergence, the bias of tail averaging saturated at a level approximately proportional to $\alpha$, Richardson–Romberg extrapolation reduced the final bias relative to tail averaging at the same $\alpha$, and constant-stepsize tail averaging and extrapolation substantially outperformed diminishing-stepsize baselines in transient speed, especially in the harder $4\times 4$ Gridworld [2401.13884]. Additional experiments with linear function approximation showed qualitatively similar behavior, although that paper’s formal theory remained tabular [2401.13884].

The current theory remains sharply delimited. The strongest results concern finite MDPs with tabular representation, fixed stationary behavior policies, bounded rewards, small enough constant stepsizes, and—when explicit $O(\alpha)$ bias expansions are desired—a positive action-gap ensuring local linearization of the max operator [2401.13884]. General nonlinear function approximation, non-stationary behavior policies, adaptive stepsizes, and deep Q-learning fall outside the scope of these proofs [2401.13884]. A plausible implication is that constant-stepsize Q-learning should be viewed less as a single asymptotic object than as a family of regimes whose stationary bias, fluctuation law, and finite-time envelope depend sensitively on sampling, smoothness at the optimum, and the presence or absence of optimal-action ties.

Source: https://www.emergentmind.com/topics/constant-stepsize-q-learning