Constant-Stepsize Q-Learning

• Constant-stepsize Q-learning is a reinforcement learning method that uses a fixed learning rate, enabling rapid progress in finite MDPs while converging in distribution rather than to a single point.
• It leverages stochastic approximation, Markov-chain analysis, and switching-system formalisms to establish exponential convergence rates and quantify steady-state bias and variance.
• Techniques like averaging and Richardson–Romberg extrapolation are utilized to reduce the bias and control the mean-square error in the stationary regime.

{} to=arxiv_search 玩北京赛车 鲁夜夜啪കം 天天种彩票json {"query":"Constant Stepsize Q-learning Distributional Convergence Bias and Extrapolation (Zhang et al., 2024)", "max_results": 5} {} to=arxiv_search Constant-stepsize Q-learning is the variant of Q-learning in which the learning rate is fixed, $\alpha_k \equiv \alpha > 0$, rather than diminished over time. In discounted finite Markov decision processes, it is widely used because it delivers fast transient progress, but its asymptotic behavior differs fundamentally from diminishing-stepsize schemes: instead of almost sure point convergence to $Q^*$, the iterates typically converge in law to a stationary distribution concentrated around the optimal action-value function. Recent work has analyzed this regime through stochastic approximation, Markov-chain, cone-contractive, and switching-system formalisms, yielding results on exponential convergence in Wasserstein distance, central limit theorems, steady-state bias, Richardson–Romberg extrapolation, and finite-time error envelopes (Zhang et al., 2024, Zhang et al., 2024, Lee et al., 2021).

1. Definition and canonical update

In a finite discounted MDP $(S,A,T,r,\gamma)$ with bounded rewards and discount factor $\gamma \in (0,1)$, asynchronous constant-stepsize Q-learning updates only the visited state-action pair at time $t$: $$Q_{t+1}(s_t,a_t) = (1-\alpha)\, Q_t(s_t,a_t) + \alpha\, \big(r_t + \gamma \max_{a'} Q_t(s_{t+1}, a')\big),$$ while $Q_t(s,a)$ is unchanged for $(s,a)\neq (s_t,a_t)$ (Zhang et al., 2024). In vector form, with $q_k \in \mathbb{R}^{|S||A|}$ and transition tuple $x_k=(s_k,a_k,s_{k+1})$, the recursion is written as

$Q^*$0

with $Q^*$1 and $Q^*$2 zeroing out all entries except the visited coordinate, where it inserts the temporal-difference target minus the current value (Zhang et al., 2024).

The constant-stepsize regime has been studied under several sampling models. In the Markovian off-policy setting, the behavior policy induces a time-homogeneous Markov chain on $Q^*$3, assumed irreducible and aperiodic with stationary distribution $Q^*$4; the associated mixing time is

$Q^*$5

(Zhang et al., 2024). Other analyses consider i.i.d. sampling of $Q^*$6 with full coverage $Q^*$7 for all state-action pairs, particularly in switching-system and direct Lyapunov treatments (Lee et al., 2021, Lee, 21 Apr 2026).

The expected update operator is central. Under stationarity,

$Q^*$8

and $Q^*$9 is a contraction in $(S,A,T,r,\gamma)$0 (Zhang et al., 2024). The Bellman optimality operator is

$(S,A,T,r,\gamma)$1

so constant-stepsize Q-learning can be viewed as a noisy contractive recursion around the fixed point $(S,A,T,r,\gamma)$2 (Zhang et al., 2024).

2. Stochastic-approximation and Markov-chain formulations

A defining feature of the constant-stepsize setting is that the iterate process alone need not be Markov under Markovian data, but the joint process does become a time-homogeneous Markov chain. Specifically, while $(S,A,T,r,\gamma)$3 is not Markov because updates depend on the data chain $(S,A,T,r,\gamma)$4, the augmented chain $(S,A,T,r,\gamma)$5 is time-homogeneous on $(S,A,T,r,\gamma)$6 when $(S,A,T,r,\gamma)$7 is constant (Zhang et al., 2024). This observation makes distributional convergence and stationary-law analysis possible.

The main small-stepsize regime used to guarantee ergodicity is

$(S,A,T,r,\gamma)$8

for some constant $(S,A,T,r,\gamma)$9, where $\gamma \in (0,1)$0 is the contraction coefficient of the expected operator (Zhang et al., 2024). Under the finite-state irreducible and aperiodic assumption, this condition holds for sufficiently small $\gamma \in (0,1)$1 because the mixing time grows only logarithmically.

This Markov-chain viewpoint is compatible with a broader stochastic-approximation perspective. General nonlinear SA results under Markovian noise show that when $\gamma \in (0,1)$2, one obtains exponential convergence to a neighborhood whose radius scales as $\gamma \in (0,1)$3 in the mean-square sense, with the logarithmic factor arising from mixing-time control (Chen et al., 2019). In the Q-learning literature, this general paradigm underlies both tabular constant-stepsize analyses and linear-function-approximation results under explicit stability conditions on the behavior policy (Chen et al., 2019).

The same recursion has also been represented as a stochastic switching system. In one formulation, asynchronous constant-stepsize Q-learning can be written as a stochastic affine switching system

$\gamma \in (0,1)$4

where the switching signal is the greedy policy induced by $\gamma \in (0,1)$5 (Lee et al., 2021). A later direct switching representation removes the affine comparison term by introducing a stochastic policy $\gamma \in (0,1)$6 such that

$\gamma \in (0,1)$7

which yields the switched linear conditional-mean recursion

$\gamma \in (0,1)$8

for the error $\gamma \in (0,1)$9 (Lee, 21 Apr 2026). These reformulations expose instance-dependent drift rates that are sharper than row-sum arguments.

3. Distributional convergence and stationary behavior

The central asymptotic result for tabular asynchronous constant-stepsize Q-learning is distributional convergence of the iterates to a unique stationary law. Under the Markovian-data assumptions and the small-stepsize condition above, the joint chain $t$0 converges in an extended Wasserstein-2 metric to a unique stationary distribution $t$1, and the marginal law of $t$2 converges exponentially fast in $t$3 to the law $t$4 of a stationary random vector $t$5 (Zhang et al., 2024). The rate is

$t$6

for all $t$7 (Zhang et al., 2024).

The stationary fluctuations are nonvanishing. The same result bounds the stationary variance as

$t$8

which formalizes the standard constant-stepsize trade-off: faster forgetting of initialization at larger $t$9, but a wider stationary neighborhood (Zhang et al., 2024). Corresponding moment convergence results show exponential decay of the first two moments of $$Q_{t+1}(s_t,a_t) = (1-\alpha)\, Q_t(s_t,a_t) + \alpha\, \big(r_t + \gamma \max_{a'} Q_t(s_{t+1}, a')\big),$$0 toward those of $$Q_{t+1}(s_t,a_t) = (1-\alpha)\, Q_t(s_t,a_t) + \alpha\, \big(r_t + \gamma \max_{a'} Q_t(s_{t+1}, a')\big),$$1 (Zhang et al., 2024).

For nonsmooth contractive SA motivated by Q-learning, a related Wasserstein result establishes geometric ergodicity of the stationary law under i.i.d. sampling. For synchronous and asynchronous Q-learning, there exist constants $$Q_{t+1}(s_t,a_t) = (1-\alpha)\, Q_t(s_t,a_t) + \alpha\, \big(r_t + \gamma \max_{a'} Q_t(s_{t+1}, a')\big),$$2 and $$Q_{t+1}(s_t,a_t) = (1-\alpha)\, Q_t(s_t,a_t) + \alpha\, \big(r_t + \gamma \max_{a'} Q_t(s_{t+1}, a')\big),$$3 such that, for $$Q_{t+1}(s_t,a_t) = (1-\alpha)\, Q_t(s_t,a_t) + \alpha\, \big(r_t + \gamma \max_{a'} Q_t(s_{t+1}, a')\big),$$4,

$$Q_{t+1}(s_t,a_t) = (1-\alpha)\, Q_t(s_t,a_t) + \alpha\, \big(r_t + \gamma \max_{a'} Q_t(s_{t+1}, a')\big),$$5

and the stationary mean-square error satisfies

$$Q_{t+1}(s_t,a_t) = (1-\alpha)\, Q_t(s_t,a_t) + \alpha\, \big(r_t + \gamma \max_{a'} Q_t(s_{t+1}, a')\big),$$6

where $$Q_{t+1}(s_t,a_t) = (1-\alpha)\, Q_t(s_t,a_t) + \alpha\, \big(r_t + \gamma \max_{a'} Q_t(s_{t+1}, a')\big),$$7 encodes asynchronous coverage (Zhang et al., 2024). This result emphasizes that geometric convergence to a stationary distribution is not restricted to the smooth setting.

A further refinement is the Gaussian approximation of the centered-and-scaled steady state. For contractive nonlinear SA, later work gives explicit non-asymptotic Wasserstein bounds of order $$Q_{t+1}(s_t,a_t) = (1-\alpha)\, Q_t(s_t,a_t) + \alpha\, \big(r_t + \gamma \max_{a'} Q_t(s_{t+1}, a')\big),$$8 between the stationary law of $$Q_{t+1}(s_t,a_t) = (1-\alpha)\, Q_t(s_t,a_t) + \alpha\, \big(r_t + \gamma \max_{a'} Q_t(s_{t+1}, a')\big),$$9 and a Gaussian law, and specializes this framework to tabular Q-learning under differentiability conditions near $Q_t(s,a)$0 (Wang et al., 15 Feb 2026). The covariance matrix is characterized by a Lyapunov equation involving the linearized drift and the long-run covariance of the Markovian noise (Wang et al., 15 Feb 2026).

4. Averaging, central limit theory, and bias

A major advance in the constant-stepsize theory is the separation between variance reduction by averaging and bias reduction by extrapolation. For the averaged iterate

$Q_t(s,a)$1

the centered partial sums

$Q_t(s,a)$2

satisfy a central limit theorem and a functional CLT: $Q_t(s,a)$3 and the interpolated process converges to Brownian motion with covariance $Q_t(s,a)$4 (Zhang et al., 2024). This yields asymptotic normality of long time averages centered at the stationary mean $Q_t(s,a)$5.

The same paper gives a non-asymptotic analysis of Polyak–Ruppert tail averaging. For burn-in $Q_t(s,a)$6,

$Q_t(s,a)$7

and under the small-stepsize regime with a unique optimal policy separated by a positive action-gap, the mean error obeys

$Q_t(s,a)$8

while the variance scales as $Q_t(s,a)$9 and the asymptotic squared bias is of order $(s,a)\neq (s_t,a_t)$0 (Zhang et al., 2024). The linear coefficient $(s,a)\neq (s_t,a_t)$1 is explicit and depends on $(s,a)\neq (s_t,a_t)$2.

This $(s,a)\neq (s_t,a_t)$3 bias expansion enables Richardson–Romberg extrapolation. With stepsizes $(s,a)\neq (s_t,a_t)$4 and $(s,a)\neq (s_t,a_t)$5, the extrapolated estimator

$(s,a)\neq (s_t,a_t)$6

satisfies

$(s,a)\neq (s_t,a_t)$7

so the leading mean bias drops from $(s,a)\neq (s_t,a_t)$8 to $(s,a)\neq (s_t,a_t)$9 and the squared bias from $q_k \in \mathbb{R}^{|S||A|}$0 to $q_k \in \mathbb{R}^{|S||A|}$1, while the variance and optimization-error terms remain of the same order (Zhang et al., 2024).

A key qualification is that the bias order depends on local smoothness of the Bellman max. In nonsmooth contractive SA, and specifically in Q-learning, the small-stepsize stationary bias can instead scale as $q_k \in \mathbb{R}^{|S||A|}$2. The decisive distinction is whether the MDP has reachable tied optimal actions. When there exists a tied state that is not rooted (Type A), one has

$q_k \in \mathbb{R}^{|S||A|}$3

when there are no such effective ties (Type B), the leading $q_k \in \mathbb{R}^{|S||A|}$4 term vanishes, and under sufficient moments

$q_k \in \mathbb{R}^{|S||A|}$5

for any $q_k \in \mathbb{R}^{|S||A|}$6 covered by the moment assumptions (Zhang et al., 2024). This resolves an apparent contradiction in the literature: the $q_k \in \mathbb{R}^{|S||A|}$7 expansion is a local-linearization result under a positive action-gap, whereas the $q_k \in \mathbb{R}^{|S||A|}$8 bias is a genuinely nonsmooth phenomenon caused by the kink of the max operator at reachable ties.

5. Finite-time bounds and control-theoretic viewpoints

Parallel to the stationary-law literature, several works analyze constant-stepsize Q-learning through finite-time envelopes. For synchronous Q-learning, cone-contractive analysis yields a deterministic sandwich relation on the error and, under constant $q_k \in \mathbb{R}^{|S||A|}$9, leads to an expected $x_k=(s_k,a_k,s_{k+1})$0 bound of the form

$x_k=(s_k,a_k,s_{k+1})$1

with $x_k=(s_k,a_k,s_{k+1})$2 (Wainwright, 2019). The transient decays geometrically, while the steady-state floor is variance-dominated at order $x_k=(s_k,a_k,s_{k+1})$3 in $x_k=(s_k,a_k,s_{k+1})$4.

For asynchronous Q-learning under i.i.d. visitation, a switching-system analysis constructs lower and upper comparison processes and proves a finite-time bound for the running average

$x_k=(s_k,a_k,s_{k+1})$5

of the form

$x_k=(s_k,a_k,s_{k+1})$6

thereby making the classical constant-stepsize bias-variance trade-off explicit in the averaged-iterate error (Lee et al., 2021). The same framework explains overestimation through the nonnegative contribution induced by the Bellman maximum (Lee et al., 2021).

A sharper direct-switching theory represents the error recursion as

$x_k=(s_k,a_k,s_{k+1})$7

and identifies the intrinsic drift rate with the joint spectral radius

$x_k=(s_k,a_k,s_{k+1})$8

which satisfies

$x_k=(s_k,a_k,s_{k+1})$9

and can be strictly smaller (Lee, 21 Apr 2026). This yields finite-time last-iterate bounds such as

$Q^*$00

as well as a computable quadratic-certificate version based on LMIs and common quadratic Lyapunov functions (Lee, 21 Apr 2026).

A sign-separated refinement decomposes the error into positive and negative parts, $Q^*$01. The negative part is dominated by a lower comparison LTI system associated with a fixed optimal policy, while the positive part is controlled by a switching family over all deterministic policies. The resulting certificates satisfy

$Q^*$02

so the negative-side envelope is no slower and may be faster than the positive-side envelope (Lee, 15 May 2026). This formalizes a max-induced asymmetry: positive errors can be selected and propagated by the Bellman maximum, whereas negative errors admit an optimal-policy lower comparison, directly connecting constant-stepsize dynamics to overestimation (Lee, 15 May 2026).

6. Extensions, applications, and scope

The tabular theory has motivated several extensions. In restless-bandit index learning, constant-stepsize Q-learning appears as the fast timescale of a two-timescale stochastic approximation scheme, with the slow timescale updating a subsidy or Whittle index. Under Lipschitz, contraction, bounded-noise, and coverage assumptions, the steady-state neighborhood scales as

$Q^*$03

where $Q^*$04 is the Q-learning stepsize and $Q^*$05 is the slower index-update stepsize (Mittal et al., 2024). This makes constant-stepsize Q-learning a component in broader adaptive-control procedures rather than merely a standalone RL algorithm.

With linear function approximation, finite-sample guarantees exist under explicit stability assumptions on the behavior policy. For off-policy linear Q-learning with geometrically mixing Markovian samples, constant stepsize yields

$Q^*$06

so the asymptotic mean-square neighborhood is $Q^*$07 (Chen et al., 2019). The same paper emphasizes that no projection step is needed and that the samples need not be i.i.d. (Chen et al., 2019).

Other work addresses the instability of constant-step Q-learning with function approximation by modifying the algorithm itself. A stabilized linear-architecture variant uses a second-order streaming update, target networks, and a replay-like policy-replay mechanism; within each epoch, the targets are fixed, the data distribution is stationary, and the update is exactly equivalent to regularized least squares (Zanette et al., 2022). An implicit variant reformulates the update as a fixed-point equation, producing the adaptive normalization

$Q^*$08

which substantially enlarges the range of stable constant stepsizes in both Q-learning and SARSA under the paper’s assumptions (Kim et al., 26 Jan 2026).

Empirically, the tabular theory of constant-stepsize asynchronous Q-learning has been validated on a $Q^*$09 Gridworld and a $Q^*$10 slippery Frozen-Lake setting. In those experiments, larger $Q^*$11 led to faster convergence, the bias of tail averaging saturated at a level approximately proportional to $Q^*$12, Richardson–Romberg extrapolation reduced the final bias relative to tail averaging at the same $Q^*$13, and constant-stepsize tail averaging and extrapolation substantially outperformed diminishing-stepsize baselines in transient speed, especially in the harder $Q^*$14 Gridworld (Zhang et al., 2024). Additional experiments with linear function approximation showed qualitatively similar behavior, although that paper’s formal theory remained tabular (Zhang et al., 2024).

The current theory remains sharply delimited. The strongest results concern finite MDPs with tabular representation, fixed stationary behavior policies, bounded rewards, small enough constant stepsizes, and—when explicit $Q^*$15 bias expansions are desired—a positive action-gap ensuring local linearization of the max operator (Zhang et al., 2024). General nonlinear function approximation, non-stationary behavior policies, adaptive stepsizes, and deep Q-learning fall outside the scope of these proofs (Zhang et al., 2024). A plausible implication is that constant-stepsize Q-learning should be viewed less as a single asymptotic object than as a family of regimes whose stationary bias, fluctuation law, and finite-time envelope depend sensitively on sampling, smoothness at the optimum, and the presence or absence of optimal-action ties.