Polyak-Ruppert Averaged Q-learning

• Polyak-Ruppert Averaged Q-learning is a technique that averages Q-iterates to mitigate transient bias and stabilize learning.
• It leverages full and tail averaging schemes to derive asymptotic linearity, Gaussian approximations, and optimal finite-sample error bounds.
• The approach extends to synchronous, asynchronous, and function-approximation settings, enabling robust online inference and improved convergence guarantees.

Searching arXiv for the cited papers to ground the article with fresh references. Polyak–Ruppert averaged Q-learning is the practice of replacing the last Q-learning iterate by an average of iterates, most commonly the Cesàro average

$\bar Q_T=\frac{1}{T}\sum_{t=1}^T Q_t,$

or, in some regimes, a tail average over the final segment of the trajectory. In the stochastic-approximation view of Q-learning, averaging suppresses transient bias and exposes a more regular fluctuation structure, which has led to central limit theorems, functional central limit theorems, non-asymptotic concentration bounds, and parameter-free optimal-rate results across synchronous, asynchronous, discounted, average-reward, and function-approximation settings (Li et al., 2021, Khodadadian et al., 27 May 2025, Naskar et al., 8 Aug 2025, Liu, 23 Sep 2025, Rubtsov et al., 8 Apr 2026, Rubtsov et al., 17 May 2026).

1. Basic formulation and averaging schemes

In discounted tabular Q-learning, the asynchronous update studied in several recent analyses updates only the visited coordinate: $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$ with all other entries unchanged (Rubtsov et al., 8 Apr 2026). A closely related formulation writes

$$Q_{k+1}(s,a)=Q_k(s,a)+\alpha_k\mathbf{1}_{\{(s,a)=(S_k,A_k)\}} \Bigl(\mathcal R(S_k,A_k)+\gamma\max_{a'}Q_k(S_k',a')-Q_k(s,a)\Bigr),$$

which is the asynchronous tabular recursion used in a high-probability concentration analysis (Khodadadian et al., 27 May 2025). In synchronous tabular settings with a generative model, all state-action coordinates are updated simultaneously through an empirical Bellman operator $\widehat T_t$, and the averaged estimator remains

$\bar Q_T=\frac1T\sum_{t=1}^T Q_t$

(Li et al., 2021).

The most common averaging rule is full-trajectory Polyak–Ruppert averaging. In that setting, inference and concentration are formulated in terms of $\bar Q_T-Q^*$, or equivalently in terms of the normalized partial sum

$\frac{1}{\sqrt{T}}\sum_{t=1}^T (Q_t-Q^*).$

This equivalence is exploited in both asymptotic and non-asymptotic analyses (Li et al., 2021, Liu, 23 Sep 2025).

Recent work shows that the averaging window is itself a design choice rather than a fixed convention. For power-law decay to zero step sizes $\eta_{t,n}=\eta(1-t/n)^\nu$, including the linear-decay-to-zero case $\nu=1$, the usual full Polyak–Ruppert average is described as not appropriate, because early iterates behave too much like constant-step-size iterates and may not be centered well enough around $Q^\star$. In that regime, the relevant estimator is a tail average over the last $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$0 iterates (Bonnerjee et al., 5 Apr 2026).

A related variant, sample-averaged Q-learning, combines iterate averaging with within-update batch averaging. Its update uses a batch-averaged empirical Bellman operator, while inference is still based on the time-average

$$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$1

This places Polyak–Ruppert averaging inside a broader family of averaged Q-learning estimators (Panda et al., 2024).

2. Asymptotic statistical theory in synchronous tabular settings

A detailed asymptotic theory was established for synchronous, tabular, discounted Q-learning under a generative model (Li et al., 2021). The central object is the standardized partial-sum process

$$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$2

Under bounded fourth moments, a Lipschitz or gap condition controlling the greedy-policy switch, and slowly decaying step sizes such as $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$3 with $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$4, this process converges weakly to a rescaled Brownian motion: $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$5 At $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$6, this yields the ordinary central limit theorem

$$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$7

(Li et al., 2021).

The same analysis identifies averaged Q-learning as a regular asymptotically linear estimator. Specifically,

$$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$8

where $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$9. The influence function is therefore

$$Q_{k+1}(s,a)=Q_k(s,a)+\alpha_k\mathbf{1}_{\{(s,a)=(S_k,A_k)\}} \Bigl(\mathcal R(S_k,A_k)+\gamma\max_{a'}Q_k(S_k',a')-Q_k(s,a)\Bigr),$$0

and the estimator attains the semiparametric efficiency bound among regular asymptotically linear estimators in that model (Li et al., 2021).

A practical consequence is fully online inference through a self-normalized statistic constructed from the full partial-sum path. The limiting distribution is pivotal and does not require explicit estimation of the asymptotic covariance, which differentiates the method from batch-mean or covariance-plug-in procedures (Li et al., 2021).

The finite-sample side of the same theory gives a non-asymptotic $$Q_{k+1}(s,a)=Q_k(s,a)+\alpha_k\mathbf{1}_{\{(s,a)=(S_k,A_k)\}} \Bigl(\mathcal R(S_k,A_k)+\gamma\max_{a'}Q_k(S_k',a')-Q_k(s,a)\Bigr),$$1 analysis for $$Q_{k+1}(s,a)=Q_k(s,a)+\alpha_k\mathbf{1}_{\{(s,a)=(S_k,A_k)\}} \Bigl(\mathcal R(S_k,A_k)+\gamma\max_{a'}Q_k(S_k',a')-Q_k(s,a)\Bigr),$$2. For polynomial step sizes, the dominant term matches the instance-dependent lower bound in the tabular setting, and in the worst case the leading term yields the optimal minimax sample complexity $$Q_{k+1}(s,a)=Q_k(s,a)+\alpha_k\mathbf{1}_{\{(s,a)=(S_k,A_k)\}} \Bigl(\mathcal R(S_k,A_k)+\gamma\max_{a'}Q_k(S_k',a')-Q_k(s,a)\Bigr),$$3 for $$Q_{k+1}(s,a)=Q_k(s,a)+\alpha_k\mathbf{1}_{\{(s,a)=(S_k,A_k)\}} \Bigl(\mathcal R(S_k,A_k)+\gamma\max_{a'}Q_k(S_k',a')-Q_k(s,a)\Bigr),$$4-accuracy in $$Q_{k+1}(s,a)=Q_k(s,a)+\alpha_k\mathbf{1}_{\{(s,a)=(S_k,A_k)\}} \Bigl(\mathcal R(S_k,A_k)+\gamma\max_{a'}Q_k(S_k',a')-Q_k(s,a)\Bigr),$$5 (Li et al., 2021). Entropy-regularized Q-learning admits analogous asymptotic and finite-sample results without the hard-max Lipschitz assumption, because the soft Bellman operator is smooth and has a unique fixed point (Li et al., 2021).

3. High-probability concentration and the stochastic-approximation viewpoint

A recent general theorem treats Polyak–Ruppert averaging as a modular stochastic-approximation construction rather than a Q-learning-specific device (Khodadadian et al., 27 May 2025). The base recursion is

$$Q_{k+1}(s,a)=Q_k(s,a)+\alpha_k\mathbf{1}_{\{(s,a)=(S_k,A_k)\}} \Bigl(\mathcal R(S_k,A_k)+\gamma\max_{a'}Q_k(S_k',a')-Q_k(s,a)\Bigr),$$6

and the averaged iterate is

$$Q_{k+1}(s,a)=Q_k(s,a)+\alpha_k\mathbf{1}_{\{(s,a)=(S_k,A_k)\}} \Bigl(\mathcal R(S_k,A_k)+\gamma\max_{a'}Q_k(S_k',a')-Q_k(s,a)\Bigr),$$7

The theorem assumes a norm $$Q_{k+1}(s,a)=Q_k(s,a)+\alpha_k\mathbf{1}_{\{(s,a)=(S_k,A_k)\}} \Bigl(\mathcal R(S_k,A_k)+\gamma\max_{a'}Q_k(S_k',a')-Q_k(s,a)\Bigr),$$8 whose square is $$Q_{k+1}(s,a)=Q_k(s,a)+\alpha_k\mathbf{1}_{\{(s,a)=(S_k,A_k)\}} \Bigl(\mathcal R(S_k,A_k)+\gamma\max_{a'}Q_k(S_k',a')-Q_k(s,a)\Bigr),$$9-smooth, a fixed point $\widehat T_t$0 of the mean operator $\widehat T_t$1, invertibility of $\widehat T_t$2, local pseudo-smoothness of $\widehat T_t$3, and sub-Gaussian noise at the fixed point and in the Jacobian (Khodadadian et al., 27 May 2025).

The theorem is deliberately modular. If, for every $\widehat T_t$4, the non-averaged iterates satisfy

$\widehat T_t$5

with probability at least $\widehat T_t$6, then the averaged iterate obeys

$\widehat T_t$7

with probability at least $\widehat T_t$8. The term $\widehat T_t$9 is the main averaging term and contains the $\bar Q_T=\frac1T\sum_{t=1}^T Q_t$0 rate with leading $\bar Q_T=\frac1T\sum_{t=1}^T Q_t$1 concentration, whereas $\bar Q_T=\frac1T\sum_{t=1}^T Q_t$2 is a transient term associated with the part of the trajectory preceding entry into the local pseudo-smoothness region. If the operator is globally smooth, or if $\bar Q_T=\frac1T\sum_{t=1}^T Q_t$3, then $\bar Q_T=\frac1T\sum_{t=1}^T Q_t$4 disappears entirely. The leading behavior is summarized as

$\bar Q_T=\frac1T\sum_{t=1}^T Q_t$5

(Khodadadian et al., 27 May 2025).

Specialized to asynchronous tabular Q-learning with i.i.d. samples, the same framework uses the average

$\bar Q_T=\frac1T\sum_{t=1}^T Q_t$6

The sampling distribution is induced by a behavior policy $\bar Q_T=\frac1T\sum_{t=1}^T Q_t$7, and the exploration parameter is

$\bar Q_T=\frac1T\sum_{t=1}^T Q_t$8

A key structural condition is greedy uniqueness: for every $\bar Q_T=\frac1T\sum_{t=1}^T Q_t$9, the greedy action $\bar Q_T-Q^*$0 is unique. This yields the local pseudo-smoothness required by the general theorem. With $\bar Q_T-Q^*$1, $\bar Q_T-Q^*$2, the analysis uses $\bar Q_T-Q^*$3 and $\bar Q_T-Q^*$4, and the mean operator satisfies an explicit contraction bound whose factor is

$\bar Q_T-Q^*$5

provided

$\bar Q_T-Q^*$6

With step size $\bar Q_T-Q^*$7 and $\bar Q_T-Q^*$8, the averaged Q-learning iterate satisfies

$\bar Q_T-Q^*$9

with probability at least $\frac{1}{\sqrt{T}}\sum_{t=1}^T (Q_t-Q^*).$0 (Khodadadian et al., 27 May 2025).

That result is positioned as the first high-probability bound on the full error distribution for averaged Q-learning, with a step size that does not depend on $\frac{1}{\sqrt{T}}\sum_{t=1}^T (Q_t-Q^*).$1 (Khodadadian et al., 27 May 2025). A plausible implication is that Polyak–Ruppert averaging can be analyzed in Q-learning without choosing a confidence-specific schedule, provided a sufficiently sharp raw-iterate bound is available.

The broader stochastic-approximation background for such results includes linear analyses in which averaged iterates admit exact covariance formulas in the CLT and directional non-asymptotic concentration inequalities whose leading term matches that covariance up to universal constants (Mou et al., 2020).

4. Asynchronous updates, Markovian sampling, and Gaussian approximation

The asynchronous case is fundamentally different from the synchronous generative-model setting because the data are Markovian and only one coordinate is updated at each iteration. One line of work formulates the trajectory $\frac{1}{\sqrt{T}}\sum_{t=1}^T (Q_t-Q^*).$2 as an irreducible and aperiodic finite-state Markov chain with stationary distribution $\frac{1}{\sqrt{T}}\sum_{t=1}^T (Q_t-Q^*).$3, defines the state-action visitation matrix

$\frac{1}{\sqrt{T}}\sum_{t=1}^T (Q_t-Q^*).$4

and measures exploration through

$\frac{1}{\sqrt{T}}\sum_{t=1}^T (Q_t-Q^*).$5

Under the local quadratic regularity condition

$\frac{1}{\sqrt{T}}\sum_{t=1}^T (Q_t-Q^*).$6

polynomial step sizes $\frac{1}{\sqrt{T}}\sum_{t=1}^T (Q_t-Q^*).$7 with $\frac{1}{\sqrt{T}}\sum_{t=1}^T (Q_t-Q^*).$8, and the matrix

$\frac{1}{\sqrt{T}}\sum_{t=1}^T (Q_t-Q^*).$9

the normalized averaged error converges to a Gaussian law with covariance $\eta_{t,n}=\eta(1-t/n)^\nu$0, and the partial-sum process converges weakly in $\eta_{t,n}=\eta(1-t/n)^\nu$1 to a scaled Brownian motion (Liu, 23 Sep 2025). The same work gives a non-asymptotic central limit theorem in 1-Wasserstein distance; its simplified corollary with $\eta_{t,n}=\eta(1-t/n)^\nu$2 has rate

$\eta_{t,n}=\eta(1-t/n)^\nu$3

up to the manuscript’s formatting imperfections (Liu, 23 Sep 2025).

A more refined Gaussian approximation result assumes that the chain of transition triples $\eta_{t,n}=\eta(1-t/n)^\nu$4 is uniformly geometrically ergodic and studies the usual Polyak–Ruppert average

$\eta_{t,n}=\eta(1-t/n)^\nu$5

With polynomial stepsize $\eta_{t,n}=\eta(1-t/n)^\nu$6, $\eta_{t,n}=\eta(1-t/n)^\nu$7, the finite-sample distribution of $\eta_{t,n}=\eta(1-t/n)^\nu$8 is approximated by a Gaussian law over hyper-rectangles, with the best leading rate obtained at $\eta_{t,n}=\eta(1-t/n)^\nu$9: $\nu=1$0 up to logarithmic factors (Rubtsov et al., 8 Apr 2026). The limiting covariance is

$\nu=1$1

and the proof relies on a Poisson-equation decomposition of the Markovian noise into a martingale-difference component plus smaller remainders (Rubtsov et al., 8 Apr 2026).

These Markovian analyses correct a common oversimplification. Averaged Q-learning is not restricted to i.i.d. or synchronous sampling. The available theory covers non-IID trajectories, exploration-dependent covariances, and process-level weak convergence, although at the cost of stronger mixing and local regularity assumptions (Liu, 23 Sep 2025, Rubtsov et al., 8 Apr 2026).

5. Step-size design, tail averaging, and inferential regimes

The choice of learning-rate schedule determines not only optimization dynamics but also the correct averaging window. For the power-law decay-to-zero family

$\nu=1$2

with linear decay to zero as the special case $\nu=1$3, the iterates behave like a constant-step-size method for $\nu=1$4 with $\nu=1$5, because $\nu=1$6 there, and only stabilize near the end where $\nu=1$7 (Bonnerjee et al., 5 Apr 2026). In this triangular-array setting, the correct Polyak–Ruppert estimator is not the full average over $\nu=1$8, but a tail average over the last $\nu=1$9 iterates (Bonnerjee et al., 5 Apr 2026).

The non-asymptotic theory for that regime shows two phases. The first is fast initial forgetting, with $Q^\star$0-type decay. The second is a terminal statistical error of order

$Q^\star$1

which is analogous in spirit to the $Q^\star$2 error of polynomially decaying schedules (Bonnerjee et al., 5 Apr 2026). Under Assumptions 1 and 2, compactness of $Q^\star$3, and

$Q^\star$4

the tail average satisfies the central limit theorem

$Q^\star$5

for any $Q^\star$6 and $Q^\star$7 (Bonnerjee et al., 5 Apr 2026). The covariance $Q^\star$8 is stated to be positive definite or semidefinite and independent of $Q^\star$9, but an exact closed-form expression is described as highly intractable (Bonnerjee et al., 5 Apr 2026).

That same analysis establishes a strong invariance principle for the tail partial sums and derives a time-uniform Gaussian approximation. Because direct estimation of $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$00 is difficult, the paper argues that the strong approximation hints at an easily implementable Gaussian bootstrap procedure by running multiple independent chains of the approximating Gaussian recursion in parallel (Bonnerjee et al., 5 Apr 2026).

The main interpretive distinction is therefore not merely between averaged and non-averaged Q-learning, but between full averaging and tail averaging under different step-size geometries. A plausible implication is that “Polyak–Ruppert averaged Q-learning” is not a single estimator class with a universal windowing rule; the averaging operator must be matched to the learning-rate schedule.

6. Extensions beyond standard discounted tabular Q-learning

Recent work extends Polyak–Ruppert averaging to settings where the Bellman operator is a nonlinear semi-norm contraction rather than a standard norm contraction. In that framework, the update has mean map $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$01 satisfying

$$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$02

which includes average-reward Q-learning in the span semi-norm

$$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$03

and discounted Q-learning in $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$04 (Naskar et al., 8 Aug 2025). The difficulty is that the span semi-norm is not monotone, which had obstructed earlier Polyak–Ruppert analyses. The key technical device is to couple the semi-norm contraction with a suitably induced monotone norm and to recast the averaged error as a linear recursion plus a nonlinear perturbation (Naskar et al., 8 Aug 2025). With the universal step size

$$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$05

the averaged iterate achieves parameter-free optimal rates: $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$06 with $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$07 independent agents (Naskar et al., 8 Aug 2025). The stated framework covers synchronous and asynchronous updates, single-agent and distributed settings, and both simulator-generated and Markovian data (Naskar et al., 8 Aug 2025).

Entropy-regularized asynchronous Q-learning with linear function approximation furnishes another extension. There, the parameter update is

$$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$08

with polynomially decaying step sizes $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$09, $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$10, and Polyak–Ruppert average

$$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$11

Under uniform geometric ergodicity, bounded features, and regularity of the projected soft Bellman equation, the normalized averaged error $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$12 admits a Gaussian approximation in convex distance. Optimizing over $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$13 gives $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$14 and rate

$$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$15

up to polylogarithmic factors (Rubtsov et al., 17 May 2026). The asymptotic covariance is

$$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$16

and entropy regularization smooths the Bellman operator and guarantees uniqueness of the soft optimal policy, avoiding degeneracy that can obstruct central limit theorems in ordinary Q-learning (Rubtsov et al., 17 May 2026).

Sample-averaged Q-learning provides a different extension in which averaging occurs both within updates and across time. With batch size $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$17, step size $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$18, $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$19, $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$20, and iterate average $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$21, the asymptotic normalization becomes

$$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$22

so the batch schedule directly changes the rate. The resulting functional central limit theorem supports random-scaling confidence intervals that avoid direct estimation of the covariance matrix $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$23 (Panda et al., 2024).

Setting	Averaging object	Representative result
Synchronous discounted tabular	Full average $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$24	FCLT, RAL property, semiparametric efficiency (Li et al., 2021)
Asynchronous discounted tabular	Full average $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$25	Non-asymptotic CLT and Gaussian approximation under Markovian sampling (Liu, 23 Sep 2025, Rubtsov et al., 8 Apr 2026)
PD2Z/LD2Z schedules	Tail average	CLT and strong approximation for the tail window (Bonnerjee et al., 5 Apr 2026)
Semi-norm contractions	Full average $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$26	Parameter-free $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$27 optimal rates (Naskar et al., 8 Aug 2025)
Entropy-regularized linear FA	Full average $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$28	High-dimensional Gaussian approximation in convex distance (Rubtsov et al., 17 May 2026)
Batch/sample-averaged Q-learning	Time-average $$Q_{t+1}(s_t,a_t)=Q_t(s_t,a_t)+\alpha_t\Bigl(r_t+\gamma\max_{a\in A}Q_t(s_{t+1},a)-Q_t(s_t,a_t)\Bigr),$$29 with within-update batching	FCLT and random-scaling inference (Panda et al., 2024)

Across these variants, the common role of Polyak–Ruppert averaging is stable: it converts a recursively updated Q-estimator into an object with sharper asymptotic variance, tractable Gaussian limits, or improved finite-sample concentration. The main points of divergence are equally stable: the admissible step-size schedules, the required local smoothness or uniqueness assumptions, the sampling model, and whether the correct averaging operator is the full Cesàro average or a tail window.