Gaussian Approximation for Asynchronous Q-learning

Abstract: In this paper, we derive rates of convergence in the high-dimensional central limit theorem for Polyak-Ruppert averaged iterates generated by the asynchronous Q-learning algorithm with a polynomial stepsize $k^{-ω},\, ω\in (1/2, 1]$. Assuming that the sequence of state-action-next-state triples $(s_k, a_k, s_{k+1})_{k \geq 0}$ forms a uniformly geometrically ergodic Markov chain, we establish a rate of order up to $n^{-1/6} \log^{4} (nS A)$ over the class of hyper-rectangles, where $n$ is the number of samples used by the algorithm and $S$ and $A$ denote the numbers of states and actions, respectively. To obtain this result, we prove a high-dimensional central limit theorem for sums of martingale differences, which may be of independent interest. Finally, we present bounds for high-order moments for the algorithm's last iterate.

• The paper presents a non-asymptotic Gaussian approximation for Polyak–Ruppert averaged iterates in asynchronous Q-learning using polynomial step sizes.
• It establishes high-dimensional Berry–Esseen-type bounds with a convergence rate of order n^(-1/6) and only polylogarithmic dependence on state-action dimensions.
• The analysis leverages a novel high-dimensional martingale CLT to enable valid inference and confidence interval construction in reinforcement learning.

Gaussian Approximation for Asynchronous Q-learning: Technical Summary

Overview and Motivation

This work addresses the non-asymptotic Gaussian approximation for Polyak–Ruppert (PR) averaged iterates in asynchronous Q-learning algorithms when employing polynomial step sizes. Asynchronous Q-learning, in contrast to its synchronous counterpart, observes and updates only a single state-action pair per iteration, reflecting practical scenarios with limited data access. The primary technical challenge is rigorously quantifying the convergence of the averaged Q-learning iterates’ distribution to a multivariate Gaussian, accounting for Markovian noise and high-dimensionality (i.e., large state/action spaces). This non-asymptotic analysis is critical for constructing valid confidence intervals and for statistical inference on Q-learning iterates in high dimensions.

Main Contributions

The paper’s core contributions include:

• Non-asymptotic high-dimensional Berry–Esseen-type bounds for the distribution of $\sqrt{n}(\bar{Q}_n - Q^*)$ over hyper-rectangles in $\mathbb{R}^{SA}$, where $n$ is the number of samples, and $S, A$ are the sizes of the state and action spaces.
• Establishment of a convergence rate of order $n^{-1/6}\,\log^4(nSA)$ (up to constants) under geometric ergodicity for the Markov chain of transition tuples, for polynomial step sizes $\alpha_k \asymp k^{-\omega}$, $\omega \in (1/2, 1]$.
• A general high-dimensional martingale central limit theorem (CLT) with explicit rates for sums of vector-valued martingale differences, removing the necessity for lower bounds on individual conditional variances—thus improving upon [KOJEVNIKOV2022109448].
• Non-asymptotic high-order moment bounds for the last iterate $Q_n$, with explicit dependencies on the mixing time, planning horizon, and policy exploration parameters.

Numerically, the absence of polynomial dependence on state/action dimension in the Berry–Esseen bounds is particularly strong, aligning with best-possible rates for high-dimensional normal approximations over hyper-rectangles.

Technical Approach

The analysis proceeds by relating the Q-learning update dynamics to a stochastic approximation recursion, linearized around $Q^*$ via Bellman operator expansion. The update rule in the asynchronous setting is

$$Q_{t+1}(s_t, a_t) = Q_t(s_t, a_t) + \alpha_t \big[ r_t + \gamma \max_{a'} Q_t(s_{t+1}, a') - Q_t(s_t, a_t) \big]$$

with the Polyak–Ruppert average $\mathbb{R}^{SA}$0.

Key innovations involve:

• Reduction to a martingale structure: Leveraging Poisson equation techniques [douc:moulines:priouret:soulier:2018], the Markovian noise is decomposed, yielding a leading martingale term plus a negligible remainder.
• High-dimensional CLT under weak local conditions: The main CLT (Theorem~1) gives rates that avoid conditions on the minimal conditional variances of the increments, instead expressing the rate via global process characteristics. This is essential due to the coordinate-wise update structure of asynchronous Q-learning.
• Polylogarithmic dependence on dimensionality: By measuring convergence over hyper-rectangles, the dimension enters only inside logarithms, leveraging smoothing inequalities and recent high-dimensional Gaussian approximation tools [chernozhukov2022improvedcentrallimittheorem].
• Moment and mixing analysis: The moment bounds for $\mathbb{R}^{SA}$1 incorporate explicit dependencies on ergodicity (mixing) rates and exploration probability, reflecting how these system characteristics affect both the means and higher moments of the Q-learning iterates.

Main Results

Non-asymptotic Gaussian Approximation

For the PR averaged Q-learning iterate in the asynchronous trajectory setting, under uniform geometric ergodicity and optimality gap conditions, and polynomial step sizes $\mathbb{R}^{SA}$2 with $\mathbb{R}^{SA}$3, the following holds:

• The Kolmogorov distance $\mathbb{R}^{SA}$4 between the law of $\mathbb{R}^{SA}$5 and the Gaussian $\mathbb{R}^{SA}$6, measured over all hyper-rectangles in $\mathbb{R}^{SA}$7, admits the bound

$\mathbb{R}^{SA}$8

for suitably chosen $\mathbb{R}^{SA}$9 (specifically, optimizing at $n$0). This rate matches recent results in Wasserstein distance [liu2025central] but improves as it applies to the (stronger) Kolmogorov distance and does not scale polynomially with $n$1.

Moment Bounds

For the last iterate, for any $n$2 and $n$3:

$n$4

capturing the effects of the step size, discount factor, and minimal visitation probability of the behavior policy.

Comparison to Prior Work

Previous analyses have established sample complexity and moment bounds for synchronous Q-learning and for asynchronous Q-learning with different step schedules. This work advances these by:

• Improving the dimensionality dependence: Prior Berry–Esseen bounds for martingale CLT over hyper-rectangles required local lower bounds on all conditional variances, which may not be available in asynchronous Q-learning. The present work only relies on global characteristics, thus making the result broadly applicable.
• Sharper rates and inferential procedures: The $n$5 convergence rate, without $n$6 scaling outside logarithms, achieves the optimal (up to logs) rate for high-dimensional coordinate-wise inference.
• Bridging theoretical and practical analysis: Explicit dependence on mixing and exploration properties informs both theory and practice for large, weakly mixing MDPs.

Theoretical and Practical Implications

The main result enables valid high-dimensional inference (e.g., confidence region coverage, simultaneous tests) for Q-learning in realistic online settings without a generative model. The results directly inform how long to run Q-learning to guarantee approximate normality for inferential procedures.

The general high-dimensional martingale CLT developed here has broader implications for stochastic approximation in reinforcement learning and elsewhere, potentially extending to other RL algorithms with complex Markovian or non-i.i.d. noise.

Future Research Directions

Two particularly promising directions are:

• Improvement of convergence rates to $n$7 by leveraging refined expansions or bounding the smallest singular values appearing in the general CLT, potentially matching best-known rates in other high-dimensional stochastic approximation settings.
• Validation and analysis of multiplier bootstrap procedures for high-dimensional confidence interval construction in Q-learning, paralleling advances seen for stochastic gradient-type algorithms.

Conclusion

This work establishes theoretically sharp non-asymptotic Gaussian approximations for asynchronous Q-learning with Polyak–Ruppert averaging in the high-dimensional regime. The results are both technically significant for high-dimensional stochastic processes and practically valuable for statistical inference in large-scale reinforcement learning environments, expanding the toolkit available for RL analysis and applications.

---

Reference:

"Gaussian Approximation for Asynchronous Q-learning" (2604.07323)