Sharp asymptotic theory for Q-learning with LDTZ learning rate and its generalization

Abstract: Despite the sustained popularity of Q-learning as a practical tool for policy determination, a majority of relevant theoretical literature deals with either constant ($η{t}\equiv η$) or polynomially decaying ($η{t} = ηt^{-α}$) learning schedules. However, it is well known that these choices suffer from either persistent bias or prohibitively slow convergence. In contrast, the recently proposed linear decay to zero (\texttt{LD2Z}: $η{t,n}=η(1-t/n)$) schedule has shown appreciable empirical performance, but its theoretical and statistical properties remain largely unexplored, especially in the Q-learning setting. We address this gap in the literature by first considering a general class of power-law decay to zero (\texttt{PD2Z}-$ν$: $η{t,n}=η(1-t/n)^ν$). Proceeding step-by-step, we present a sharp non-asymptotic error bound for Q-learning with \texttt{PD2Z}-$ν$ schedule, which then is used to derive a central limit theory for a new \textit{tail} Polyak-Ruppert averaging estimator. Finally, we also provide a novel time-uniform Gaussian approximation (also known as \textit{strong invariance principle}) for the partial sum process of Q-learning iterates, which facilitates bootstrap-based inference. All our theoretical results are complemented by extensive numerical experiments. Beyond being new theoretical and statistical contributions to the Q-learning literature, our results definitively establish that \texttt{LD2Z} and in general \texttt{PD2Z}-$ν$ achieve a best-of-both-worlds property: they inherit the rapid decay from initialization (characteristic of constant step-sizes) while retaining the asymptotic convergence guarantees (characteristic of polynomially decaying schedules). This dual advantage explains the empirical success of \texttt{LD2Z} while providing practical guidelines for inference through our results.

• The paper establishes sharp asymptotic analysis for Q-learning with LD2Z and PD2Z schedules, offering novel non-asymptotic error bounds and convergence guarantees.
• It introduces tail Polyak-Ruppert averaging to secure CLT-type results and optimal variance reduction, outperforming traditional averaging techniques.
• Numerical studies validate that LD2Z/PD2Z achieves rapid initial error decay and robust asymptotic performance, supporting its broad adoption in reinforcement learning.

Asymptotic Theory for Q-learning with LD2Z and PD2Z Step-sizes

Introduction

This paper presents an in-depth asymptotic analysis of Q-learning, focusing on linearly decaying-to-zero (LD2Z) learning rate schedules and their parametric generalization, power-law decay to zero (PD2Z). Classic choices of step-size—constant and polynomially decaying—respectively suffer from steady-state bias or slow convergence in practice. LD2Z, and more general PD2Z, have shown empirical advantages in both optimization and deep learning contexts; this work provides the first sharp theoretical characterization of their statistical and inferential properties within Q-learning. The analysis covers non-asymptotic error bounds, distributional results for tail Polyak-Ruppert averaging, and strong invariance principles enabling Gaussian approximation. Extensive numerical studies further validate theoretical claims.

Problem Formulation and Step-size Schedules

The authors analyze Q-learning for discounted, finite Markov Decision Processes (MDPs), with $Q$-updates using time-inhomogeneous stochastic approximation,

$$Q_{t,n} = (1-\eta_{t,n}) Q_{t-1,n} + \eta_{t,n} \widehat{B}_t Q_{t-1,n}$$

where $\widehat{B}_t$ is the empirical Bellman operator. The key parameter is the learning rate schedule:

• LD2Z: $\eta_{t,n} = \eta (1-t/n)$, a linear decay to zero.
• PD2Z: $\eta_{t,n} = \eta (1-t/n)^\nu$, with $\nu > 0$.

LD2Z matches empirical "knee schedules" used for LLMs and vision transformers, but prior theoretical understanding focused on constant or polynomial decay.

Non-Asymptotic Error Bounds

Under weak moment conditions (finite $p$-th moment of the reward, mild local Lipschitz condition for the Bellman operator), sharp non-asymptotic $\mathcal{L}_p$ error bounds are established for Q-learning with PD2Z schedules. The bound distinguishes a transient phase (initial iterates) and an asymptotic phase (final iterates). Notably,

• In the early phase ($t \leq n - O(n^{\nu/(\nu+1)})$), the error decays exponentially with $t$, matching the behavior of constant step-size methods.
• In the terminal phase ($$Q_{t,n} = (1-\eta_{t,n}) Q_{t-1,n} + \eta_{t,n} \widehat{B}_t Q_{t-1,n}$$0 near $$Q_{t,n} = (1-\eta_{t,n}) Q_{t-1,n} + \eta_{t,n} \widehat{B}_t Q_{t-1,n}$$1), the error converges polynomially, with rate $$Q_{t,n} = (1-\eta_{t,n}) Q_{t-1,n} + \eta_{t,n} \widehat{B}_t Q_{t-1,n}$$2.

This dual behavior—fast forgetting of initialization, then guaranteed convergence—is shown to be intrinsic to PD2Z/LD2Z regimes. The authors also analyze the sample complexity and demonstrate the weak sensitivity to the parameter $$Q_{t,n} = (1-\eta_{t,n}) Q_{t-1,n} + \eta_{t,n} \widehat{B}_t Q_{t-1,n}$$3 for large $$Q_{t,n} = (1-\eta_{t,n}) Q_{t-1,n} + \eta_{t,n} \widehat{B}_t Q_{t-1,n}$$4.

Figure 1: Comparison of polynomially decaying, LD2Z, and constant step-size Q-learning—LD2Z exhibits rapid initial error decay with no long-term bias.

Comparative Performance of Learning Schedules

Extensive simulations in gridworld environments support the theoretical predictions. LD2Z and PD2Z consistently outperform both constant and polynomially decaying step-sizes in both early and final-phase performance.

Figure 2: Empirical performance comparison for LD2Z, PD2Z ($$Q_{t,n} = (1-\eta_{t,n}) Q_{t-1,n} + \eta_{t,n} \widehat{B}_t Q_{t-1,n}$$5), and constant learning rates—PD2Z schedules reduce error more rapidly and stably than constant learning rate.

The experimental results highlight pronounced advantages in both transient and steady-state error. PD2Z is robust to the choice of $$Q_{t,n} = (1-\eta_{t,n}) Q_{t-1,n} + \eta_{t,n} \widehat{B}_t Q_{t-1,n}$$6, justifying the empirical prevalence of LD2Z.

Distributional Theory and Polyak-Ruppert Averaging

The paper demonstrates that standard Polyak-Ruppert (PR) averaging fails to achieve asymptotic normality when applied across all iterates under LD2Z/PD2Z. Instead, tail PR averaging—aggregating only the final $$Q_{t,n} = (1-\eta_{t,n}) Q_{t-1,n} + \eta_{t,n} \widehat{B}_t Q_{t-1,n}$$7 iterates—enables CLT-type results:

$$Q_{t,n} = (1-\eta_{t,n}) Q_{t-1,n} + \eta_{t,n} \widehat{B}_t Q_{t-1,n}$$8

where $$Q_{t,n} = (1-\eta_{t,n}) Q_{t-1,n} + \eta_{t,n} \widehat{B}_t Q_{t-1,n}$$9 is the tail average over the final iterates. This tail averaging achieves optimal scaling and variance reduction, outperforming classical averaging empirically.

Strong Invariance Principle and Gaussian Coupling

A key innovation is the strong invariance principle for Q-learning with PD2Z schedules: the partial sum process of the centered Q-iterates is coupled (up to $\widehat{B}_t$0 error) to a non-stationary Gaussian process with explicit covariance structure matching the Q-learning process. This construction enables time-uniform bootstrap inference with quantifiable error, going beyond classical functional CLTs.

Figure 3: Q–Q plots comparing sup-norm partial sum distributions of Q-iterates and their Gaussian approximations, illustrating accuracy of the strong invariance principle.

Empirical Q–Q plots confirm the near-exact agreement between the supremum distributions of the Q-learning process and its Gaussian surrogate, highlighting the practical utility of the strong invariance principle in uncertainty quantification and hypothesis testing for RL algorithms.

Implications, Limitations, and Future Directions

This work establishes that LD2Z and PD2Z schedules give Q-learning a best-of-both-worlds property: rapid reduction of initialization bias combined with robust asymptotic guarantees traditionally associated with polynomial decays. The strong theoretical backing justifies the widespread adoption of linearly decaying step-sizes in large-scale RL and deep learning. The tail PR averaging and bootstrap inference via strong Gaussian approximations open the door to statistically principled uncertainty quantification for value function estimation and policy optimization.

A practical limitation is that LD2Z requires pre-specification of the iteration horizon $\widehat{B}_t$1, making it most suited to offline RL. The theory does allow for moderate misspecification of $\widehat{B}_t$2, but direct extension to online/adaptive horizon settings remains open. Further, explicit rate-matching and strong approximations in high-dimensional function approximation (e.g., deep Q-networks) are a subject for future exploration.

Conclusion

The paper provides comprehensive sharp asymptotic theory and practical guidance for the use of linearly and power-law decaying step-size schedules in Q-learning, including novel distributional results and strong invariance principles that enable statistical inference. The theoretical and experimental findings elucidate why LD2Z and PD2Z are highly effective in practice and suggest their broader adoption and further study in both RL and general stochastic approximation frameworks.

(2604.04218)