Lyapunov-Certified Direct Switching Theory for Q-Learning

Abstract: Q-learning is one of the most fundamental algorithms in reinforcement learning. We analyze constant-stepsize Q-learning through a direct stochastic switching system representation. The key observation is that the Bellman maximization error can be represented exactly by a stochastic policy. Therefore, the Q-learning error admits a switched linear conditional-mean recursion with martingale-difference noise. The intrinsic drift rate is the joint spectral radius (JSR) of the direct switching family, which can be strictly smaller than the standard row-sum rate. Using this representation, we derive a finite-time final-iterate bound via a JSR-induced Lyapunov function and then give a computable quadratic-certificate version.

• The paper introduces a direct stochastic-policy switching representation that exactly linearizes the Bellman maximization error, eliminating the affine term in Q-learning analysis.
• It characterizes convergence via the joint spectral radius and derives finite-time error bounds using a novel Lyapunov function approach.
• When a common quadratic Lyapunov function exists, the framework yields practical quadratic certificates and sharper instance-dependent error bounds.

Lyapunov-Certified Direct Switching Theory for Q-Learning

Overview of Contributions

The paper develops a new theoretical framework for analyzing constant-stepsize Q-learning as a stochastic switched linear system, introducing a direct stochastic-policy switching representation that precisely characterizes the error dynamics of Q-learning without resorting to comparison systems or affine terms. The principal insight is that the Bellman maximization error admits an exact, convex-hull-based linearization via stochastic policies, enabling the removal of the affine term typically present in previous switched-system formulations. The deterministic drift rate of the error, central to convergence analysis, is governed by the joint spectral radius (JSR) of a switching family defined by the set of deterministic stationary policies. The paper establishes finite-time error bounds based on this direct JSR rate and provides a computable quadratic-certificate version when the switching family admits a common quadratic Lyapunov function (CQLF).

Direct Stochastic-Policy Switching Representation

The core theoretical innovation is the exact linearization of the Bellman-maximization error through stochastic policy averaging. For any Q-function $Q$, the difference $V_Q - V^*$ (where $V^*$ is the optimal value function) lies in the convex hull of the corresponding actionwise Q-errors. This observation allows for the representation: $V_Q - V^* = \Pi^{\mu_Q}(Q - Q^*)$ where $\mu_Q$ is a suitable stochastic policy. Substituting this stochastic-policy linearization into the Q-learning recursion, the error dynamics can be written as: $e_{k+1} = M_{\mu_k} e_k + \alpha w_k$ with $$M_{\mu_k} = I - \alpha D + \alpha \gamma D P \Pi^{\mu_k}$$, where $\mu_k$ is measurable with respect to the process filtration and $w_k$ is a martingale-difference noise term. Critically, the deterministic drift is now purely linear—no affine term remains—and is governed by the direct switching family indexed by deterministic policies.

Joint Spectral Radius and Finite-Time Bounds

The deterministic drift rate, which determines exponential convergence, is characterized by the JSR: $\rho_\alpha^{\mathrm{dir}} = (\mathcal{M}_\alpha)$ with $V_Q - V^*$0 denoting the family $V_Q - V^*$1, the set of matrices corresponding to all deterministic policies. Importantly, $V_Q - V^*$2 can be strictly smaller than the standard row-sum rate (the upper bound used in classical analysis), particularly in settings with nonuniform sampling or mixing transition kernels.

Finite-time final-iterate bounds are derived using a Lyapunov function constructed from products of switching matrices—a JSR-induced Lyapunov function: $V_Q - V^*$3 for $V_Q - V^*$4. The bound on the expected error is: $V_Q - V^*$5 where $V_Q - V^*$6 is a norm-equivalence constant and $V_Q - V^*$7 is a bound on the noise variance.

Quadratic Certificate and Computable Bounds

When the switching family admits a CQLF, the analysis simplifies via the quadratic expansion, and the cross-term vanishes exactly. If $V_Q - V^*$8 and $V_Q - V^*$9 satisfy $V^*$0 for all $V^*$1, then: $V^*$2 with $V^*$3 the condition number of $V^*$4. Such quadratic certificates can be checked using LMIs, complementing or superseding bounds obtained from weighted sup-norm contraction.

Weighted Infinity-Norm Certificates

To further refine the deterministic drift rate, the paper leverages weighted infinity-norm contraction certificates. For a vector $V^*$5, the contraction factor $V^*$6 is: $V^*$7 The analysis yields explicit error bounds in terms of $V^*$8 and the condition number $V^*$9. Optimization over $V_Q - V^* = \Pi^{\mu_Q}(Q - Q^*)$0 can certify smaller drift rates than the row-sum bound; this facilitates sharper, instance-dependent bounds without requiring a quadratic certificate.

Numerical Illustration

A concrete example demonstrates that the direct JSR rate can be strictly smaller than the row-sum bound. In a two-state, single-action MDP with nonuniform sampling, the spectral radius of the switching matrix was empirically observed to be approximately $V_Q - V^* = \Pi^{\mu_Q}(Q - Q^*)$1, compared to the row-sum bound at $V_Q - V^* = \Pi^{\mu_Q}(Q - Q^*)$2. This difference, while modest in this small case, can translate into significant improvements in convergence rate in larger or more complex MDPs.

Implications and Theoretical Significance

The direct switching-system representation resolves limitations in prior analyses that relied on comparison systems resulting in conservative exponential rates and additional transient terms. The precise characterization of the error drift via the JSR and the provision of Lyapunov certificates (either quadratic or weighted norm) enables sharper, instance-dependent error bounds for Q-learning. Practically, this framework assists in certifying convergence rates and noise floors in reinforcement learning, crucial for settings where finite-time performance guarantees are mandatory.

Theoretically, the connection between stochastic switched systems, JSR, and Lyapunov function constructions in RL may catalyze further research into advanced certificate-based sample complexity analyses, robust RL algorithms, and the development of new contraction criteria or switching family structures beyond deterministic policies. Research directions include generalization to Markovian observation models, adaptive or diminishing stepsize schedules, and integration into more complex RL architectures (e.g., deep Q-learning, stochastic games).

Conclusion

The paper presents a rigorous, certificate-driven switching-system analysis of constant-stepsize Q-learning, yielding exact and computable finite-time bounds. By leveraging the convex-hull linearization of the Bellman maximization error and characterizing the deterministic drift via the JSR of a direct switching family, the analysis provides sharper, instance-dependent convergence rates and noise floors compared to classical comparison-system approaches. The integration of quadratic and weighted norm cancellation offers both theoretical elegance and practical computability, establishing a solid foundation for further developments in reinforcement learning error analysis.

["Lyapunov-Certified Direct Switching Theory for Q-Learning" (2604.19569)]