Q-learning: Foundations & Innovations

• Q-learning is a model-free RL algorithm that incrementally estimates optimal action-value functions using Bellman updates from observed transitions.
• Advanced variants like double Q-learning and smoothed Q-learning mitigate bias and instability, enhancing performance in both tabular and large-scale regimes.
• Recent innovations extend Q-learning to continuous-time systems, combinatorial optimizations, and adaptive learning rates, boosting convergence and robustness.

Q-learning is a foundational model-free reinforcement learning (RL) algorithm that estimates the optimal action-value function for Markov decision processes (MDPs) by incremental updates from sampled experiences. At each iteration, Q-learning improves its approximation of the optimal $Q^*(s,a)$ by applying the Bellman optimality operator, using only observations of tuple transitions $(s,a,r,s')$—that is, without requiring prior knowledge of transition dynamics. The algorithm is a cornerstone of both theoretical and applied RL, underpinning practically all major advances in value-based RL in both tabular and large-scale function-approximation regimes, and serving as the substrate for many algorithmic innovations targeted at mitigating bias, sample inefficiency, and instability.

1. Canonical Algorithm, Assumptions, and Convergence

Standard Q-learning operates over finite MDPs $$\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, r, \gamma)$$, with state space $\mathcal{S}$, action space $\mathcal{A}$, transition kernel $P$, bounded reward function $r$, and discount factor $\gamma\in[0,1)$. The algorithm maintains a table (or parameterization) of Q-values $Q_t(s,a)$. Upon observing a transition $(s_t, a_t, r_t, s_{t+1})$, it updates as

$$Q_{t+1}(s_t,a_t) = Q_t(s_t,a_t) + \alpha_t(s_t,a_t) \left[ r_t + \gamma\max_{a'} Q_t(s_{t+1},a') - Q_t(s_t,a_t) \right]$$

where $0 < \alpha_t(s,a) \leq 1$ is a Robbins–Monro stepsize sequence. All other $Q(s',a')$ remain unchanged (Regehr et al., 2021).

The following conditions are necessary for almost sure convergence to the unique fixed point of the Bellman optimality operator $Q^*$ (Regehr et al., 2021):

• The MDP is finite (i.e., $|\mathcal{S}|$, $|\mathcal{A}|$ finite).
• The reward function is bounded.
• Discount factor $\gamma<1$ to ensure contractivity.
• Each $(s,a)$ is visited infinitely often.
• Stepsizes satisfy $\sum_{t}\alpha_t(s,a)=\infty$, $\sum_{t}\alpha_t(s,a)^2<\infty$.

Under these conditions, $Q_t$ converges to $Q^*$ w.p. 1, as proven using stochastic approximation theory and contraction mappings (Banach’s fixed point theorem) (Regehr et al., 2021).

2. Variants Addressing Estimation Bias and Instability

Overestimation Bias and Alternatives

Q-learning’s use of the $\max$ operator over noisy Q-estimates introduces a positive bias, known as maximization bias, which can slow or destabilize learning in stochastic-reward environments (Barber, 2023, Zhu et al., 2020). Double Q-learning mitigates this by decoupling action selection and evaluation—maintaining two Q-functions $Q^A$, $Q^B$ and selecting the maximizer using one and evaluating using the other—thereby nearly eliminating upward bias at the cost of slower convergence (Barber, 2023).

Smoothed Q-learning (Barber, 2023) generalizes this further by replacing the hard $\max$ with a weighted average; for example, via clipped-max or softmax distributions: $$Q_{t+1}(s_t,a_t) = Q_t(s_t,a_t) + \alpha_t \left[ r_t + \gamma \sum_{a'} q_t(a'|s_{t+1}) Q_t(s_{t+1},a') - Q_t(s_t,a_t)\right]$$ where $q_t(a'|s_{t+1})$ is a smoothing distribution that interpolates between uniform (SARSA) and $\max$ (Q-learning). For suitable annealing of smoothness, convergence and reduced overestimation are guaranteed (Barber, 2023).

Self-correcting Q-learning further interpolates between single and double estimator regimes, using a bias correction of the form $$Q_n^{\beta}(s',a') = Q_n(s',a') - \beta [Q_n(s',a') - Q_{n-1}(s',a')]$$ with tunable $\beta$, achieving almost unbiased estimation while requiring only one Q-table and providing theoretical convergence guarantees (Zhu et al., 2020).

3. Variants for Stability under Function Approximation and Large-Scale Regimes

Instability with Linear Function Approximation

The classical Q-learning update can diverge when combined with linear function approximation due to the mode-mixing effect induced by $\max$, as demonstrated in counter-examples (Lim et al., 2022). Regularized Q-learning modifies the gradient-based update to include an $\ell_2$ regularizer: $$\theta_{t+1} = \theta_t + \alpha_t \left[ \delta_t \phi(s_t, a_t) - \eta \theta_t \right]$$ with $$\delta_t = r_t + \gamma \max_{a'} \phi(s_{t+1}, a')^T \theta_t - \phi(s_t, a_t)^T \theta_t$$, and $\eta>0$. This regularizes each switching-system mode, ensuring every linear subsystem is Hurwitz and establishing global exponential stability. The formal error bound on the fixed point scales as $O(1/\eta)$ (Lim et al., 2022).

Nonlinear and Tree-Based Approximators

Q-learning with Online Random Forests (RL-ORF) replaces neural or tabular Q-approximators with ensembles of online random forests, one per action, updated using observed transitions and TD-targets via regression-style splitting and online bagging (Min et al., 2022). Empirical results in classic RL benchmarks (e.g., OpenAI Gym 'blackjack', 'cartpole-v1') indicate improved sample efficiency and reduced overfitting in moderate-dimensional tasks compared to deep Q-networks. The "expanding forests" technique, where the ensemble size grows with more samples, further stabilizes learning (Min et al., 2022).

Deep Q-Networks and Architectural Extensions

Q-learning’s practical success in high-dimensional domains is due to its combination with deep function approximators (DQN and variants) and constituent architecture refinements. Expert Q-learning (Meng et al., 2021) generalizes Dueling-DQN by splitting Q-values into zero-mean relative advantages and a separate, semi-supervised network for state-values, which can be trained on coarse-grained, human-provided labels. This yields higher empirical robustness and reduced overestimation bias, as shown in stochastic-adversarial Othello (Meng et al., 2021).

4. Specializations for Structured Domains and Continuous-Time Systems

Control and LQ Problems

Q-learning is extendable to control-theoretic settings with random parameters. For infinite-horizon discrete-time linear-quadratic (LQ) control with i.i.d. random coefficients, a Q-learning algorithm recasts the algebraic Riccati equation as a fixed point in Q-space and solves it online via stochastic approximation (Du et al., 2020):

$$Q_{t+1} = Q_t + \alpha_t \left(N_{t+1} + [A_{t+1}, B_{t+1}]^T \Pi(Q_t) [A_{t+1}, B_{t+1}] - Q_t\right)$$

$Q_t$ converges a.s. to the unique solution $Q^*$ if and only if the LQ problem is well-posed (i.e., the Riccati equation is solvable). The resultant adaptive feedback controller stabilizes the system a.s. (Du et al., 2020).

Continuous-Time Q-Learning via HJB Equations

Q-learning has been extended to the continuous-time domain using a Hamilton–Jacobi–Bellman (HJB) PDE formalism. The continuous-time Q-function is the unique viscosity solution to the HJB equation: $$\gamma Q(x,u) - \ell(x,u) - \nabla_x Q \cdot f(x,u) + M|\nabla_u Q| = 0$$ An on-policy sample-based operator approximates the solution via batch rollouts and neural network function approximation. Convergence to the viscosity solution is established under standard assumptions. Empirical demonstrations show scalability to high (up to 20) dimensional state–action spaces (Kim et al., 2019).

5. Adaptive Learning Rates and Exploration–Exploitation Schedules

Learning rate selection is critical in stabilizing Q-learning and balancing bias-variance trade-off. The Geometric Nash Approach (Bonsu, 9 Aug 2024) formulates a batch-wise optimal learning rate as the cosine of the half-angle between the episode-length and reward vectors, which simultaneously equilibrates exploration and exploitation objectives in a Nash equilibrium sense: $$\alpha^* = \sqrt{\frac{1}{2}\Bigl(1 + \frac{T \cdot R}{\|T\| \|R\|}\Bigr)}$$ This approach provides both theoretical performance bounds and practical acceleration in stabilization across a range of episodes, with empirical improvements in variance and speed of convergence (Bonsu, 9 Aug 2024).

6. Applied Q-Learning in Discrete Combinatorial and Industrial Domains

Q-learning is effective in deterministic, combinatorial domains, as demonstrated in assembly-sequencing for manufacturing (Neves et al., 2023). Here, a compact tabular Q-table over bitmask-encoded assembly states and actions was able to reliably discover optimal or near-optimal assembly sequences with >98% frequency for an 8-task problem, provided problem-specific reward shaping and careful action-space pruning are used. Empirkal results confirm rapid convergence when infeasible actions are pruned or punishments are large (Neves et al., 2023).

7. Algorithmic Innovations and Rapid Convergence Heuristics

Alternatives to the standard TD-target can accelerate convergence under specific conditions. Relative-reward-based Q-learning (Pandey et al., 2010) proposes to replace the instantaneous reward in the TD target with $\max\{r_t, r_{t-1}\}$, i.e., the maximum of the current and previous immediate reward. In deterministic grid-worlds, this technique empirically reduces the episodes needed for convergence by up to 40%. A formal convergence guarantee is not established, and stability may be compromised in stochastic or multi-agent settings (Pandey et al., 2010).

---

References:

• (Regehr et al., 2021) "An Elementary Proof that Q-learning Converges Almost Surely"
• (Barber, 2023) "Smoothed Q-learning"
• (Zhu et al., 2020) "Self-correcting Q-Learning"
• (Lim et al., 2022) "Regularized Q-learning"
• (Min et al., 2022) "Q-learning with online random forests"
• (Du et al., 2020) "A Q-learning algorithm for discrete-time linear-quadratic control with random parameters of unknown distribution: convergence and stabilization"
• (Kim et al., 2019) "Hamilton-Jacobi-Bellman Equations for Q-Learning in Continuous Time"
• (Meng et al., 2021) "Expert Q-learning: Deep Reinforcement Learning with Coarse State Values from Offline Expert Examples"
• (Neves et al., 2023) "A study on a Q-Learning algorithm application to a manufacturing assembly problem"
• (Bonsu, 9 Aug 2024) "A Geometric Nash Approach in Tuning the Learning Rate in Q-Learning Algorithm"
• (Pandey et al., 2010) "Reinforcement Learning by Comparing Immediate Reward"