--- title: RVI Q-Learning url: https://www.emergentmind.com/topics/rvi-q-learning type: topic --- # RVI Q-Learning Relative Value Iteration (RVI) Q-Learning is the canonical model-free stochastic approximation analogue of Schweitzer’s classical relative value iteration for average-reward Markov Decision Processes (MDPs) and semi-Markov decision processes (SMDPs). It is distinguished by its explicit use of a reference functional to “pin” the action-value functions, ensuring stability and almost-sure convergence in weakly communicating and unichain MDPs and SMDPs, including settings with risk sensitivity and robustness. RVI Q-Learning admits extensive theoretical guarantees and algorithmic extensions, notably asynchronous update schemes and compatibility with both tabular and function-approximation settings. ## 1. Foundations: Average-Reward Optimality Equation and Shift-Invariance The average-reward problem in finite MDPs seeks a policy π maximizing the long-run average reward: \[ \beta^* = \max_{\pi} \liminf_{N\to\infty} \mathbb{E}\left[\frac{1}{N} \sum_{n=0}^{N-1} r(X_n, U_n)\right]. \] Unlike the discounted setting, the Bellman optimality equations for the average-reward criterion admit solution sets not unique up to shifts—especially in weakly communicating MDPs where multiple recurrent classes may exist under optimal policies. Given the optimal average reward $r_*$ and some action-value function $q^*(s,a)$, the fundamental average-reward optimality equation is \[ q^*(s,a) = r(s,a) - r_* + \sum_{s'} p_{ss'}^a \max_{a'} q^*(s',a'). \] Because any additive shift of $q^*$ is again a solution unless constrained, RVI Q-Learning enforces a reference constraint (through a shift-invariant, homogeneous, and Lipschitz function $f$), thereby selecting a unique member (or a section) of the solution set [2408.16262][2209.15141]. ## 2. The RVI Q-Learning Algorithm and Its Update Rule RVI Q-Learning iteratively adjusts Q-values by a temporal-difference error, with each update subtracting a running baseline $f(Q_n)$: \[ Q_{n+1}(s,a) = Q_n(s,a) + \alpha_{\nu_n(s,a)} \left[R_{n+1}^{s,a} - f(Q_n) + \max_{a'} Q_n(s',a') - Q_n(s,a)\right]. \] All other entries stay unchanged in the asynchronous setting. The baseline $f(Q_n)$ is commonly chosen as the value at a fixed “reference” state-action pair (e.g., $Q(s_0,a_0)$), but more general forms (e.g., convex combinations, max, or mean) are permitted as long as they are strictly increasing under scalar translation (SISTr), Lipschitz, and homogeneous [2408.16262][2409.03915][2512.06218]. The update admits tabular and function-approximation versions and requires only single-sample transition and reward observations under arbitrary (sufficiently exploratory) behavior policies. In SMDPs, the RVI update normalizes by an estimated holding time: \[ Q_{n+1}(s,a) = Q_n(s,a) + \alpha_{\nu_n(s,a)} \left(\frac{R_{n+1}^{sa}+\max_{a'}Q_n(S_{n+1}^{sa},a')-Q_n(s,a)}{T_n(s,a)\vee \eta_n} - f(Q_n)\right). \] Here $T_n(s,a)$, the running estimate of the holding time for $(s,a)$, is updated by stochastic gradient descent [2512.06218][2409.03915]. ## 3. Convergence Analysis and Solution Set Structure RVI Q-Learning is underpinned by the asynchronous stochastic approximation (SA) framework. Under appropriate step-size conditions ($\sum_n\alpha_n=\infty$, $\sum_n\alpha_n^2<\infty$), sufficiently frequent updates for each $(s,a)$, weakly communicating dynamics, and a reference function $f$ satisfying SISTr, the iterates converge almost surely to a compact, connected subset of the Bellman equation solution set subject to $f(q)=r_*$ [2408.16262][2512.06218][2209.15141][2409.03915]. For general weakly communicating MDPs (multiple recurrent classes), the limiting set $Q_\infty$ is homeomorphic to an $(n^*-1)$-dimensional convex polyhedron, where $n^*$ is the minimal number of recurrent classes under optimal policies. Imposition of $f(q)=r_*$ removes exactly one degree of freedom, yielding a well-characterized but not necessarily singleton solution set [2408.16262]. Advanced SA theory, including Borkar–Meyn stability criteria and ODE shadowing arguments, ensures that, under stricter step-size/asynchrony conditions, every path converges to a unique, sample-path-dependent point in this set [2409.03915][2512.06218]. ## 4. Reference Choices, Asynchrony, and Extensions The choice of $f$ is both a practical and theoretical consideration. The mapping $f$ must be strictly increasing under shift (SISTr) and Lipschitz to guarantee stability and avoid unbounded drift. Common choices include | Reference $f(Q)$ | Properties/Note | |---------------------------|---------------------------------------| | $Q(s_0,a_0)$ | Anchors a specific pair | | $\sum_{s,a}\eta_{s,a}Q(s,a)$ | Convex combination | | $\max_{s,a} Q(s,a)$ | Upper anchoring | | $\frac{1}{|S||A|}\sum Q$ | Global average | In SMDPs, careful adjustment for holding times is required—$Q$-updates are normalized by the expected mean holding time, itself estimated online [2512.06218][2409.03915]. RVI Q-Learning is robust to asynchrony: updates may be distributed arbitrarily across state-action pairs (as long as each is visited infinitely often), facilitating scalable and parallel implementations. Extensions exist for the options framework in hierarchical RL, with inter- and intra-option RVI Q-Learning updates also converging to Bellman-optimal solution submanifolds [2408.16262]. ## 5. Risk-Aware and Robust RVI Q-Learning RVI Q-Learning generalizes naturally to risk-aware and robust settings. For dynamic risk measures or robust control (under, e.g., contamination, total-variation, KL, Wasserstein ambiguity sets), the core update is modified so that the Bellman operator incorporates worst-case or risk-adjusted next-value distributions. Nonetheless, the baseline subtraction $-f(Q_n)$ is retained, and the convergence arguments carry through to these settings under appropriate generalized operator properties and unbiasedness of inner estimators [2503.17629][2305.10504]. For example, robust RVI Q-Learning employs an inner “support function” $\sigma_{P^a_s}(V)$ capturing the worst-case expected next value, with unbiased multilevel Monte Carlo estimators used when exact computation is impractical [2305.10504][2503.17629]. ## 6. Function Approximation and Empirical Results RVI Q-Learning can be extended to function approximation, both linear (e.g., $Q_w(s,a) = w^\top \phi(s,a)$) and nonlinear (e.g., deep Q-networks). In linear settings, RVI-style batch updates have been combined with Bayesian least-squares and posterior sampling, yielding improved exploratory efficiency over $\epsilon$-greedy (as observed in the “Angrier Birds” experiments) [1601.01297]. In nonlinear, deep RL settings, architectures such as Full-Gradient DQN minimize Bellman residuals with running gain subtraction, increasing stability and convergence speed compared to discounted or semi-gradient DQN variants [2304.03729]. Empirical evaluations demonstrate that RVI Q-Learning achieves faster convergence and better long-run average reward performance than alternatives when sufficient exploration and proper reference selection are enforced. In risk-aware and robust settings, model-free RVI Q-Learning delivers convergence to risk-sensitive or robustly optimal policies, with manageable per-iteration complexity [2304.03729][2305.10504][2503.17629][1601.01297]. ## 7. Practical Implementation and Extensions Key practical guidelines include: - Use of step-sizes $\alpha_n$ decreasing but not too rapidly, e.g., $\alpha_n = 1/(n+1)^\beta$ with $\beta \in (0.5,1]$. - Asynchronous patterns where every $(s,a)$ is updated with positive limiting frequency. - Computation or estimation of holding times $T_n(s,a)$ in SMDPs. - Rigorous choice of $f$ with verified SISTr and Lipschitz properties, ensuring the baseline subtraction self-regulates the value scale. - Plug-in of unbiased risk or robust estimators when handling non-linear Bellman operators. Ongoing research directions include advanced distributed asynchronous variants, function approximation in large or continuous spaces, relaxing shift-invariance properties, analyzing convergence in more general nonconvex solution sets, and empirical benchmarks in demanding RL domains [2512.06218][2409.03915][2408.16262][2209.15141]. --- **References**: [1601.01297], [2209.15141], [2304.03729], [2305.10504], [2408.16262], [2409.03915], [2503.17629], [2512.06218] Source: https://www.emergentmind.com/topics/rvi-q-learning