Average-reward reinforcement learning in semi-Markov decision processes via relative value iteration (2512.06218v1)

Abstract: This paper applies the authors' recent results on asynchronous stochastic approximation (SA) in the Borkar-Meyn framework to reinforcement learning in average-reward semi-Markov decision processes (SMDPs). We establish the convergence of an asynchronous SA analogue of Schweitzer's classical relative value iteration algorithm, RVI Q-learning, for finite-space, weakly communicating SMDPs. In particular, we show that the algorithm converges almost surely to a compact, connected subset of solutions to the average-reward optimality equation, with convergence to a unique, sample path-dependent solution under additional stepsize and asynchrony conditions. Moreover, to make full use of the SA framework, we introduce new monotonicity conditions for estimating the optimal reward rate in RVI Q-learning. These conditions substantially expand the previously considered algorithmic framework and are addressed through novel arguments in the stability and convergence analysis of RVI Q-learning.

• The paper introduces a generalized asynchronous RVI Q-learning algorithm that decouples holding time and Q-value estimation in SMDPs.
• It establishes almost sure convergence to a compact set of optimal solutions in weakly communicating environments using novel monotonicity conditions.
• The analysis resolves prior stability gaps for asynchronous RL and guarantees unique sample path convergence under stricter step-size and asynchrony assumptions.

Average-Reward Reinforcement Learning in Semi-Markov Decision Processes via Relative Value Iteration

Overview and Motivation

The paper "Average-reward reinforcement learning in semi-Markov decision processes via relative value iteration" (2512.06218) advances the theoretical and algorithmic foundation for solving average-reward SMDPs using model-free reinforcement learning. The principal contribution is a rigorous analysis and generalization of Relative Value Iteration (RVI) Q-learning under the asynchronous stochastic approximation (SA) framework, explicitly addressing weakly communicating SMDPs, which encompass complex state recurrence structures and are representative of hierarchical or partially connected environments.

The study builds directly on contemporary results in asynchronous SA theory within the Borkar-Meyn framework and critically addresses gaps in stability proofs for asynchronous RL algorithms present in earlier works. Of particular importance is the introduction of new monotonicity conditions around the estimation of average reward rates, significantly enlarging the class of admissible RVI algorithms and providing robust convergence guarantees under realistic asynchrony and sampling assumptions.

Formal Problem Statement and SMDP Structure

Average-reward RL seeks to optimize long-run expected reward per unit time, a crucial objective in continual control or persistent task domains. The SMDP model generalizes MDPs by allowing transition durations (holding times) to be random variables, capturing event-driven or temporally extended actions. The paper focuses on finite state and action spaces, with each action at a given state leading to a transition with an associated random holding time and reward; expected values are denoted $t_{sa}$ and $r_{sa}$, respectively, and transitions to states are governed by $p_{ss'}^a$.

Central to average-reward SMDPs is the Average-reward Optimality Equation (AOE), which defines the set of optimal values $(\bar r, q)$: $$q(s, a) = r_{sa} - t_{sa} \cdot \bar r + \sum_{s'} p_{ss'}^a \max_{a'} q(s', a'),\quad \forall (s,a)$$ The paper emphasizes weakly communicating SMDPs, where the state space contains one recurrent class under some stationary policy and possibly transient states; the set of AOE solutions is non-singleton and may have multiple degrees of freedom, especially compared to the unichain case.

RVI Q-Learning Algorithm: Generalization and Structure

The authors propose a generalized asynchronous RVI Q-learning algorithm, directly motivated by Schweitzer's classical RVI but adapted for SMDPs and endowed with high flexibility in the estimation of the optimal average-reward rate. Algorithmic innovations include:

• Asynchronous updates: At each iteration, only a random subset of state-action pairs (rather than all) are updated, reflecting real-world sampling scenarios.
• Separate estimation of holding times and Q-values: Two update sequences $(\alpha_k, \beta_k)$ are employed to independently estimate state-action values $Q_n$ and expected holding times $T_n$.
• General monotonic functions $f$ for average-reward estimation: The function $f$ maps $Q_n$ to a reward-rate estimate and is required to be strictly increasing under scalar translation (SISTr) and Lipschitz continuous. This accommodates a broad range of estimators beyond simple affine combinations.
• Self-regulation via monotonicity: The monotonicity of $f$ ensures stable reward-rate estimation even under asynchronous sampling and non-uniform update frequencies.

The algorithm is update-efficient and suitable for deployment in systems without complete model information or uniform sampling—key for distributed and hierarchical RL scenarios.

Convergence Analysis and Theoretical Results

The convergence analysis leverages a recently generalized asynchronous SA theory [YWS25a], establishing almost sure (a.s.) convergence under minimal assumptions. Key results include:

1. Compact Set Convergence

Under mild step-size and asynchrony conditions, the iterates $Q_n$ converge a.s. to a compact, connected subset $Q_f$ of AOE solutions satisfying $f(q) = r^*$. This set is homeomorphic to a convex polyhedron of dimension one less than that of the original solution set, and policies derived from limit points in $Q_f$ are guaranteed to be optimal. (Theorem 1)

2. Uniqueness Under Additional Conditions

Stronger step-size decay and asynchrony conditions coupled with shadowing properties (Hirsch-Benaïm dynamical systems theory) guarantee $Q_n \to q^*$ for a unique, sample path-dependent solution $q^* \in Q_f$. The conditions on $f$ are shown to be both necessary and sufficient for aligning the algorithm's stochastic trajectory with a unique ODE limit. (Theorem 2)

3. Generalization via Monotonicity

By generalizing the class of $f$ (SISTr instead of affine or linear), the algorithm and analysis support a much larger strategy space for average-reward estimation, expanding applicability beyond previous work.

Numerical and Practical Implications

While the paper is theory-centered, several practical implications arise:

• The algorithm is robust to the lack of uniform state-action exploration and can operate effectively when only a subset of state-action pairs receive updates at each time step.
• The separation of holding-time and Q-value estimation accommodates the stochastic nature of actions of varying duration.
• Minimal model-knowledge requirements make the approach applicable in domains with unknown transition dynamics or temporal distributions.

Contrasts and Technical Improvements

• Relaxation of uniqueness: Contradicting earlier work restricted to unichain MDPs/SMDPs, the present approach applies to weakly communicating cases with non-unique solution sets, substantially relaxing prior assumptions.
• Resolution of stability gaps: The analysis addresses previously unproven assertions and incorrect arguments in earlier RVI Q-learning convergence proofs, explicitly applying newly established SA stability theorems.
• Sample-path uniqueness: Under stricter algorithmic conditions, a unique limiting solution is guaranteed for every sample path, a stronger claim than simple set convergence.

Theoretical and Future Directions

The framework and results extend naturally to synchronous (uniform update) settings (with fewer constraints) and to classical MDPs (where holding times are always unity and biases vanish). Future directions suggested include exploring distributed implementations under communication delays, extending monotonicity conditions, and further examination of shadowing and ODE-based convergence for non-Markovian or continuous-time models.

Conclusion

The paper provides a rigorous, flexible, and broadly applicable foundation for average-reward RL in SMDPs, with strong theoretical guarantees under asynchronous update protocols. The generalization of RVI Q-learning to accommodate a vast class of average-reward estimators, combined with mathematically robust convergence arguments, establishes new standards for RL algorithms addressing persistent long-term optimality in stochastic environments with temporally non-uniform transitions.