---
title: Average-Reward Reinforcement Learning
url: https://www.emergentmind.com/topics/average-reward-reinforcement-learning
type: topic
---

# Average-Reward Reinforcement Learning

Average-reward reinforcement learning (AR-RL) is a foundational paradigm for sequential decision-making under uncertainty, focusing on the optimization of the long-term average reward rate in Markov decision processes (MDPs). Unlike discounted-reward RL, which prioritizes short-term returns through a discount factor, AR-RL treats rewards at all time steps equally, making it the preferred criterion for continuing control, steady-state system optimization, and settings without an intrinsic notion of discounting. This article surveys the theoretical foundations, algorithmic methodologies, statistical guarantees, and current research trends, with an emphasis on rigorous developments and open problems.

## 1. Formal Foundations of the Average-Reward Criterion

AR-RL considers an infinite-horizon MDP \( (S, A, P, r) \) where \(S\) is the state space, \(A\) the action space, \(P(s'|s,a)\) the transition kernel, and \(r: S \times A \rightarrow [0,1]\) the reward function. For a stationary policy \( \pi: S \rightarrow \Delta(A) \), the long-run average reward (gain) is defined as:
\[
\rho^\pi = \lim_{T \rightarrow \infty} \frac{1}{T} \mathbb{E}_\pi \left[ \sum_{t=0}^{T-1} r(s_t, a_t) \right]
\]
Under mild assumptions such as weakly communicating or unichain structure, \( \rho^\pi \) is independent of the starting state.

The bias (or relative value) function \( h^\pi: S \rightarrow \mathbb{R} \) is:
\[
h^\pi(s) = \mathbb{E}_\pi \left[ \sum_{t=0}^{\infty} (r(s_t, a_t) - \rho^\pi) \,\big|\, s_0 = s \right]
\]
and satisfies the Poisson (Bellman) equation
\[
\rho^\pi + h^\pi(s) = \mathbb{E}_{a \sim \pi(s), s' \sim P(s,a)}[r(s,a) + h^\pi(s')]
\]
The optimal average reward \( \rho^* = \max_\pi \rho^\pi \), and an optimal policy \( \pi^* \) achieves this value. The bias span \( \mathrm{sp}(h^*) = \max_s h^*(s) - \min_s h^*(s) \) plays a central role in the complexity of average-reward MDPs [2306.16394].

## 2. Algorithmic Principles and Model-Free AR-RL Methods

Contemporary AR-RL algorithms aim to solve for both the optimal gain \( \rho^* \) and the bias \( h^* \) or corresponding Q-functions, using only sampled interactions with the environment.

### 2.1 Reference-Advantage Decomposition and UCB-AVG

Recent advances leverage a reference-advantage decomposition to reduce AR-MDPs to a family of nearly undiscounted MDPs (\( \gamma \rightarrow 1 \)), enabling variance reduction and efficient confidence computation. The UCB-AVG algorithm [2306.16394] structures learning into epochs, uses sparse graphs to estimate bias differences, and projects value functions into polytopes constrained by bias span. This yields a regret bound of
\[
\widetilde{O}(S^5 A^2\,\mathrm{sp}(h^*)\,\sqrt{T})
\]
with near-optimal dependence on time and fundamental hardness parameter \( \mathrm{sp}(h^*) \).

### 2.2 Relative Value Iteration (RVI) Q-Learning

RVI Q-learning and its variants generalize Q-learning by simultaneously updating state-action values and a running estimate of the average reward. The update for Q-values and gain (average reward) in the tabular setting is:
\[
Q_{t+1}(s,a) \leftarrow Q_t(s,a) + \alpha_t \left( r_t - \rho_t + \max_{a'} Q_t(s',a') - Q_t(s,a) \right)
\]
\[
\rho_{t+1} \leftarrow \rho_t + \beta_t \left( r_t - \rho_t + \max_{a'} Q_t(s',a') - \max_{a} Q_t(s,a) \right)
\]
Convergence to \( (\rho^*, h^*) \) holds under standard conditions (diminishing step-sizes, sufficient exploration, ergodicity) [2112.13093, 2305.10504, 2409.03915, 2512.06218].

### 2.3 Function Approximation and Deep AR-RL

For large or continuous state-action spaces, AR-RL is extended to function approximation using neural networks and feature-based representations. Techniques like implicit TD(\(\lambda\)) [2510.06149] provide stable updates, and actor-critic methods with average-reward critics have been developed.

Notably, algorithms such as Average Reward Off-policy Soft Actor Critic (ARO-SAC) [2501.06700] and entropy-regularized variants (ASAC/ASQL) [2501.09080] adapt expected bellman equations and policy-improvement to the average-reward and entropy-regularized contexts, sometimes leveraging eigenvector-based frameworks [2501.09770].

## 3. Statistical Guarantees and Regret/Sample Complexity

The statistical understanding of AR-RL focuses on minimizing regret or achieving a policy with gain within \( \epsilon \) of optimal.

- For finite-space, model-free online AR-RL, regret of order \( \widetilde{O}(S^5A^2 \mathrm{sp}(h^*) \sqrt{T}) \) is achievable, matching known lower bounds in \(T\) up to polynomial factors in S, A [2306.16394].
- In the presence of generative models (simulators), sample complexity bounds of the form \( \widetilde{O}(SA\,\mathrm{sp}^2(h^*)/\epsilon^2 + S^2A\,\mathrm{sp}(h^*)/\epsilon) \) are proven, and precise characterizations of the statistical hardness via mixing times or bias span are established [2306.16394, 2505.10007].
- In continuous spaces, adaptive discretization and zooming algorithms achieve regret rates scaling as \( \widetilde{O}(T^{1-1/d_{\text{eff}}}) \), where \( d_{\text{eff}} \) captures both cover dimension and a zooming dimension reflecting problem-specific geometry [2410.19919].

Distributional robustness under KL, total variation, and \( f \)-divergence uncertainty sets is also treated with matching sample complexity under uniform ergodicity [2505.10007].

## 4. Extensions: Risk, Temporal Abstraction, and Specification

### 4.1 Risk-Aware and Subtask-Driven AR-RL

The Reward-Extended Differential (RED) framework demonstrates that the structural property of AR-MDPs—where a single TD error drives both value and gain estimates—extends to simultaneous, online optimization of multiple scalar subtasks, including conditional value-at-risk (CVaR) and quantiles, without state augmentation [2410.10578].

### 4.2 Options and Semi-Markov Models

AR-RL extends naturally to the options framework (temporal abstraction) and semi-Markov decision processes (SMDPs). Generalized RVI Q-learning algorithms with length-normalized updates converge under mild assumptions, supporting both inter- and intra-option methods [2110.13855, 2409.03915, 2512.06218].

### 4.3 Formal Specification and Mean-Payoff Objectives

AR-RL provides the proper formalism for optimizing mean-payoff and ω-regular (infinite-trace) specifications in continuing tasks. Recent work introduces model-free reductions from absolute liveness ω-regular specifications to average-reward MDPs, yielding the first practical RL methods for maximizing satisfaction probability in the continuing (non-episodic) regime and enabling lexicographic multi-objective AR-RL [2505.15693]. Temporal-logic-based reward shaping compatible with AR-RL also enables safe and efficient knowledge-driven learning [2007.01498].

## 5. Trust Region, Policy Improvement, and Performance Bounds

The lack of contraction for the average-reward Bellman operator precludes direct application of discounted-RL trust-region theory. However, recent advances establish performance-difference and trust-region bounds for AR-RL using tools such as Kemeny's constant, policy divergence measures, and average-reward-specific advantage functions [2106.03442, 2106.07329]. Algorithms such as Average Policy Optimization (APO) and Average-Reward TRPO (ATRPO) adapt trust-region updates for monotonic policy improvement in the AR setting.

\[
\rho(\pi') - \rho(\pi) \geq L_\pi(\pi') - 2\epsilon(\kappa_{\pi'}-1)\, \bar{D}_{\mathrm{TV}}(\pi' \| \pi)
\]
where \(L_\pi(\pi')\) is a surrogate, \(\epsilon\) bounds the worst-case advantage, and \(\kappa_{\pi'}\) is Kemeny's constant [2106.07329, 2106.03442].

## 6. Practical Applications and Empirical Trends

AR-RL aligns uniquely with tasks in wireless resource management, network admission control, robotics, and large-scale infrastructure systems, where maximization of steady-state throughput or long-run performance underpins operational objectives [2501.06700, 2112.13093]. Empirical studies verify that AR-RL algorithms (e.g., ARO-SAC) outperform their discounted counterparts on long-horizon or non-episodic tasks, often by significant margins (e.g., ≈15% higher throughput in wireless resource management [2501.06700]). Moreover, techniques such as entropy regularization and eigenvector-based policy evaluation have enabled robust and scalable average-reward learning in classic control domains [2501.09080, 2501.09770].

## 7. Open Problems and Future Directions

Research continues on:

- Tightening regret and sample complexity bounds, including sharp dependence on bias span, mixing time, or geometric structure [2306.16394, 2410.19919].
- Extension to function approximation and generalization: translating model-free, variance-reduced and bias-controlled techniques to deep RL [2306.16394, 2501.09080].
- Algorithmic advances in risk-aware AR-RL for general coherent risk measures such as CVaR and quantile criteria [2410.10578].
- Robust AR-RL under broader uncertainty and adversarial conditions, including sample-efficient, distributionally robust approaches [2305.10504, 2505.10007].
- Potential-based reward shaping, specification-driven reward design, and integration with formal methods [2007.01498, 2505.15693].
- Efficient planning and learning in high-dimensional and metric state–action spaces, potentially leveraging structure-aware discretization, compression, and manifold learning [2410.19919].
- Generalization to partial observability, multi-agent systems, and settings with weaker communication or mixing assumptions.

---
In summary, average-reward reinforcement learning provides a mathematically rigorous and operationally natural framework for sequential control in continuing environments. Recent theoretical and algorithmic progress offers efficient, robust, and scalable methods, matching or approaching optimal complexity, and enabling principled solutions to a growing range of long-horizon, specification-driven, and risk-sensitive tasks.

Source: https://www.emergentmind.com/topics/average-reward-reinforcement-learning