Average-Cost Optimality Equation (ACOE)

• ACOE is a fundamental condition in Markov decision processes that defines optimality by balancing immediate costs with future value through a dynamic programming equation.
• It is derived from discounted-cost models by taking the limit of relative value functions under conditions like weak continuity and inf-compact cost criteria.
• ACOE underpins applications in queueing, inventory, and mean-field systems, enabling the computation of stationary deterministic optimal policies.

The average-cost optimality equation (ACOE) is a central object in the theory of Markov decision processes (MDPs) and stochastic control, providing a necessary and sufficient condition for a policy to be optimal with respect to long-run average (per-stage) cost. The ACOE connects dynamic programming with ergodic control, is foundational for the structure and computation of optimal policies, and provides the backbone for applications across queueing, inventory control, mean-field systems, and beyond.

1. Formal Statement of the ACOE

Let $X$ be a Borel subset of a Polish space (state space), $A(x) \subset A$ a Borel-action set (possibly noncompact) for each $x \in X$, $c: X \times A \to [0,\infty]$ a one-step cost, and $P(\cdot\,|\,x,a)$ a transition probability kernel on $X$. The average-cost optimality equation is

$$\lambda + h(x) = \min_{a \in A(x)} \left\{ c(x,a) + \int_X h(y)\,P(dy|x,a) \right\}, \quad \forall x\in X$$

where $\lambda \in \mathbb R$ is the optimal average cost per unit time, and $h:X \to \mathbb R$ is a "differential" or "relative" value function. The pair $(\lambda,h)$ solves the ACOE under the following properties:

• $h(x)$ is measurable (typically lower-semicontinuous),
• $a \mapsto c(x,a)$ is inf-compact or $c$ is $K$-inf-compact over $X \times A$,
• The minimum is attained, so measurable selectors $\varphi(x)\in \arg\min_{a\in A(x)}\{\cdots\}$ exist,
• The integral $\int |h(y)|\,P(dy|x,a)$ is finite when $a\in A(x)$.

The ACOE governs the structure of stationary deterministic optimal policies and provides the threshold between Bellman-type optimality inequalities and actual equations. Solutions $(\lambda,h)$ are unique up to an additive constant in $h$ (Feinberg et al., 2024).

2. Sufficient Conditions and Modern Existence Results

Contemporary results systematically weaken classical requirements on compactness, continuity, and uniform integrability. Key conditions ensuring the validity of the ACOE (see especially (Feinberg et al., 2024, Feinberg et al., 2012, Feinberg et al., 2016, Feinberg et al., 2017)) are:

• Continuity/Compactness:
  • Weak model (W*): $c$ is $K$-inf-compact on $X\times A$, $P(\cdot|x,a)$ is weakly continuous in $(x,a)$.
  • Setwise (S*): For each $x$, $a\mapsto c(x,a)$ is inf-compact; $P(B|x,\cdot)$ is setwise continuous for all $B$.
• Relative Value Boundedness:
  • $m_n := \inf_{x\in X} v_{\alpha_n}(x)$ for the discounted value.
  • $u_{\alpha_n}(x) := v_{\alpha_n}(x) - m_n$.
  • $$w_* := \liminf_{n\to\infty} (1-\alpha_n) m_n < \infty$$.
  • Boundedness condition $B'$: $\liminf_{n\to\infty} u_{\alpha_n}(x) < \infty$ for each $x$.
  • Stronger $B$: $\sup_{x,n} u_{\alpha_n}(x) < \infty$.
• Equicontinuity/Integrability:
  • (EC) Uniform equicontinuity and a uniform integrable envelope for $u_{\alpha_n}$.
  • (LEC) Lower-semi-equicontinuity in $x$, pointwise limit existence of $u_{\alpha_n}(x)$, and uniform integrability w.r.t. $P(\cdot|x,a)$ in $(x,a)$.

Under W* or S*, $B'$, and LEC, the ACOE is satisfied; $(\lambda,h)$ with $h(x):= \liminf_{n\to\infty}u_{\alpha_n}(x)$ is measurable, and policies selecting minimizers solve the average-cost control problem (Feinberg et al., 2024).

3. Derivation from Discounted to Average Cost, and Proof Techniques

The transition from the discounted-cost optimality equation to the ACOE is critical:

• The value function $v_\alpha(x)$ for $0<\alpha<1$ solves

$$v_\alpha(x) = \min_a \left\{ c(x,a) + \alpha \int v_\alpha(y) P(dy|x,a) \right\}$$

• Define $u_\alpha(x) := v_\alpha(x) - m_\alpha$, $w_\alpha := (1-\alpha)m_\alpha$.
• Under the boundedness assumptions, $\{u_{\alpha_n}\}$ is pointwise bounded; diagonal/lower-semicontinuity arguments yield $h(x) = \liminf_{n\to\infty} u_{\alpha_n}(x)$.
• Weakly continuous/inf-compact conditions permit passage of $\liminf$ through $P$ and $\min$, so the limiting function satisfies

$$w_* + h(x) = \min_{a\in A(x)} \left\{ c(x,a) + \int h(y) P(dy|x,a) \right\}$$

• Measurable selection yields the existence of deterministic stationary minimizing controls (Feinberg et al., 2024, Feinberg et al., 2012, Feinberg et al., 2017).

Alternate approaches, such as occupation measure convex-analytic methods (employing ergodic occupation measures), Poisson/relative value iteration (RVI) schemes, and reduction to discounted MDPs (e.g., HV–AG transformation), also appear as foundational derivations (Arapostathis et al., 2019, Feinberg et al., 2015, Feinberg et al., 2017). The vanishing discount approach remains the standard, but split-chain constructions or Lyapunov drift stability hypotheses allow further generalization.

4. Policy Structure and Uniqueness

The ACOE under stated conditions admits solutions where for each $x$,

$$A^*(x) = \arg\min_{a \in A(x)} \left\{ c(x,a) + \int h(y)P(dy|x,a) \right\}$$

A measurable selector $\varphi(x): X \to A$ choosing a minimizer at each $x$ defines a deterministic stationary policy that is average-cost optimal. Any such policy solves both the average-cost optimality inequality and the equality, and achieves $\lambda$ from every initial state.

The solution $(\lambda,h)$ to the ACOE is unique up to a constant shift in $h$; i.e., if $(\lambda,h)$ and $(\lambda',h')$ both solve the ACOE and $h,h'$ are bounded below (e.g., lower semicontinuous), then $\lambda=\lambda'$ and $h' = h + C$. This is a generalization of the classical uniqueness theorem for the Bellman equation in ergodic control (Feinberg et al., 2024).

5. Comparison with Classical and Alternative Conditions

Classically, average-cost optimality analysis relied on:

• Communicating or unichain structure for finite-state models,
• Lyapunov (drift) conditions to ensure positive recurrence,
• Uniform compactness of action sets, and strong Feller continuity of transitions,
• Uniform equicontinuity of discounted value functions.

Recent advances weaken these requirements, replacing them by:

• $K$-inf-compactness or inf-compactness of costs rather than action set compactness,
• Weak or setwise continuity instead of strong-Feller continuity,
• One-sided boundedness in the limit of discounted relative values,
• Lower-semi-equicontinuity and uniform integrability (LEC) instead of full equicontinuity.

This encompasses a broader range of stochastic control models, such as queueing or inventory systems with noncompact action sets and weak continuity properties, which often fall outside the reach of classical assumptions (Feinberg et al., 2024, Feinberg et al., 2016, Feinberg et al., 2012).

6. Illustrative Examples and Applications

The wide applicability of the ACOE is demonstrated by explicit examples in (Feinberg et al., 2024):

• Single-Action Indicator–Cost: $X=[0,1]$, $A=\{a_0\}$, $P(0|x,a_0)=1$, $c(x,a_0)=\mathbf{1}_{x\neq0}$. The relative value function $h(x)=\mathbf{1}_{x\neq0}$ is lower-semicontinuous, and $\lambda=0$.
• Dirichlet–Cost MDP: $X=[0,1]$, $A=\{a_1\}$, $P(0|x,a_1)=1$, $c(x,a_1)=D(x)$ (Dirichlet function). The bias function $h(x)=D(x)$ fails to be lower-semicontinuous but the ACOE still holds under the weaker integrability and limit assumptions.

Broader applications include the derivation of optimal $(s,S)$ policies in inventory systems, mean-field game limit problems, and models with state- or action-dependent control constraints. Computationally, the ACOE provides the foundation for value iteration, policy iteration, and linear programming methods for average-cost control (Feinberg et al., 2024, Feinberg et al., 2016, Arapostathis et al., 2019).

7. Impact and Extensions

The ACOE is essential for the theoretical and computational treatment of Markov control problems under the average-cost criterion. Its solution structure and existence theory underpin modern direct algorithms and facilitate the analysis of ergodic control for noncompact, weakly continuous, and complex stochastic dynamic models. The recent generalizations to weaker boundedness and continuity, as established in (Feinberg et al., 2024), have expanded its reach to previously intractable classes of queueing, inventory, and stochastic network models. Furthermore, its connections with ergodic occupation measures, split-chain Poisson equations, and mean-field limits continue to drive advances in both theory and large-system applications.