---
title: Average-Cost Optimality Equation (ACOE)
url: https://www.emergentmind.com/topics/average-cost-optimality-equation-acoe
type: topic
---

# Average-Cost Optimality Equation (ACOE)

The average-cost optimality equation (ACOE) is a central object in the theory of Markov decision processes (MDPs) and stochastic control, providing a necessary and sufficient condition for a policy to be optimal with respect to long-run average (per-stage) cost. The ACOE connects dynamic programming with ergodic control, is foundational for the structure and computation of optimal policies, and provides the backbone for applications across queueing, inventory control, mean-field systems, and beyond.

## 1. Formal Statement of the ACOE

Let $X$ be a Borel subset of a Polish space (state space), $A(x) \subset A$ a Borel-action set (possibly noncompact) for each $x \in X$, $c: X \times A \to [0,\infty]$ a one-step cost, and $P(\cdot\,|\,x,a)$ a transition probability kernel on $X$. The average-cost optimality equation is
\[
\lambda + h(x) = \min_{a \in A(x)} \left\{ c(x,a) + \int_X h(y)\,P(dy|x,a) \right\}, \quad \forall x\in X
\]
where $\lambda \in \mathbb R$ is the optimal average cost per unit time, and $h:X \to \mathbb R$ is a "differential" or "relative" value function. The pair $(\lambda,h)$ solves the ACOE under the following properties:
- $h(x)$ is measurable (typically lower-semicontinuous),
- $a \mapsto c(x,a)$ is inf-compact or $c$ is $K$-inf-compact over $X \times A$,
- The minimum is attained, so measurable selectors $\varphi(x)\in \arg\min_{a\in A(x)}\{\cdots\}$ exist,
- The integral $\int |h(y)|\,P(dy|x,a)$ is finite when $a\in A(x)$.

The ACOE governs the structure of stationary deterministic optimal policies and provides the threshold between Bellman-type optimality inequalities and actual equations. Solutions $(\lambda,h)$ are unique up to an additive constant in $h$ [2412.01594].

## 2. Sufficient Conditions and Modern Existence Results

Contemporary results systematically weaken classical requirements on compactness, continuity, and uniform integrability. Key conditions ensuring the validity of the ACOE (see especially [2412.01594], [1202.4122], [1609.08252], [1711.06803]) are:

- **Continuity/Compactness:**  
  - *Weak model (W\*)*: $c$ is $K$-inf-compact on $X\times A$, $P(\cdot|x,a)$ is weakly continuous in $(x,a)$.
  - *Setwise (S\*)*: For each $x$, $a\mapsto c(x,a)$ is inf-compact; $P(B|x,\cdot)$ is setwise continuous for all $B$.

- **Relative Value Boundedness:**  
  Fix any sequence $\{\alpha_n\uparrow1\}$ (discount factors):
  - $m_n := \inf_{x\in X} v_{\alpha_n}(x)$ for the discounted value.
  - $u_{\alpha_n}(x) := v_{\alpha_n}(x) - m_n$.
  - $w_* := \liminf_{n\to\infty} (1-\alpha_n) m_n < \infty$.
  - Boundedness condition $B'$: $\liminf_{n\to\infty} u_{\alpha_n}(x) < \infty$ for each $x$.
  - Stronger $B$: $\sup_{x,n} u_{\alpha_n}(x) < \infty$.

- **Equicontinuity/Integrability:**  
  To strengthen from the optimality inequality to the full ACOE, require one of:
  - (EC) Uniform equicontinuity and a uniform integrable envelope for $u_{\alpha_n}$.
  - (LEC) Lower-semi-equicontinuity in $x$, pointwise limit existence of $u_{\alpha_n}(x)$, and uniform integrability w.r.t. $P(\cdot|x,a)$ in $(x,a)$.

Under W\* or S\*, $B'$, and LEC, the ACOE is satisfied; $(\lambda,h)$ with $h(x):= \liminf_{n\to\infty}u_{\alpha_n}(x)$ is measurable, and policies selecting minimizers solve the average-cost control problem [2412.01594].

## 3. Derivation from Discounted to Average Cost, and Proof Techniques

The transition from the discounted-cost optimality equation to the ACOE is critical:
- The value function $v_\alpha(x)$ for $0<\alpha<1$ solves
  \[
  v_\alpha(x) = \min_a \left\{ c(x,a) + \alpha \int v_\alpha(y) P(dy|x,a) \right\}
  \]
- Define $u_\alpha(x) := v_\alpha(x) - m_\alpha$, $w_\alpha := (1-\alpha)m_\alpha$.
- Under the boundedness assumptions, $\{u_{\alpha_n}\}$ is pointwise bounded; diagonal/lower-semicontinuity arguments yield $h(x) = \liminf_{n\to\infty} u_{\alpha_n}(x)$.
- Weakly continuous/inf-compact conditions permit passage of $\liminf$ through $P$ and $\min$, so the limiting function satisfies
  \[
  w_* + h(x) = \min_{a\in A(x)} \left\{ c(x,a) + \int h(y) P(dy|x,a) \right\}
  \]
- Measurable selection yields the existence of deterministic stationary minimizing controls [2412.01594], [1202.4122], [1711.06803].

Alternate approaches, such as occupation measure convex-analytic methods (employing ergodic occupation measures), Poisson/relative value iteration (RVI) schemes, and reduction to discounted MDPs (e.g., HV–AG transformation), also appear as foundational derivations [1902.01048], [1507.00664], [1711.06803]. The vanishing discount approach remains the standard, but split-chain constructions or Lyapunov drift stability hypotheses allow further generalization.

## 4. Policy Structure and Uniqueness

The ACOE under stated conditions admits solutions where for each $x$,
\[
A^*(x) = \arg\min_{a \in A(x)} \left\{ c(x,a) + \int h(y)P(dy|x,a) \right\}
\]
A measurable selector $\varphi(x): X \to A$ choosing a minimizer at each $x$ defines a deterministic stationary policy that is average-cost optimal. Any such policy solves both the average-cost optimality inequality and the equality, and achieves $\lambda$ from every initial state.

The solution $(\lambda,h)$ to the ACOE is unique up to a constant shift in $h$; i.e., if $(\lambda,h)$ and $(\lambda',h')$ both solve the ACOE and $h,h'$ are bounded below (e.g., lower semicontinuous), then $\lambda=\lambda'$ and $h' = h + C$. This is a generalization of the classical uniqueness theorem for the Bellman equation in ergodic control [2412.01594].

## 5. Comparison with Classical and Alternative Conditions

Classically, average-cost optimality analysis relied on:
- Communicating or unichain structure for finite-state models,
- Lyapunov (drift) conditions to ensure positive recurrence,
- Uniform compactness of action sets, and strong Feller continuity of transitions,
- Uniform equicontinuity of discounted value functions.

Recent advances weaken these requirements, replacing them by:
- $K$-inf-compactness or inf-compactness of costs rather than action set compactness,
- Weak or setwise continuity instead of strong-Feller continuity,
- One-sided boundedness in the limit of discounted relative values,
- Lower-semi-equicontinuity and uniform integrability (LEC) instead of full equicontinuity.

This encompasses a broader range of stochastic control models, such as queueing or inventory systems with noncompact action sets and weak continuity properties, which often fall outside the reach of classical assumptions [2412.01594], [1609.08252], [1202.4122].

## 6. Illustrative Examples and Applications

The wide applicability of the ACOE is demonstrated by explicit examples in [2412.01594]:
- **Single-Action Indicator–Cost:** $X=[0,1]$, $A=\{a_0\}$, $P(0|x,a_0)=1$, $c(x,a_0)=\mathbf{1}_{x\neq0}$. The relative value function $h(x)=\mathbf{1}_{x\neq0}$ is lower-semicontinuous, and $\lambda=0$.
- **Dirichlet–Cost MDP:** $X=[0,1]$, $A=\{a_1\}$, $P(0|x,a_1)=1$, $c(x,a_1)=D(x)$ (Dirichlet function). The bias function $h(x)=D(x)$ fails to be lower-semicontinuous but the ACOE still holds under the weaker integrability and limit assumptions.

Broader applications include the derivation of optimal $(s,S)$ policies in inventory systems, mean-field game limit problems, and models with state- or action-dependent control constraints. Computationally, the ACOE provides the foundation for value iteration, policy iteration, and linear programming methods for average-cost control [2412.01594], [1609.08252], [1902.01048].

## 7. Impact and Extensions

The ACOE is essential for the theoretical and computational treatment of Markov control problems under the average-cost criterion. Its solution structure and existence theory underpin modern direct algorithms and facilitate the analysis of ergodic control for noncompact, weakly continuous, and complex stochastic dynamic models. The recent generalizations to weaker boundedness and continuity, as established in [2412.01594], have expanded its reach to previously intractable classes of queueing, inventory, and stochastic network models. Furthermore, its connections with ergodic occupation measures, split-chain Poisson equations, and mean-field limits continue to drive advances in both theory and large-system applications.

Source: https://www.emergentmind.com/topics/average-cost-optimality-equation-acoe