---
title: State-Action Occupancy Measure
url: https://www.emergentmind.com/topics/state-action-occupancy-measure-saom
type: topic
---

# State-Action Occupancy Measure

The state-action occupancy measure (SAOM) is a foundational object in the analysis and algorithmic design of Markov decision processes (MDPs) and reinforcement learning (RL). Defined as the expected (possibly discounted) visitation measure over state-action pairs under a policy, the SAOM enables convex-analytic, geometric, and probabilistic analyses of long-term behaviors, policy evaluation, and policy optimization across classical MDPs, general-utility RL frameworks, non-Markovian policy settings, and recent advances in RL for preference optimization and large-scale function approximation.

## 1. Formal Definition and Characterizations

The SAOM quantifies the (discounted or undiscounted) expected visitation frequency of each state-action pair under a policy. In a $\gamma$-discounted MDP $(\mathcal{S},\mathcal{A},P,\rho,\gamma)$ with stationary policy $\pi$, the occupancy measure is
\[
\lambda^\pi(s,a) = (1-\gamma) \sum_{t=0}^\infty \gamma^t \Pr_\pi(s_t=s,a_t=a)
\]
or more generally,
\[
\mu^\pi_\nu(B) = \mathbb{E}_\nu^\pi\Bigl[\sum_{t=0}^{T_A-1} \mathbf{1}_{(X_t,A_t)\in B}\Bigr]
\]
for absorbing MDPs, where $T_A$ is the absorption time and $B\subset X\times A$ is any measurable set [2309.07059].

The measure satisfies a system of linear characteristic (or "flow-balance") equations. For absorbing MDPs, the balance equation is
\[
\mu^\pi_\nu(dx,da) = \nu(dx)\, \pi(da|x) + \int_{X\times A}\mu^\pi_\nu(dy,db)\, Q(dx|y,b)\,\pi(da|x)
\]
where $Q(\cdot|x,a)$ is the transition kernel [2309.07059]. For the standard discounted setting,
\[
\lambda^\pi(s,a) = (1-\gamma)\rho(s)\pi(a|s) + \gamma \sum_{s',a'} \lambda^\pi(s',a') P(s|s',a') \pi(a|s)
\]
[2306.01854, 2410.04108].

For non-Markovian (history-dependent) policies $\pi$, $\mu^\pi_\gamma$ is well-defined for all Borel subsets, and any such policy has a Markovian equivalent; i.e., for any $\pi$, there exists a Markov policy $\tilde\pi$ such that $\mu^\pi_\gamma = \mu^{\tilde\pi}_\gamma$ [2205.13950]. In finite spaces, $\tilde{\pi}(a|s) = \lambda^\pi(s,a)/\lambda^\pi(s)$.

## 2. Theoretical Role in RL and MDP Analysis

The SAOM encapsulates all information relevant to long-term returns and is the sufficient statistic for any objective function that depends only on aggregated visit frequencies (e.g., expected return, constraints, pure exploration, or divergence to expert distributions) [2306.01854, 2410.04108]. In the convex-analytic treatment of absorbing MDPs, the set of occupation measures provides a complete description of the feasible space for optimization. In the general-utility RL setting, objectives can be arbitrary functions $F$ of $\lambda^\pi$, including non-linear and non-additive functionals, reducing policy optimization to optimization over occupancy measures [2410.04108, 2306.01854].

The occupation measure set satisfies compactness properties under uniform absorbency and regularity conditions. Pathological "phantom measures"—solutions to the linear equations not arising from any policy—appear exactly when nonzero invariant measures exist for the substochastic kernel outside the absorbing set. Absence of such invariant measures ensures all solutions are true occupation measures. Uniform absorbency (the hitting-time tail control) is necessary and sufficient for compactness of the set of occupation measures [2309.07059].

## 3. SAOM in Policy Optimization and Advanced Objectives

In modern RL with general utilities, policy optimization is formulated as
\[
\max_{\theta} F(\lambda^{\pi_\theta})
\]
where $F$ may encode imitation (e.g., divergence to expert), risk preferences, exploration bonuses, or safety constraints [2410.04108, 2306.01854].

Gradient-based methods employ the relationship
\[
\nabla_\theta F(\lambda^{\pi_\theta}) = [\nabla_\theta \lambda^{\pi_\theta}]^\top \nabla_\lambda F(\lambda^{\pi_\theta})
\]
with $\nabla_\theta \lambda^{\pi_\theta}$ estimated via likelihood ratio and $\nabla_\lambda F$ computed for the chosen utility. In large state-action spaces, $\lambda^{\pi_\theta}$ is estimated with function approximation (e.g., linear in features or parameterized densities). Maximum likelihood estimation (MLE) of the marginal state distribution, using samples from $d^\pi$, ensures high-fidelity occupancy approximation with sample complexity scaling in the feature dimension, not the state-action space size [2410.04108].

Variance reduction, normalization, and multi-step or feature-based estimators are employed to reduce the stochastic error in occupancy and gradient estimates, as in single-loop N-VR-PG and PG-OMA algorithms. These approaches achieve strong sample complexity guarantees for first-order optimality and, under concavity and sufficient policy representation, for global optimality [2410.04108, 2306.01854].

## 4. Geometry, Exploration, and Optimal Transport Analyses

The SAOM induces a geometric structure on the policy space, enabling the use of metrics such as Wasserstein distance for trajectory analysis [2402.09113]. In this framework, the sequence of policies generated by an RL algorithm traces a trajectory in the space of occupation measures. Quantitative exploration diagnostics include:

- **Effort of Sequential Learning (ESL):**
\[
\mathrm{ESL} = \frac{\sum_{t=0}^{T-1} W_c(D_t,D_{t+1})}{W_c(D_0,d^{\pi^*})}
\]
measuring the path length in SAOM-space relative to the geodesic [2402.09113].

- **Optimal Movement Ratio (OMR):**
\[
\mathrm{OMR} = \frac{\sum_{t=0}^{T-1} \Delta R_t}{\sum_{t=0}^{T-1} W_c(D_t,D_{t+1})}
\]
quantifying the fraction of trajectory movement that directly reduces regret [2402.09113].

These metrics offer insight into exploration efficiency, learning "wandering," and policy improvement, and their empirical behavior correlates with algorithmic choices and task complexity.

## 5. Advanced Uses: Preference Learning, Intrinsic Motivation, and Non-Markovian Policies

In preference-based and multi-turn RL settings, e.g., direct preference optimization (DPO and DMPO), SAOMs replace per-state policy constraints by global occupancy-based KL constraints, yielding closed-form optima where the partition function is state-independent and cancels across trajectory comparisons. This eliminates length bias in Bradley-Terry models for trajectory preferences. The occupancy-form KL also provides robustness to compounding error in behavioral cloning and incorporates expert-like coverage globally [2406.14868].

In intrinsic motivation and occupancy maximization frameworks, the agent seeks to maximize path or SAOM entropy, producing rich, exploratory, and goal-directed behaviors without external rewards. The Bellman-style equations tie the entropy-rewarded value to the expected visitation distribution, and computational approaches (e.g., entropic value iteration) yield convergence to occupancy-maximizing policies [2205.10316].

A key theoretical result is that for any history-dependent (non-Markovian) policy, there exists a memoryless Markovian policy with identical occupancy. Thus, all SAOM-based theorems applicable to stationary policies extend directly to general policies, streamlining analysis of replay, off-policy learning, and blended data-generation processes [2205.13950].

## 6. Practical Implications and Pathologies

Occupation measure-based representations form the basis for scalable RL algorithms, especially when the Bellman structure breaks in general-utility or non-linear scenarios. Compactness results guarantee the feasibility and convergence of LP-based, gradient-based, and mixture-model-based methods; phantom measures are precluded by the absence of nontrivial invariant components [2309.07059].

Estimation of SAOMs is central in both tabular and function-approximation regimes. Empirical studies confirm that sample complexity and approximation accuracy depend on the dimension of the function approximator and the quality of the underlying occupancy estimation procedure, not directly on the size of the underlying state-action space [2410.04108, 2306.01854].

Analysis of pathologies demonstrates that without tight control (e.g., via uniform absorbency or discounting), families of occupation measures can escape compactness or include spurious non-policy distributions. The relevant compactness and absence-of-phantoms criteria are now precisely characterized [2309.07059].

## 7. Summary Table: Key Definitions and Properties

| Concept                                      | Formal Definition / Equation                                                                         | Reference         |
|-----------------------------------------------|------------------------------------------------------------------------------------------------------|-------------------|
| Discounted Occupancy (SAOM)                   | $\lambda^\pi(s,a) = (1-\gamma)\sum_{t=0}^\infty\gamma^t\Pr_\pi(s_t=s,a_t=a)$                         | [2410.04108]      |
| Absorbing MDP Occupation Measure              | $\mu^\pi_\nu(B) = \mathbb{E}_\nu^\pi[\sum_{t=0}^{T_A-1} \mathbf{1}_{(X_t,A_t)\in B}]$                | [2309.07059]      |
| Occupancy-Measure Balance Equation            | $\mu^\pi_\nu(dx,da) = \nu(dx)\pi(da|x) + \int \mu^\pi_\nu(dy,db) Q(dx|y,b)\pi(da|x)$                 | [2309.07059]      |
| Markovian Equivalence for Any Policy          | $\exists\,\tilde\pi:\ \mu^\pi_\gamma = \mu^{\tilde\pi}_\gamma$; $\tilde\pi(a|s)=\lambda^\pi(s,a)/\lambda^\pi(s)$ | [2205.13950]      |
| ESL (Effort of Sequential Learning)           | $\mathrm{ESL} = \sum_{t=0}^{T-1}W_c(D_t,D_{t+1}) / W_c(D_0,d^{\pi^*})$                               | [2402.09113]      |
| Occupancy-based Preference-Optimized KL       | $D_{KL}(d^\pi(s,a)\|d^{\pi_{\rm ref}}(s,a))$, yielding global norm $Z$                               | [2406.14868]      |
| Function Approximation-based SAOM Estimation  | $\hat\lambda(s,a) = p_\omega(s)\pi(a|s)$, $p_\omega$ via MLE on $d^\pi$ samples                      | [2410.04108]      |

## References

- "Absorbing Markov Decision Processes" [2309.07059]
- "How Does Your RL Agent Explore? An Optimal Transport Analysis of Occupancy Measure Trajectories" [2402.09113]
- "Non-Markovian policies occupancy measures" [2205.13950]
- "Complex behavior from intrinsic motivation to occupy action-state path space" [2205.10316]
- "Towards Scalable General Utility Reinforcement Learning: Occupancy Approximation, Sample Complexity and Global Optimality" [2410.04108]
- "Reinforcement Learning with General Utilities: Simpler Variance Reduction and Large State-Action Space" [2306.01854]
- "Direct Multi-Turn Preference Optimization for Language Agents" [2406.14868]

Source: https://www.emergentmind.com/topics/state-action-occupancy-measure-saom