---
title: Markov Decision Process Formulation
url: https://www.emergentmind.com/topics/markov-decision-process-mdp-formulation
type: topic
---

# Markov Decision Process Formulation

A Markov Decision Process (MDP) provides a rigorous mathematical framework for modeling sequential decision-making in stochastic environments. It describes the interaction of a decision-maker, or agent, with an environment characterized by probabilistic state transitions and reward structures, enabling the formal synthesis and analysis of optimal policies under uncertainty. MDPs form the foundational model for many branches of dynamic optimization, reinforcement learning, and control theory.

## 1. Canonical MDP Structure and Notation

A classic MDP is defined as a tuple $(S, A, P, r, \gamma)$, with the following components:

- **State space $S$:** A finite or Borel-measurable set representing all possible environment configurations.
- **Action space $A$:** A set of feasible control actions, possibly state-dependent with $A_s$ available in $s$.
- **Transition kernels $P(s'|s,a)$:** The probability law for state evolution such that $P(s'|s,a) = \Pr\{X_{t+1} = s' | X_t = s, Y_t = a\}$.
- **(Vector-valued) reward function $r_t(s,a) \in \mathbb{R}^k$:** The immediate (possibly vector-valued) reward when taking action $a$ in $s$ at time $t$; terminal rewards $r_H(s)$ may also be specified for finite horizon problems.
- **Discount factor $\gamma \in (0,1)$:** (optional) Governs the trade-off between immediate and long-run rewards (infinite-horizon discounted MDPs).

Policies $\pi$ are mappings specifying a probability distribution over actions for each state (possibly non-stationary or randomized). Expectations and objectives are formed with respect to these induced Markov chains.

## 2. Occupancy Measures, Linear Programming, and Flow Constraints

For both finite- and infinite-horizon settings, the action of a policy $\pi$ can be algebraically encoded via occupancy measures:

\[
x_t(s,a) = \Pr^\pi[X_t = s, Y_t = a] \qquad (\text{for } t = 1, \ldots, H-1)
\]
\[
x_H(s) = \Pr^\pi[X_H = s]
\]

Occupancy variables must satisfy coupled linear flow-balance equations:

- At $t=1$ (initial occupation):
  \[
  \sum_{a\in A_j} x_1(j,a) = \alpha(j)
  \]
- For $t=1,\ldots,H-2$:
  \[
  \sum_{a\in A_j} x_{t+1}(j,a) = \sum_{s\in S} \sum_{a\in A_s} P_t(j|s,a) x_t(s,a)
  \]
- Terminal:
  \[
  x_H(j) = \sum_{s\in S}\sum_{a\in A_s} P_{H-1}(j|s,a)x_{H-1}(s,a)
  \]
- Nonnegativity:
  \[
  x_t(s,a)\geq 0, \quad x_H(s)\geq 0
  \]

The feasible set of $x$ is the polyhedron $\Omega$ cut out by these (and policy-dependence yields a bijection $\pi \mapsto x^\pi$). This construction is central for both single- and multi-objective MDPs [2502.13697].

## 3. Optimization Formulations: Scalar and Vector-valued Objectives

Classic infinite-horizon, discounted-reward MDPs admit primal and dual linear programming (LP) formulations [2012.09417]:

- **Primal (value-function based):**
  \[
  \min_{v\in\mathbb{R}^{|S|}}\quad \sum_s e_s v_s,\quad \text{s.t.}\ r^a_s + \gamma\sum_t P^a_{st}v_t - v_s \leq 0\;\;\forall\,s,a
  \]

- **Dual (state-action occupation):**
  \[
  \max_{\mu^a \geq 0}\;\sum_{s,a} \mu^a_s r^a_s
  \quad \text{s.t.}\ \sum_{a} (I-\gamma P^a{}^T) \mu^a = e
  \]

- **Multiobjective (vector-valued):** For $k$ objectives, the total expected reward vector is
  \[
  V(\pi) = \sum_{s\in S} \alpha(s) v^\pi(s) \in \mathbb{R}^k
  \]
  Optimality is defined in terms of Pareto efficiency: no other $V(\pi')$ is at least as large in all components and strictly greater in one. The set of achievable vectors is $C(\Omega)$, the image of the feasible polyhedron $\Omega$ under the reward linear map [2502.13697].

- **Entropy-regularized MDPs:** Include additional negative entropy terms yielding soft-max Bellman/Fenchel dual constraints [2012.09417].

The vector LP formulation for a finite-horizon MDP with vector rewards is given by
\[
\text{V-maximize} \quad Cx = \Bigl( \sum_{t, s, a} x_t(s,a) r_t^i(s,a) + \sum_s x_H(s) r_H^i(s) \Bigr)_{i=1}^k,\quad \text{s.t. } x \in \Omega
\]
Pareto-efficient solutions correspond precisely to LP Pareto optima (no $x'\in\Omega$ with $Cx' \geq Cx^*,\;Cx'\neq Cx^*$) [2502.13697].

## 4. Characterization and Enumeration of Optimal Policies

Extreme points of the polyhedron $\Omega$ (i.e., basic feasible solutions of the vector LP) correspond exactly to deterministic policies. Pareto efficiency in policy space coincides with Pareto efficiency in the occupancy variable space. An explicit graph search (ENUMEFFICIENT) traverses the adjacency structure of $\Omega$'s vertices: starting from an initial efficient vertex (found by a scalarization), the algorithm pivots via adjacency (simplex steps), testing each adjacent vertex for efficiency. This process enumerates the complete set of Pareto-efficient deterministic policies [2502.13697].

Given a Pareto-optimal occupancy vector $x^*\in\Omega$, the deterministic policy $\pi^x$ is reconstructed componentwise by normalizing:
\[
q(a|s,\pi^x_t) = \frac{x_t(s,a)}{\sum_{a'}x_t(s,a')}
\]
for each $t$ and $s$.

This algebraic approach allows exhaustive characterization of all efficient deterministic strategies for finite-horizon, multiobjective MDPs, facilitating both completeness and explicit policy synthesis.

## 5. Extensions and Advanced Problem Classes

The LP-based paradigm and occupancy-measure framework extend naturally to extensive classes of MDP variations:

- **Distributionally Robust and Risk-sensitive MDPs:** Models with parameter uncertainty, e.g., unknown transition kernels and random rewards, admit generalized convex or conic programming formulations (SOCP, MISOCP, copositive, or biconvex) under suitable uncertainty sets [2212.08126, 2106.02558, 2209.10187].
- **Expectation and Constraints:** Infinite-horizon expected-total-reward problems with Borel state-action spaces and constraints reduce to infinite-dimensional convex programs over generalized occupation measures, maintaining equivalence between convex-analytic and policy-based optima under mild regularity and Slater-type conditions [1903.08853].
- **Belief-MDPs and Privacy:** Partially observable (e.g., privacy-preserving data sharing) domains are reformulated as belief-space MDPs, with particle-based approximations yielding finite-state tractable surrogates [2602.04262].
- **Vector-valued and Multi-objective Control:** Full characterization of trade-offs among conflicting objectives is afforded by the polyhedral structure of occupancy measures; efficient enumeration of all Pareto-optimal deterministic strategies becomes tractable [2502.13697].
- **Augmented and Exogenous Processes:** Externally driven non-stationarity, e.g., MDPs with exogenous temporal processes, are rigorously addressed via history-augmented state spaces and policy iteration over finite-memory truncations, with explicit bounds on the suboptimality introduced by memory truncation [2305.16056].

## 6. Practical Algorithms and Scalability

Solving large-scale MDPs requires leveraging compressed representations and scalable algorithms:

- **Tensor Decomposition:** For high-dimensional finite MDPs, transition traffic is compressed via CP (CANDECOMP–PARAFAC) decomposition, reducing per-iteration and memory complexity by orders of magnitude and enabling solution of problems with over $10^6$ states [2103.00331].
- **Policy Iteration with Adaptive Enhancements:** Heuristics such as adaptive edge-pruning in policy iteration address sudden non-stationarity or non-Markovian exogenous processes [2304.13922, 2305.16056].
- **Enumeration Algorithms:** Graph traversal and simplex-pivoting enable full enumeration of the efficient deterministic policy set for multiobjective formulations [2502.13697].
- **Entropy Regularization and Risk Measures:** Sophisticated problem classes admit solution via primal-dual convex conic programs, regularized Bellman operators, and recursive convex approximation techniques for risk-sensitive objectives [2502.13697, 2106.02558, 2209.10187].
- **Online and Approximate Scheduling:** Practical low-complexity heuristics that closely track the LP-MDP optimum are derived by mapping predictive models (e.g., AR(1) for channel throughput) onto resource allocation policies [1209.2067].

## 7. Theoretical Foundations and Equivalence Results

Key theoretical guarantees underpin the MDP formulation:

- The occupancy measure mapping is a bijection between regular policies and the flow-balance polyhedron [2502.13697].
- All Pareto-efficient finite-horizon MDP policies are obtainable as vertices (extreme points) of the vector-LP feasible polyhedron.
- In infinite-horizon average or total-reward settings, policy and convex-analytic optima coincide under mild compactness and regularity [1903.08853].
- For robust and risk-averse classes, entropic or risk-aggregation operations (e.g., CVaR, Wasserstein, $\phi$-divergence) yield convex or conic programs with explicit solution structure [2106.02558, 2212.08126, 2209.10187].
- Policy iteration and Bellman recursion structures are preserved under augmented state spaces, external processes, and belief-MDP reductions [2305.16056, 2602.04262].

The mathematical structure of the MDP formulation—spanning stochastic processes, convex geometry, and optimization—enables rigorous synthesis and complete characterization of optimal, and Pareto-optimal, decision strategies across a wide range of domains and theoretical extensions.

Source: https://www.emergentmind.com/topics/markov-decision-process-mdp-formulation