Constrained Markov Decision Process

• CMDP is a stochastic control framework that optimizes rewards while imposing explicit constraints on costs and state occupancy.
• It employs methods like extreme point reduction, convex optimization, and concave envelope approaches to tackle various reward structures and continuous actions.
• Practical applications in robotics, finance, and loan management show its scalability and capability to control costs under strict operational constraints.

A Constrained Markov Decision Process (CMDP) is a stochastic control framework in which an agent selects actions to optimize an objective (e.g., maximize expected cumulative reward or minimize cost) while simultaneously enforcing additional constraints, typically on state occupancy or costs incurred during the decision process. CMDPs generalize classical Markov Decision Processes (MDPs) by introducing auxiliary cost functions and constraint thresholds, enabling explicit representation of safety, budget, or performance limits. This framework underpins a breadth of applications including operations management, robotics, finance, and autonomous systems.

1. Formal Structure and Solution Classes

A standard CMDP is defined by $$(S, A, P, r, \{c_j\}_{j=1}^m, \{q_j\}_{j=1}^m, \gamma, \alpha)$$ where $S$ is the set of states, $A$ the action sets (possibly continuous or polytopic), $P$ the transition kernel, $r$ the reward function, $c_j$ auxiliary (constrained) cost functions, $q_j$ constraint thresholds, $\gamma$ the discount factor, and $\alpha$ the initial distribution. The canonical finite-horizon or infinite-horizon discounted CMDP seeks a stationary policy $\pi^*$ maximizing the expectation of $r$ subject to $\mathbb{E}_\pi[c_j] \leq q_j$ for all $j$.

Solution methods for CMDPs bifurcate depending on action-space structure, reward/constraint convexity, and availability of model data:

• Extreme Point Reduction: If admissible actions at a state form a convex polytope $A(s)$ and the reward is affine (e.g., $r(s, a) = e_s^T a + f_s$), CMDP optimality is attained by restricting attention to policies supported only on the extreme points of $A(s)$. This equivalence is formalized in Theorem 3.1 (Petrik et al., 2013).
• Convex Optimization Formulation: For general concave (potentially piecewise-linear) rewards, and convex action sets, CMDPs are reduced to scalable convex programs over jointly chosen state transition variables $u(s, s')$, with extended reward and constraint functions evaluated as functions of normalized flow ratios.
• Concave Envelope Approach: For non-concave rewards, the CMDP is relaxed by replacing the objective with its concave envelope $$g(x) = \sup\{t : (x, t)\in \operatorname{conv}(\operatorname{hypo} f)\}$$, enabling tractable optimization followed by construction of a randomized policy for the original problem.

2. Continuous Action and Transition Modulation

CMDPs with continuous action spaces arise in cases where actions correspond to modulations of transition probabilities within specified polyhedral sets $A(s)$. In this setting, affine or piecewise-linear rewards allow reduction to finitely many extreme actions (see Section 3, (Petrik et al., 2013)). For more general reward structures (including nonconcave functions), the CMDP optimization is lifted to operate on joint transition variables. This lifting—where the action is represented not directly but via the occupation measure $u(s, s')$—circumvents the need for explicit enumeration of all extreme points and preserves feasibility by convex constraints: $$f_j\left(\frac{u(s, \cdot)}{d(s)}\right) \le 0,\quad \forall j,$$ with $d(s) = \sum_{s'} u(s, s')$.

The ability to treat continuous modulation as a high-dimensional convex program is significant for scaling CMDP solutions to instances with large state and action cardinality, as explicit enumeration becomes rapidly infeasible.

3. Convex Optimization Formulation

For concave reward functions, the CMDP is recast as a convex optimization problem with the following structure:

• Decision Variables: $u(s, s')$, the joint probabilities representing transitions under the prospective policy, and $d(s)$, the state visitation probability.
• Extended Reward: To ensure positive homogeneity, define

$$\bar{r}(s, a) = 1^T a \cdot r\left(s, \frac{a}{1^T a}\right).$$

• Feasibility Constraints: For action set $A(s)$ defined by $f_j(a)\leq0$,

$f_j\left(\frac{u(s, \cdot)}{d(s)}\right)\leq0,$

ensures transition probabilities induced by $u(s, s')/d(s)$ remain in $A(s)$.

• Objective:

$$\max \sum_{s\in S} d(s)\,\bar{r}\left(s, \frac{u(s,\cdot)}{d(s)}\right)$$

• Additional Constraints: Flow conservation, initial conditions, and quality constraints $d(s)\leq q(s)$ for relevant states.

For non-concave rewards, the concave envelope approach replaces $\bar{r}(s,a)$ with its concave envelope, solving the CMDP efficiently. The formal structure is given in Equations (4.3) and (5.6) in (Petrik et al., 2013).

4. Practical Applications and Numerical Evidence

A substantive application is in loan delinquency portfolio management. States represent delinquency statuses, and actions correspond to interventions adjusting transition probabilities. The interventions incur cost according to the deviation from baseline transition matrices.

• In real-world loan portfolios (with 8 states, 4 modifiable), the proposed global optimization method reduced expected servicing costs by 13.97% compared to baselines, with intervention efforts smoothed over time.
• In synthetic instances, the convex optimization method for concave rewards scaled to hundreds of states (solving in seconds), while the extreme point enumeration method became infeasible beyond 30 states.
• Sensitivity analyses demonstrated that solution cost and intervention effort are tightly controlled by the quality constraint (e.g., probability of default), informing policy selection in operational environments.

These results underscore the dual benefits of computational tractability and tight cost control, especially under regulatory or business-imposed constraints.

5. Mathematical Characterization

Several essential mathematical objects and formulas underpin CMDP analysis:

Symbol/Formula	Description
$d = Q P_1 \dots P_{T-1}$	State visitation probability vector over trajectory (Section 2)
$\bar{r}(s,a) = 1^T a\, r(s, a / 1^T a)$	The positively homogeneous extended reward
$\sum_{a \in A(s)} u(s,a) = d(s)$	Flow conservation for joint state-action transitions
$$g(x) = \sup\{t : (x, t) \in \mathrm{conv}(\mathrm{hypo}\, f)\}$$	Concave envelope for transforming non-concave rewards
Optimization formulation	$$\max \sum_{s\in S} d(s)\bar{r}\left(s, \frac{u(s,\cdot)}{d(s)}\right)$$ subject to constraints

The CMDP model captures the exact interplay of policy-induced transition structure, reward/cost modulations, and state-action occupancy constraints.

6. Comparison with Other Solution Approaches

Compared to classical MDP methods (value or policy iteration), and existing approaches for CMDPs (which often assume small or discrete action sets), the extreme point reduction for affine rewards is precise but computationally intractable in high dimension due to exponential growth in the number of extreme points. In contrast, the convex optimization formulation for concave (or concave-envelope) rewards enables scalability and efficiency.

Empirical evidence shows that for problems with more than 30 states, only the convex approach is computationally practical. Furthermore, for non-concave reward functions, the concave envelope technique yields solutions that significantly outperform naive linear approximations, particularly in cases where quadratic rewards would otherwise limit standard methods.

The CMDP solution approaches discussed provide both strong theoretical guarantees and practical, scalable algorithms for high-dimensional stochastic control problems previously lacking viable methods (Petrik et al., 2013).