Risk-Constrained MDPs: Models, Methods & Applications

• Risk-constrained MDPs are decision frameworks that incorporate explicit risk measures—such as safety constraints, CVaR, and stochastic dominance—into policy optimization.
• They extend standard MDPs by embedding chance constraints, recursive risk measures, and risk-sensitive criteria, enabling robust learning under uncertainty.
• Computational methods like SMT, LP formulations, and difference convex programming facilitate solving these models for applications in robotics, finance, inventory control, and energy management.

Searching arXiv for recent and foundational papers on risk-constrained MDPs and related formulations. Risk-constrained Markov decision processes (MDPs) are MDP formulations in which optimization is carried out under an explicit notion of risk attached to trajectories, cumulative costs, safety events, or return distributions. In the literature, this label covers hard probabilistic safety constraints such as $\mathbb{P}^{\mdp^\sigma}(\Diamond T)\le \lambda$, constrained risk-sensitive discounted criteria based on $E[\exp(\gamma\sum_t \beta^t R_t)]$, stochastic dominance constraints on occupation measures, recursive applications of static risk measures, AVaR/CVaR and VaR criteria, Bayesian posterior-risk formulations, and cumulative prospect theory (CPT) objectives (Junges et al., 2015, M et al., 2022, Haskell et al., 2012, Bäuerle et al., 2020, Carpin et al., 2016, Brihaye et al., 14 May 2025).

1. Conceptual scope

One line of work treats risk-constrained problems as safety-constrained MDPs: the agent must optimize expected performance subject to a hard probabilistic safety requirement, and this requirement must hold even during learning and exploration when costs are unknown (Junges et al., 2015). Another line studies constrained risk-sensitive Markov decision processes (CRSMDPs) on finite state and action spaces with a discounted exponential-utility criterion and a mixture of standard discounted and risk-sensitive constraints over finite and infinite horizons (M et al., 2022). A third line uses stochastic dominance constraints on the empirical distribution of reward, yielding linear constraints on occupation measures (Haskell et al., 2012).

A distinct but closely related strand replaces expectation by recursive risk measures. In that setting the optimality criterion is based on the recursive application of static risk measures, which yields Bellman equations for finite and infinite horizons and, for coherent risk measures, a connection to distributionally robust MDPs (Bäuerle et al., 2020). Additional formulations include total-cost AVaR minimization on transient absorbing MDPs (Carpin et al., 2016), fixed-point methods for finite-horizon entropic-risk constrained MDPs (Singh et al., 2022), lexicographic planning that minimizes expected cost subject to optimal CVaR (Rigter et al., 2021), and Bayesian posterior-risk models in which risk is taken with respect to the posterior distribution of unknown parameters (Lin et al., 2021).

The term therefore does not denote a single canonical risk functional. In some formulations the constraint is a hard chance constraint; in others it is a coherent dynamic risk bound; in others still, risk appears as the primary objective rather than as a side constraint. This suggests that “risk-constrained MDPs” functions as an umbrella description for several related paradigms rather than a single formalism (Junges et al., 2015, M et al., 2022, Bäuerle et al., 2020, Brihaye et al., 14 May 2025).

2. Canonical mathematical formulations

In safety-constrained exploration, the environment is a finite, fully observable MDP $\mdp=(S,s_{\mathit{init}},\Act,\pmdp)$ with known transition probabilities and unknown but bounded costs $\rho(s,a)$, with bounds $l_{(s,a)}\le \rho(s,a)\le u_{(s,a)}$ learned only by executing actions (Junges et al., 2015). Safety is expressed as a probabilistic reachability constraint

$\varphi=\mathsf{P}_{\le \lambda}(\Diamond T),$

and performance as an expected cumulative cost specification

$\psi=\mathsf{E}_{\le \kappa}(\Diamond G).$

The synthesis problem is to find $\sigma$ such that

$\mathbb{P}^{\mdp^\sigma}(\Diamond T)\le \lambda \quad\text{and}\quad \mathbb{E}^{\mdp^\sigma}[\text{cost to reach }G]\le \kappa.$

In CRSMDPs, the state and action spaces are finite, the discount factor satisfies $\beta\in(0,1)$, and the risk factor satisfies $E[\exp(\gamma\sum_t \beta^t R_t)]$0 (M et al., 2022). For an MR policy $E[\exp(\gamma\sum_t \beta^t R_t)]$1, objective performance is

$E[\exp(\gamma\sum_t \beta^t R_t)]$2

and feasibility may simultaneously involve infinite-horizon standard discounted constraints, infinite-horizon risk-sensitive constraints, finite-horizon standard discounted constraints, and finite-horizon risk-sensitive constraints. The base problem is

$E[\exp(\gamma\sum_t \beta^t R_t)]$3

Stochastic dominance-constrained MDPs impose distributional constraints on a secondary reward $E[\exp(\gamma\sum_t \beta^t R_t)]$4 through increasing concave stochastic order (Haskell et al., 2012). With

$E[\exp(\gamma\sum_t \beta^t R_t)]$5

the dominance-constrained problem is

$E[\exp(\gamma\sum_t \beta^t R_t)]$6

In occupation-measure form, the constraint becomes $E[\exp(\gamma\sum_t \beta^t R_t)]$7.

Recursive-risk MDPs replace conditional expectation in the Bellman recursion by a one-step risk measure (Bäuerle et al., 2020). In the stationary infinite-horizon case,

$E[\exp(\gamma\sum_t \beta^t R_t)]$8

The infinite-horizon value is the unique fixed point of $E[\exp(\gamma\sum_t \beta^t R_t)]$9 under the paper’s contractivity assumptions.

For total-cost AVaR, the objective is

$\mdp=(S,s_{\mathit{init}},\Act,\pmdp)$0

with

$\mdp=(S,s_{\mathit{init}},\Act,\pmdp)$1

on a finite transient MDP with an absorbing structure (Carpin et al., 2016).

For VaR, the paper distinguishes the VaR of steady-state rewards over an infinite horizon and the VaR of accumulated rewards over a finite horizon (Xia et al., 30 Jul 2025). In the finite-horizon case,

$\mdp=(S,s_{\mathit{init}},\Act,\pmdp)$2

Paradigm	Canonical formulation	Representative source
Safety-constrained exploration	$\mdp=(S,s_{\mathit{init}},\Act,\pmdp)$3 with expected-cost objective	(Junges et al., 2015)
CRSMDP	$\mdp=(S,s_{\mathit{init}},\Act,\pmdp)$4	(M et al., 2022)
Stochastic dominance	$\mdp=(S,s_{\mathit{init}},\Act,\pmdp)$5 or $\mdp=(S,s_{\mathit{init}},\Act,\pmdp)$6	(Haskell et al., 2012)
Recursive risk	$\mdp=(S,s_{\mathit{init}},\Act,\pmdp)$7	(Bäuerle et al., 2020)
Total-cost AVaR	$\mdp=(S,s_{\mathit{init}},\Act,\pmdp)$8	(Carpin et al., 2016)
VaR MDP	Optimize $\mdp=(S,s_{\mathit{init}},\Act,\pmdp)$9 of steady-state or accumulated rewards	(Xia et al., 30 Jul 2025)

Bayesian-risk formulations apply a risk functional in nested form to expected total cost with respect to the Bayesian posterior distribution of unknown parameters, with policies $\rho(s,a)$0 and posterior updates by Bayes’ rule (Lin et al., 2021). CPT-based models evaluate a strategy by the CPT-value of the induced prospect over weighted reachability or mean-payoff outcomes, rather than by an expectation or coherent-risk functional (Brihaye et al., 14 May 2025). Verification-oriented work further embeds CPT in temporal-logic-constrained MDPs by reducing temporal objectives to reachability on a product structure (Cubuktepe et al., 2018).

3. Policy classes and structural results

Policy structure depends strongly on the risk formalism. In safety-constrained exploration, a memoryless scheduler is $\rho(s,a)$1, with deterministic policies as the Dirac special case, and safe exploration is expressed through a permissive scheduler $\rho(s,a)$2 that denotes a family of compliant policies (Junges et al., 2015). Safety of $\rho(s,a)$3 means that every compliant scheduler satisfies the chance constraint, so a permissive scheduler is a safe region of policy space rather than a single point.

CRSMDPs restrict attention to Markovian randomized (MR) policies, represented as infinite sequences of row-stochastic decision matrices, and distinguish stationary and ultimately stationary (US) policies (M et al., 2022). The paper proves existence of optimal MR policies when the feasible region is nonempty, and the approximation schemes produce near-optimal US policies. In the recursive-risk setting, finite-horizon optimal policies can be chosen Markov, and in the infinite-horizon discounted case the optimal policy is stationary under the paper’s contractivity assumptions (Bäuerle et al., 2020).

For CPT weighted reachability, memoryless randomized strategies are necessary and sufficient for optimality (Brihaye et al., 14 May 2025). The paper gives an explicit counterexample showing that deterministic strategies need not attain the optimal CPT-value, so randomization is structurally essential. VaR formulations exhibit a different pattern: deterministic stationary policies are optimal for steady-state VaR MDPs, whereas deterministic history-dependent policies are optimal for finite-horizon VaR MDPs because the augmented state must encode cumulative reward or a remaining goal variable (Xia et al., 30 Jul 2025).

In lexicographic CVaR planning, the refinement stage optimizes expected cost among policies that preserve optimal CVaR by constraining worst-case cost never to exceed the relevant VaR threshold (Rigter et al., 2021). In Bayesian-risk MDPs, the policy must depend jointly on the physical state and the posterior $\rho(s,a)$4, since $\rho(s,a)$5 is part of the controlled state process (Lin et al., 2021). For constrained risk-averse MDPs with dynamic coherent risk measures, Markovian policies are synthesized under the assumption that the coherent risk measures admit a Markov risk transition mapping (Ahmadi et al., 2021).

4. Computational methods

Chance-constrained safe exploration is treated by first computing safe permissive schedulers via SMT and then running RL inside the induced safe sub-MDP (Junges et al., 2015). The SMT encoding introduces Boolean variables $\rho(s,a)$6 indicating allowed actions and real variables $\rho(s,a)$7 representing maximal bad-state reachability probabilities, with constraints

$\rho(s,a)$8

The resulting encoding is sound and complete for safe deterministic permissive schedulers, and Q-learning is then restricted to allowed actions. This enforces the safety constraint during learning, not just for the terminal policy.

For CRSMDPs, the main approximation strategy is to truncate the infinite horizon at time $\rho(s,a)$9, tighten or relax the constraint bounds, and solve two approximating finite-horizon problems $l_{(s,a)}\le \rho(s,a)\le u_{(s,a)}$0 and $l_{(s,a)}\le \rho(s,a)\le u_{(s,a)}$1 (M et al., 2022). The finite-horizon problems are reduced to LPs by augmenting state with multiplicative risk-sensitive accumulators $l_{(s,a)}\le \rho(s,a)\le u_{(s,a)}$2, introducing occupation measures $l_{(s,a)}\le \rho(s,a)\le u_{(s,a)}$3, and encoding both standard discounted and exponential-utility constraints linearly in those occupation measures. The inner approximation yields $l_{(s,a)}\le \rho(s,a)\le u_{(s,a)}$4-optimal feasible policies under the paper’s local-minimum condition on the max-violation map $l_{(s,a)}\le \rho(s,a)\le u_{(s,a)}$5; the outer approximation yields $l_{(s,a)}\le \rho(s,a)\le u_{(s,a)}$6-feasible near-optimal policies without that condition.

Stochastic dominance constraints also admit an occupation-measure formulation (Haskell et al., 2012). In the average-reward case the primal LP is

$l_{(s,a)}\le \rho(s,a)\le u_{(s,a)}$7

and the dual introduces a value-function term $l_{(s,a)}\le \rho(s,a)\le u_{(s,a)}$8, an average-reward scalar $l_{(s,a)}\le \rho(s,a)\le u_{(s,a)}$9, and a measure $\varphi=\mathsf{P}_{\le \lambda}(\Diamond T),$0 over shortfall levels $\varphi=\mathsf{P}_{\le \lambda}(\Diamond T),$1. The dual Bellman inequalities contain a new pricing term

$\varphi=\mathsf{P}_{\le \lambda}(\Diamond T),$2

so the constrained MDP behaves as if it were risk-neutral with modified one-step reward $\varphi=\mathsf{P}_{\le \lambda}(\Diamond T),$3.

Finite-horizon entropic-risk constrained MDPs admit a different route: fixed-point equations and policy-dependent LPs built from forward factors $\varphi=\mathsf{P}_{\le \lambda}(\Diamond T),$4 and backward risk-sensitive $\varphi=\mathsf{P}_{\le \lambda}(\Diamond T),$5-factors $\varphi=\mathsf{P}_{\le \lambda}(\Diamond T),$6 (Singh et al., 2022). The paper defines a set-valued mapping $\varphi=\mathsf{P}_{\le \lambda}(\Diamond T),$7 of LP optimizers and proves that any optimal policy satisfies the fixed-point condition $\varphi=\mathsf{P}_{\le \lambda}(\Diamond T),$8. This yields a local-improvement method, and a global algorithm with random restarts whose complexity grows only linearly with the horizon.

Total-cost AVaR problems are handled by introducing a timeout horizon $\varphi=\mathsf{P}_{\le \lambda}(\Diamond T),$9, constructing a surrogate problem $\psi=\mathsf{E}_{\le \kappa}(\Diamond G).$0, and proving a computable suboptimality gap between $\psi=\mathsf{E}_{\le \kappa}(\Diamond G).$1 and $\psi=\mathsf{E}_{\le \kappa}(\Diamond G).$2 (Carpin et al., 2016). Cumulative cost is discretized with step size $\psi=\mathsf{E}_{\le \kappa}(\Diamond G).$3, the state is augmented to $\psi=\mathsf{E}_{\le \kappa}(\Diamond G).$4 where $\psi=\mathsf{E}_{\le \kappa}(\Diamond G).$5 is a discrete cumulative-cost index and $\psi=\mathsf{E}_{\le \kappa}(\Diamond G).$6 is a stage counter, and occupancy measures $\psi=\mathsf{E}_{\le \kappa}(\Diamond G).$7 are used to reconstruct the distribution $\psi=\mathsf{E}_{\le \kappa}(\Diamond G).$8. For fixed $\psi=\mathsf{E}_{\le \kappa}(\Diamond G).$9 in the Rockafellar–Uryasev representation, the optimization over $\sigma$0 is an LP.

Dynamic coherent risk constraints can be cast directly as difference convex programs (DCPs) (Ahmadi et al., 2020, Ahmadi et al., 2021). For MDPs, the Bellman-type constraints

$\sigma$1

lead to DCPs because the Markov risk transition mapping $\sigma$2 is convex in the value function. These DCPs are solved with disciplined convex-concave programming (DCCP), yielding lower bounds on the constrained risk-averse problem and Markovian policies synthesized from the resulting value function and multipliers.

CPT-based verification uses a related convex-concave strategy (Cubuktepe et al., 2018). The nonlinear weighting function is approximated by a posynomial

$\sigma$3

which is represented as a difference of convex functions because monomials with $\sigma$4 are concave and monomials with $\sigma$5 are convex. The local policy-optimization problem at each state and time is then a DC program solved by CCP.

Bayesian-risk MDPs with CVaR exploit a nested posterior-risk Bellman recursion and a POMDP-like $\sigma$6-function representation (Lin et al., 2021). The exact $\sigma$7-function set grows rapidly, so the paper introduces lower and upper bounds and an approximate recursion $\sigma$8; under convexity assumptions, the approximate value $\sigma$9 is convex in the CVaR auxiliary variables $\mathbb{P}^{\mdp^\sigma}(\Diamond T)\le \lambda \quad\text{and}\quad \mathbb{E}^{\mdp^\sigma}[\text{cost to reach }G]\le \kappa.$0, which enables gradient or stochastic-gradient descent over those parameters.

A lexicographic alternative first computes a CVaR-optimal policy and then solves a second augmented MDP that minimizes expected cost subject to never exceeding the relevant VaR threshold; this improves expected cost while keeping the optimal CVaR unchanged (Rigter et al., 2021).

5. Representative applications

The application profile is broad but heavily concentrated in robotics, autonomous systems, finance, inventory, and energy management. In safety-constrained exploration, three robotic or embedded benchmarks are used: Janitor, Following a line fragment, and Communicating explorer (Junges et al., 2015). For Janitor and Follow Line, computing a locally maximal permissive scheduler allows the algorithm to quickly narrow the gap between lower and upper bounds on expected cost, and often after $\mathbb{P}^{\mdp^\sigma}(\Diamond T)\le \lambda \quad\text{and}\quad \mathbb{E}^{\mdp^\sigma}[\text{cost to reach }G]\le \kappa.$1–$\mathbb{P}^{\mdp^\sigma}(\Diamond T)\le \lambda \quad\text{and}\quad \mathbb{E}^{\mdp^\sigma}[\text{cost to reach }G]\le \kappa.$2 iterations the bounds are already tight. For Communicating explorer, bounds are initially loose because a policy that never communicates appears cost-effective but unsafe.

In the total-cost AVaR setting, a rapid deployment scenario is modeled as a robot that must reach a target location within a temporal deadline while increased speed is associated with increased probability of failure (Carpin et al., 2016). In the reported simulations, the risk-neutral policy has $\mathbb{P}^{\mdp^\sigma}(\Diamond T)\le \lambda \quad\text{and}\quad \mathbb{E}^{\mdp^\sigma}[\text{cost to reach }G]\le \kappa.$3 runs with cost $\mathbb{P}^{\mdp^\sigma}(\Diamond T)\le \lambda \quad\text{and}\quad \mathbb{E}^{\mdp^\sigma}[\text{cost to reach }G]\le \kappa.$4, whereas the AVaR-based policy has fewer than $\mathbb{P}^{\mdp^\sigma}(\Diamond T)\le \lambda \quad\text{and}\quad \mathbb{E}^{\mdp^\sigma}[\text{cost to reach }G]\le \kappa.$5 such runs. The same framework also returns an approximate full distribution over total costs, not merely a scalar risk value.

Stochastic dominance is illustrated with a portfolio optimization example in which discounted transaction cost is minimized subject to discounted return dominating a benchmark in increasing concave order (Haskell et al., 2012). VaR optimization is demonstrated on a microgrid problem where renewable generation, storage, and demand define the state, and the steady-state reward is the trade with the main grid (Xia et al., 30 Jul 2025). In that case, the optimal VaR is positive at $\mathbb{P}^{\mdp^\sigma}(\Diamond T)\le \lambda \quad\text{and}\quad \mathbb{E}^{\mdp^\sigma}[\text{cost to reach }G]\le \kappa.$6, meaning that with $\mathbb{P}^{\mdp^\sigma}(\Diamond T)\le \lambda \quad\text{and}\quad \mathbb{E}^{\mdp^\sigma}[\text{cost to reach }G]\le \kappa.$7 confidence the system has surplus power, while at $\mathbb{P}^{\mdp^\sigma}(\Diamond T)\le \lambda \quad\text{and}\quad \mathbb{E}^{\mdp^\sigma}[\text{cost to reach }G]\le \kappa.$8 and $\mathbb{P}^{\mdp^\sigma}(\Diamond T)\le \lambda \quad\text{and}\quad \mathbb{E}^{\mdp^\sigma}[\text{cost to reach }G]\le \kappa.$9 the optimal VaR is negative.

Inventory control recurs across several formulations. The finite-horizon exponential Risk-CMDP paper studies inventory control with a risk-sensitive objective and a risk-sensitive constraint, reports feasible computation for horizons up to $\beta\in(0,1)$0, and observes ultimately stationary policy structure in the examples (Singh et al., 2022). Bayesian-risk MDPs are evaluated on both gambler’s betting and inventory control problems; when data are scarce, BR-MDP policies have substantially lower variance than nominal policies and substantially better mean than DR-MDP policies (Lin et al., 2021).

CPT and dynamic coherent-risk verification papers emphasize gridworld and navigation tasks (Cubuktepe et al., 2018, Ahmadi et al., 2020). In the $\beta\in(0,1)$1 gridworld navigation example, the risk-neutral policy has average cost of successful runs about $\beta\in(0,1)$2 with $\beta\in(0,1)$3 crashes, while the CPT risk-averse policy has average cost of successful runs about $\beta\in(0,1)$4 with $\beta\in(0,1)$5 crashes. The constrained risk-averse rover navigation problem with CVaR and EVaR likewise produces more conservative paths and lower failure rates than expectation-based planning (Ahmadi et al., 2020). Additional CPT examples include consensus protocol verification and a ride-sharing decision problem in which the passenger’s policy responds to surge multipliers and waiting time (Cubuktepe et al., 2018).

Finally, lexicographic CVaR planning on stochastic shortest-path MDPs is evaluated on four domains and improves the expected cost compared to the state-of-the-art algorithm while achieving the optimal CVaR (Rigter et al., 2021).

6. Assumptions, limitations, and open directions

The assumptions are highly model-specific. Safety-constrained exploration assumes finite, fully observable MDPs with known transition probabilities and unknown but bounded local costs, and the safety specification is restricted to a single reachability constraint $\beta\in(0,1)$6 (Junges et al., 2015). CRSMDPs assume finite state and action spaces, discounted costs, uniformly bounded immediate costs, and exponential utility; the inner approximation theorem additionally requires that $\beta\in(0,1)$7 is not a local minimum of the max-violation map $\beta\in(0,1)$8, and the paper shows that this condition is essential through counterexamples (M et al., 2022). Stochastic dominance formulations allow Borel state and action spaces but assume weak continuity of the transition kernel, bounded upper semicontinuous $\beta\in(0,1)$9, and a benchmark $E[\exp(\gamma\sum_t \beta^t R_t)]$00 with support in a compact interval $E[\exp(\gamma\sum_t \beta^t R_t)]$01 (Haskell et al., 2012). Recursive-risk models admit unbounded costs, but only under bounding-function conditions that make the Bellman operator contractive (Bäuerle et al., 2020).

Algorithmic bottlenecks differ accordingly. Safe permissive scheduler synthesis is limited by SMT scalability and the trade-off between permissiveness and solver burden (Junges et al., 2015). The LP formulation for CRSMDPs suffers from augmented-state growth exponential in the horizon and in the number of risk-sensitive constraints (M et al., 2022). Dominance-constrained MDPs reduce to infinite-dimensional LPs in general Borel spaces (Haskell et al., 2012). Total-cost AVaR requires transient absorbing structure, positive bounded stage costs in transient states, and a bilinear outer optimization in the VaR auxiliary variable $E[\exp(\gamma\sum_t \beta^t R_t)]$02 that the paper resolves by grid search (Carpin et al., 2016). DCCP- and CCP-based methods solve DC approximations and generally return local optima or local saddle points rather than certified global solutions (Cubuktepe et al., 2018, Ahmadi et al., 2020).

Several conceptual distinctions recur across the literature. Safety-chance constraints of the form $E[\exp(\gamma\sum_t \beta^t R_t)]$03 are not the same as coherent or law-invariant risk measures on return distributions (Junges et al., 2015). VaR is non-additive, which is why traditional dynamic programming is inapplicable in the form used for expected-value MDPs, whereas CVaR is coherent and admits convex representations (Xia et al., 30 Jul 2025, Carpin et al., 2016). Stochastic dominance constrains the entire return distribution relative to a benchmark rather than a single tail functional (Haskell et al., 2012). CPT is behaviorally expressive but non-convex, and the corresponding optimization relies on approximations and local methods (Brihaye et al., 14 May 2025, Cubuktepe et al., 2018).

The open directions stated in the papers are correspondingly diverse: richer risk measures such as CVaR or law-invariant measures in models that currently use only chance constraints or exponential utility, more general temporal safety and temporal logic beyond reachability, POMDP extensions with safety guarantees, unknown transition probabilities, Borel state and action spaces, unbounded costs, tighter feasibility conditions for finite-horizon approximations, and iterative algorithms beyond LP reduction (Junges et al., 2015, M et al., 2022, Bäuerle et al., 2020, Lin et al., 2021, Ahmadi et al., 2021). CPT-based work also suggests adaptation to risk-constrained formulations by combining multi-objective reachability regions with CPT feasibility sets, although the cited paper itself treats CPT as the primary optimization objective rather than as an explicit side constraint (Brihaye et al., 14 May 2025).