---
title: Risk-Constrained Markov Decision Processes
url: https://www.emergentmind.com/topics/risk-constrained-markov-decision-processes-mdps-fe34aae2-c57b-48be-91d7-01de90206fc2
type: topic
---

# Risk-Constrained Markov Decision Processes

Searching arXiv for recent and foundational papers on risk-constrained MDPs and closely related risk-aware MDP formulations.
Risk-Constrained Markov Decision Processes (MDPs) are sequential decision models in which policy synthesis is governed not only by expected performance but also by explicit or implicit control of risk over trajectories, returns, costs, or model uncertainty. Within the literature, the term covers several mathematically distinct families: formulations that optimize expected reward or cost subject to a risk bound such as a chance, CVaR, or stochastic-dominance constraint; formulations that embed risk directly into the objective through dynamic coherent, exponential, utility-based, or behavioral risk functionals; and related verification- or uncertainty-oriented models that become relevant when risk is defined over specification satisfaction, catastrophic-event reachability, or posterior uncertainty rather than ordinary additive cost. Across these variants, a recurring theme is that expectation-only control can be inadequate when rare but severe outcomes matter, while risk-aware formulations generally require augmented states, nonstandard Bellman recursions, occupation-measure LPs, stochastic games, or nonconvex optimization machinery to recover tractable policy synthesis [2110.12746], [2012.02423], [2109.04082].

## 1. Conceptual scope and problem classes

The most direct risk-constrained formulation in the provided literature takes the form “optimize a nominal performance criterion subject to an explicit risk limit.” A canonical example is discounted-reward optimization with a hard bound on the probability of ever reaching a failure state, formalized as
\[
\max_{\pi} \ \mathit{Payoff}(\pi,s_0) \quad\text{subject to}\quad \mathit{Risk}(\pi,s_0)\le \Delta,
\]
where
\[
\mathit{Risk}(\pi,s)=P^\pi_s\!\left(\bigcup_{i=0}^{\infty}\{S_i \in F_M\}\right).
\]
This is a chance-constrained reachability-risk MDP rather than a standard constrained expected-cost model, because the constrained quantity is a trajectory-level event probability, not an additive auxiliary cost [2002.12086].

A broader constrained class uses dynamic coherent risk measures in both the objective and the constraints. In that setting, the discounted infinite-horizon objective is
\[
J_{\gamma}(\kappa_0,\pi)=\rho^\gamma\big(c(q_0,\alpha_0),c(q_1,\alpha_1),\ldots\big),
\]
with constraints
\[
D_\gamma^i(\kappa_0,\pi)=\rho^\gamma\big(d^i(q_0,\alpha_0),d^i(q_1,\alpha_1),\ldots\big)\le \beta^i,\qquad i=1,\dots,n_c.
\]
Here risk is not merely added as a penalty; both performance and feasibility are evaluated through nested coherent risk functionals of discounted cost streams [2012.02423], [2109.04082].

A different but closely related tradition constrains the entire empirical distribution of a performance variable through stochastic dominance. In the average-reward setting, increasing concave stochastic dominance of the empirical distribution of a quantity \(z(s_t,a_t)\) over a benchmark \(Y\) is enforced through the continuum of inequalities
\[
Z_{\eta}\left(\pi,\nu\right)\geq\mathbb{E}\left[\left(Y-\eta\right)_{-}\right], \quad \forall\eta\in\left[a,b\right],
\]
where
\[
Z_{\eta}\left(\pi,\nu\right)\triangleq\liminf_{T\rightarrow\infty}\frac{1}{T}\mathbb{E}_{\nu}^{\pi}\left[\sum_{t=0}^{T-1}\left(z\left(s_{t},a_{t}\right)-\eta\right)_{-}\right].
\]
This constrains a policy against the preferences of all increasing concave utility functions rather than one chosen scalar risk functional [1206.4568].

The literature also includes formulations that are not explicit risk-budget CMDPs but are central to the field’s taxonomy. One class directly minimizes a risk measure of total cost, such as AVaR/CVaR in transient total-cost MDPs,
\[
\pi^* \in \arg\min_{\pi\in\Pi} \mathrm{AVaR}_\tau(c(\omega)),
\]
without a separate expectation objective [1602.05130]. Another class uses a lexicographic objective that first attains the best achievable CVaR of total cost and then, within the set of CVaR-optimal policies, minimizes expected total cost [2110.12746]. These are best understood as risk-averse or lexicographically risk-constrained alternatives rather than generic solvers for arbitrary exogenous risk budgets.

A further family treats risk through exponential utility or multiplicative criteria. In finite-horizon Risk-CMDPs, the objective and the constraint may both be risk-sensitive:
\[
J_r(\pi,\alpha_r)=E^\pi_{\alpha_r}\!\left[e^{ \sum_{t=1}^{T-1}r_t(X_t,A_t,X_{t+1}) +  r_T(X_T)}\right],
\]
\[
J_c(\pi,\alpha_c)=E_{\alpha_c}^\pi\!\left[e^{ \sum_{t=1}^{T-1}{c}_t(X_t,A_t,X_{t+1}) +  c_T(X_T)}\right]\le B.
\]
This is neither CVaR-constrained nor chance-constrained; it is a constrained multiplicative/exponential-risk problem [2210.02686].

Finally, several papers are relevant because they show how risk can be attached to model uncertainty, verification objectives, or trajectory semantics rather than cumulative numeric cost. Bayesian Risk MDPs apply a nested risk functional to posterior uncertainty over unknown parameters [2106.02558], [2301.11415]. Formal-language constrained MDPs encode trajectory-level safety patterns as automata-derived costs [1910.01074]. Risk-sensitive verification under cumulative prospect theory replaces linear probability evaluation by nonlinear utility and probability distortion [1803.00091]. These formulations expand the meaning of “risk-constrained MDPs” beyond additive CMDPs.

## 2. Risk measures and semantic distinctions

A central distinction in the literature is between static risk of total return and dynamic, recursively composed one-step risk mappings. In stochastic shortest path MDPs, the CVaR of total cost is defined from the total-cost random variable \(Z\) by
\[
\mathrm{VaR}_\alpha(Z)=\min\{z\mid F(z)\ge 1-\alpha\},
\]
\[
\textnormal{CVaR}_\alpha (Z) = \frac{1}{\alpha} \int_{1-\alpha}^1 \textnormal{VaR}_{1-\gamma}(Z)\, d\gamma.
\]
For continuous \(Z\),
\[
\textnormal{CVaR}_\alpha (Z) = \mathbb{E}[Z \mid Z \ge \textnormal{VaR}_\alpha (Z)].
\]
In cost minimization, this evaluates the mean of the worst \(\alpha\)-fraction of cost outcomes [2110.12746]. The same Rockafellar–Uryasev variational form underlies several algorithms:
\[
\mathrm{AVaR}_{\tau}(Y) = \min_{s\in\mathbb R} \left\{ s+\frac{1}{1-\tau}\mathbb E[(Y-s)^+] \right\},
\]
and
\[
\operatorname{CVaR}_\alpha(X) = \min_{u\in\mathbb R} \left\{ u+\frac{1}{1-\alpha}\mathbb E[(X-u)^+] \right\}.
\]
These formulas are used for total-cost AVaR optimization, Bayesian CVaR Bellman recursions, and convex approximations [1602.05130], [2106.02558], [2301.11415].

Dynamic coherent risk measures are instead defined recursively. In discounted infinite horizon, a nested coherent risk functional has the form
\[
\rho^\gamma(\boldsymbol{c})=\lim_{t\to\infty}\rho^\gamma_{0,t}(c_0,\ldots,c_t),
\]
where
\[
\begin{aligned}
\rho^\gamma_{0,t}(c_0,\ldots,c_t)
&= \rho_0\Big( c_0+\rho_1\big( \gamma c_1+\rho_2(\gamma^2 c_2+\cdots \\
&\qquad +\rho_{t-1}(\gamma^{t-1}c_{t-1}+\rho_t(\gamma^t c_t)) \cdots )\big)\Big).
\end{aligned}
\]
This time-consistent nesting is the basis for constrained risk-averse MDPs with coherent risk in the objective and constraints [2109.04082], [2012.02423], as well as for general recursive-risk Bellman theory on Borel spaces [2010.07220].

The exponential or multiplicative criterion is conceptually different. In finite-horizon Risk-CMDPs, risk sensitivity is induced by exponentiation of cumulative rewards or constraints rather than by tail averages or coherent duality:
\[
E\!\left[\exp\!\left(\sum_t \gamma \beta^t \tilde r_t + \gamma \beta^T \tilde r_T\right)\right],
\]
with an analogous constraint functional. The paper explicitly distinguishes this from CVaR and coherent-risk formulations [2210.02686].

Behavioral risk is represented in the CPT-based verification work by
\[
C(X)= \int_0^{\infty} w_{+}\!\left(\mathbb{P}\!\left(u_{+}(X)>z\right)\right)\,dz -\int_0^{\infty} w_{-}\!\left(\mathbb{P}\!\left(u_{-}(X)>z\right)\right)\,dz .
\]
This differs from both expectation and coherent risk because probabilities themselves are distorted nonlinearly, and gains and losses are transformed separately [1803.00091].

A separate semantic distinction concerns what uncertainty is risked over. In total-cost CVaR planning, risk is over trajectory-wise cumulative cost [2110.12746], [1602.05130]. In BR-MDPs, the outer risk is over the Bayesian posterior on unknown parameters, while aleatoric dynamics are averaged inside conditional expectations [2106.02558], [2301.11415]. In chance-constrained reachability formulations, risk is the probability of ever hitting a catastrophic state [2002.12086]. In \(\omega\)-regular control under unavoidable failure, “risk aversion” is defined through successive goal-reaching probabilities rather than a scalar return functional [1603.06716].

## 3. Bellman recursions, state augmentation, and structural consequences

Risk constraints and risk-aware objectives often destroy the sufficiency of the original Markov state unless one augments it with an additional statistic. One of the most basic mechanisms is cost-to-date augmentation. In total-cost AVaR minimization, the unbounded-horizon problem is approximated by timeout truncation, then cumulative cost and time are added to the state:
\[
X' = X\times \mathbb N_N \times \mathbb N_d.
\]
The augmented transition tracks discretized accumulated cost \(y\) and stage \(z\), enabling occupancy-measure optimization over cost distributions [1602.05130].

A similar phenomenon appears for utility of cumulative reward. To recover Bellman optimality for
\[
\psi^*\in \operatorname{argsup}_{\psi\in\Psi}\; \mathbb E^\psi\!\left[ U\!\left(\sum_{h=1}^H r_h\right)\,\middle|\, s_1=s \right],
\]
the state is enlarged with cumulative reward \(y\), yielding augmented states \((s,y)\) and Bellman operators
\[
\mathcal T_h^\pi V(s,y) := \mathbb E^\pi\!\left[V(s',y+r_h)\mid s_h=s,y_h=y\right],
\]
\[
\mathcal T_h V(s,y) := \max_{a\in\mathcal A} \mathbb E\!\left[V(s',y+r_h)\mid s_h=s,y_h=y\right].
\]
This is not a constrained formulation, but it illustrates a generic state-augmentation template that also underlies constrained-risk models [2311.13589].

Dynamic coherent-risk MDPs recover Bellman structure if the one-step risk measure is representable through a Markov risk transition mapping
\[
\rho_t(v(s_{t+1})) = \sigma_t(v(s_{t+1}),s_t,p(s_{t+1}\mid s_t,\alpha_t)).
\]
Under this assumption, the Bellman operator for a discounted risk-aware MDP becomes
\[
[TJ](s)=\min_{a\in\mathbb{A}}\left\{ c(s,a)+\gamma\rho\left(J(Y^{s,a})\right)\right\},
\]
or equivalently
\[
[TJ](s)=\min_{a\in\mathbb{A}}\left\{ c(s,a)+\gamma\max_{\mu\in\mathcal{Q}(s,a)} \mathbb{E}_{Y^{s,a}\sim \mu}[J(Y^{s,a})]\right\}.
\]
This yields existence of stationary deterministic optimal policies in the risk-aware objective setting [1701.01290]. In the more general Borel-space recursive-risk model, Bellman recursions of the form
\[
T  v(x) = \inf_{a \in D(x)} \rho\Big( c\big(x,a,T(x,a,Z)\big) + \beta v\big(T(x,a,Z)\big) \Big)
\]
lead to optimal Markov policies in finite horizon and stationary optimal policies in infinite horizon under coherence, the Fatou property, and weighted contraction conditions [2010.07220].

Risk-constrained discounted MDPs with coherent objective and coherent constraints produce a Bellman-style inequality system instead of a single fixed-point equation. The central optimization over value variables \(V_\gamma\) and Lagrange multipliers \(\lambda\) is
\[
\sup_{\boldsymbol{V}_\gamma,\boldsymbol{\lambda}\succeq \boldsymbol{0}} \ \langle \boldsymbol{\kappa}_0,\boldsymbol{V}_\gamma\rangle - \langle \boldsymbol{\lambda},\boldsymbol{\beta}\rangle
\]
subject to
\[
V_\gamma(s)\le c(s,\alpha)+\langle \boldsymbol{\lambda},\boldsymbol{d}(s,\alpha)\rangle + \gamma \sigma\{V_\gamma(s'),s,p(s'|s,\alpha)\}.
\]
This is a Bellman optimization, not a standard DP recursion, and in general it lower-bounds the true constrained risk-averse problem [2012.02423], [2109.04082].

CVaR planning in SSP MDPs yields yet another structural transformation. The CVaR objective can be written as a two-player zero-sum SSP stochastic game on an augmented state space
\[
S^+= S\times [0,1]\times (A\cup\{\bot\}),
\]
where the extra variable \(y\) tracks the adversary’s remaining perturbation budget. The Bellman equation becomes
\[
V_{\textnormal{CV}}(s,y,\bot)= \min_{a\in A} \left[ C(s,a)+ \max_{\xi\in\Xi(s,y,a)} \sum_{s'\in S}\xi(s')T(s,a,s')V_{\textnormal{CV}}(s',y\xi(s'),\bot) \right].
\]
This stochastic-game representation is central to both pure CVaR optimization and the lexicographic CVaR-then-expectation refinement [2110.12746].

## 4. Linear programming, occupation measures, and nonconvex optimization

The principal tractable backbone for constrained MDPs remains occupation-measure LP. In stochastic dominance-constrained MDPs, the average-reward constrained problem becomes the infinite-dimensional LP
\[
\max\ \langle\mu,\, r\rangle
\]
subject to
\[
L_{0}\mu=0,\qquad \langle\mu,1\rangle=1,\qquad L_{1}\mu\geq y,\qquad \mu\ge 0.
\]
The dominance operator is
\[
\left[L_{1}\mu\right]\left(\eta\right)\triangleq\langle\mu,\,\left(z-\eta\right)_{-}\rangle,
\]
so the continuum of dominance constraints becomes linear in the occupation measure \(\mu\). The dual reveals modified average dynamic programming equations with an additional pricing term \(u(z(s,a))\), where the dual variable corresponds to an increasing concave utility function [1206.4568].

Occupation measures also underlie total-cost AVaR planning after timeout truncation and state augmentation. With augmented-state occupancy measure
\[
\rho(x,y,z,u) = \sum_{t=0}^{\infty} \Pr_\beta^\pi[x_t=x,y_t=y,z_t=z,u_t=u],
\]
the distribution of terminal discretized cost is recovered by
\[
\theta(k)= \sum_{(x,y,z,u)\in\mathcal K'} I(y=k\wedge z=d)\rho(x,y,z,u),
\]
and the risk-averse objective becomes
\[
\min_{\rho,\theta}\ \min_{s\in[0,\overline K d]} \left[ s+\frac{1}{1-\tau}\sum_{y\in\mathbb N_N}(y-s)^+\theta(y) \right]
\]
subject to linear flow constraints [1602.05130]. This is not a budget-constrained formulation, but it demonstrates how distributional tail criteria can be embedded into occupation-measure optimization.

Risk-sensitive constrained MDPs with exponential criteria admit a markedly different optimization structure. There, the key objects are policy-dependent forward factors
\[
\theta^\pi_{m,t}(x_t) := E^\pi_{\alpha_m}\left[e^{\sum_{k=1}^{t-1}m_k(X_k,A_k,X_{k+1})}\mathbf{1}\{X_t=x_t\}\right],
\]
and backward factors
\[
Q^\pi_{m,t}(x,a) = E_{\alpha_m}^\pi\left[e^{\sum_{\tau=t}^{T-1}m_\tau(X_\tau,A_\tau,X_{\tau+1}) + m_T(X_T)}\mid X_t=x,A_t=a \right].
\]
These induce a policy-level fixed-point condition
\[
\pi \in M(\pi),
\]
where \(M(\pi)\) is the solution set of a linear program \(LP_\pi\). The condition is necessary but not sufficient; exact solution requires a global search over fixed points or equivalent reformulations [2210.02686].

Several risk-constrained or risk-aware formulations become difference-of-convex programs. In constrained coherent-risk MDPs, the Bellman inequality optimization is written as a DCP because the risk transition mapping is convex in the continuation value [2012.02423], [2109.04082]. In CPT verification, the nonlinear probability-weighting function is approximated by a posynomial
\[
f(p)=\sum_{k=1}^K c_k p^{a_k},
\]
which is decomposed into convex and concave parts and solved by convex-concave programming [1803.00091]. In BR-MDPs, Bellman inequalities with variationally represented convex risk measures lead to bilevel DCPs over belief states and risk-measure parameters, then to approximate finite-posterior DCPs solved by DCCP-style machinery [2301.11415].

The chance-constrained planning algorithm RAlph replaces global LP solution by local LPs on a sampled search tree. The root action distribution is obtained from an LP over tree-flow variables \(x_h,x_{h,a}\) maximizing predicted reward subject to a leaf-risk budget:
\[
\sum_{h\in \mathrm{leaf}(T)} x_h \cdot h.r \le \Delta.
\]
This is not an occupation-measure LP on the full MDP but a local flow LP embedded inside MCTS [2002.12086].

## 5. Representative algorithmic paradigms

A concise taxonomy of algorithmic patterns emerges from the papers.

| Paradigm | Core idea | Representative use |
|---|---|---|
| Occupation-measure LP | Linearize flow and risk constraints in measure space | Stochastic dominance constraints [1206.4568]; discretized AVaR total-cost planning [1602.05130] |
| Bellman inequality optimization | Optimize value functions and multipliers under risk-transition constraints | Constrained coherent-risk MDPs [2012.02423], [2109.04082] |
| Stochastic-game transformation | Encode CVaR as adversarial transition perturbation | CVaR planning and lexicographic CVaR-EV [2110.12746] |
| Policy-level fixed point | Re-optimize one-step deviations using policy-dependent forward/backward factors | Exponential Risk-CMDP [2210.02686] |
| DCP / DCCP / CCP | Solve nonconvex risk Bellman problems by convex-concave iteration | Constrained coherent-risk MDPs [2012.02423]; CPT verification [1803.00091]; BR-MDP approximation [2301.11415] |
| Search + local LP | Plan online with risk-feasible randomized root actions | Failure-probability chance-constrained planning [2002.12086] |
| Approximate dynamic programming | Replace exact risk backups by sample-based estimates | Large-scale coherent-risk MDPs [1701.01290] |

The lexicographic CVaR-EV algorithm is especially notable because it isolates the slack left by pure CVaR optimization. It first solves the CVaR stochastic game to obtain a CVaR-optimal policy \(\pi_{\textnormal{CV}}\), then builds a second augmented MDP with state \((s,c)\), where \(c\) is accumulated cost, and constrains actions by
\[
\cumul(h)+Q^*_{worst}(s,a)\le \textnormal{VaR}_\alpha(\mathcal{C}^\mathcal{M}_{\pi_{\textnormal{CV}}}).
\]
The resulting policy optimizes expectation among all continuations guaranteed to remain below the VaR cap, and execution switches to this second policy once the adversarial CVaR tail no longer places probability mass on the realized branch [2110.12746].

The finite-horizon exponential Risk-CMDP algorithm GRC combines stochastic-approximation local updates
\[
\pi_{k+1}= \pi_k+ \epsilon_k (\Psi_{k}(\pi_k) -\pi_k),
\]
where \(\Psi_k(\pi_k)\in M(\pi_k)\), with random restarts occurring with probability
\[
p_k=\frac{w}{k}.
\]
The method searches globally over self-consistent policy fixed points because the fixed-point equation alone is not sufficient for optimality [2210.02686].

The approximate value iteration framework for coherent-risk MDPs uses empirical risk backups
\[
\widetilde{J}(s_i)=
\min_{a\in\mathbb{A}}
\left\{
c(s_i,a)+\gamma \hat{\rho}_m\left(\{\widehat{J}_k(Y_j^{s_i,a})\}_{j=1}^m\right)
\right\},
\]
followed by function fitting. This extends simulation-based ADP from expected-cost backups to coherent one-step risk backups, with finite-sample error decomposition into approximation and statistical terms [1701.01290].

In BR-MDPs with continuous belief spaces, ABDCP restricts the posterior set to a finite \(\hat{\mathcal M}\), interpolates reachable posteriors using convex weights \(w(\mu_i,\mu^{sas'})\), solves an approximate DCP over \((s,\mu)\in\mathcal S\times \hat{\mathcal M}\), and returns a finite state controller with performance bounds
\[
\hat V^* \le V^* \le \hat V^{\hat\pi^*},
\]
terminating when the controller-evaluation gap is below \(\epsilon\) [2301.11415].

## 6. Empirical patterns, interpretive themes, and limitations

A consistent empirical theme is that expectation-optimal policies often achieve the best mean performance but noticeably worse tail or failure behavior, whereas strongly risk-averse methods improve safety or tail metrics at a computational or nominal-performance cost. In the lexicographic CVaR study, pure expected-value policies usually deliver the lowest expected cost but much worse tail risk; CVaR-only policies achieve the best CVaR but are conservative in expectation; the lexicographic CVaR-EV method closely matches CVaR-optimal tail risk while improving expected cost across inventory control, betting, deep sea treasure, and autonomous navigation domains [2110.12746].

The total-cost AVaR study reports that a robot deployment policy optimized for \(\mathrm{AVaR}_{0.95}\) of travel time reduced deadline-exceeding runs from 61 to fewer than 30 out of 1000 executions, compared with a risk-neutral policy. The result illustrates that direct tail-risk minimization can materially reshape the completion-time distribution, not merely its mean [1602.05130].

In the rover navigation experiments for constrained coherent-risk MDPs, expectation-based policies are fastest to compute but incur larger empirical failure rates under terrain uncertainty, while CVaR and especially EVaR policies choose safer routes around obstacle-dense regions. The reported failure rates fall from \(9\%\), \(18\%\), and \(21\%\) under expectation to \(1\%\), \(3\%\), and \(5\%\) under CVaR, and to \(0\%\), \(0\%\), and \(2\%\) under EVaR on \(10\times 10\), \(15\times 15\), and \(20\times 20\) grids, respectively [2012.02423].

RAlph demonstrates that chance-constrained planning based on failure reachability can scale to very large spaces, including a hallway benchmark with about \(6.5\times 10^6\) states, by combining MCTS, learned reward/risk prediction, and local constrained LP selection. The paper’s practical message is that approximate empirical constraint satisfaction can be obtained in large MDPs where exact risk-constrained LPs are infeasible, albeit without formal guarantees once learned predictors are introduced [2002.12086].

The BR-MDP studies show a different pattern: risk is over epistemic model uncertainty rather than trajectory tail events. With small data, BR-MDP with CVaR risk in the objective often improves both expected and CVaR performance relative to nominal and distributionally robust baselines, while with large data the methods converge as posterior uncertainty shrinks [2106.02558], [2301.11415]. This suggests that the conservatism of robust MDPs can sometimes be replaced by posterior-weighted risk in a time-consistent way, although this remains an objective-based alternative rather than a hard constrained guarantee.

Several limitations recur. Static CVaR of total cost is powerful but not the same as a dynamic time-consistent risk measure, and methods built around it may require history dependence or augmented continuous states [2110.12746], [1602.05130]. Dynamic coherent-risk methods preserve Bellman structure but often yield nonconvex Bellman inequality programs or rely on strong Markov-risk assumptions [2012.02423], [2109.04082], [2010.07220]. Occupation-measure LPs can express rich constraints such as stochastic dominance, but they are infinite-dimensional in general Borel models [1206.4568]. Approximate implementations frequently depend on interpolation, discretization, or local DCCP solutions without global optimality guarantees [2110.12746], [1803.00091], [2301.11415]. Offline risk evaluation from logged data is statistically difficult, with error fundamentally controlled by the second moment of importance weights in finite-horizon MDPs [2209.10444].

A plausible implication is that there is no single dominant mathematical template for “risk-constrained MDPs.” The field instead comprises a layered hierarchy: explicit feasibility formulations such as chance, dominance, and coherent-risk constraints; risk-aware objectives such as CVaR, AVaR, recursive coherent risk, exponential utility, and Bayesian posterior risk; and specification- or semantics-driven constructions where risk is defined through verification, automata, or catastrophic-event structure. What unifies them is not one universal Bellman equation, but the repeated need to lift classical MDP machinery so that rare events, tail distributions, or uncertainty sets become first-class decision variables rather than residual properties of the expected return.

Source: https://www.emergentmind.com/topics/risk-constrained-markov-decision-processes-mdps-fe34aae2-c57b-48be-91d7-01de90206fc2