Risk-Constrained Markov Decision Processes

• Risk-Constrained MDPs are sequential decision models that incorporate risk limits (e.g., chance constraints, CVaR) to mitigate rare, high-cost events.
• They employ techniques like state augmentation, nonstandard Bellman recursions, and occupation-measure LPs to optimize policies under explicit risk bounds.
• Algorithmic paradigms range from stochastic-game transformations to convex-concave programming, addressing trade-offs between safety, tractability, and expected performance.

Searching arXiv for recent and foundational papers on risk-constrained MDPs and closely related risk-aware MDP formulations. Risk-Constrained Markov Decision Processes (MDPs) are sequential decision models in which policy synthesis is governed not only by expected performance but also by explicit or implicit control of risk over trajectories, returns, costs, or model uncertainty. Within the literature, the term covers several mathematically distinct families: formulations that optimize expected reward or cost subject to a risk bound such as a chance, CVaR, or stochastic-dominance constraint; formulations that embed risk directly into the objective through dynamic coherent, exponential, utility-based, or behavioral risk functionals; and related verification- or uncertainty-oriented models that become relevant when risk is defined over specification satisfaction, catastrophic-event reachability, or posterior uncertainty rather than ordinary additive cost. Across these variants, a recurring theme is that expectation-only control can be inadequate when rare but severe outcomes matter, while risk-aware formulations generally require augmented states, nonstandard Bellman recursions, occupation-measure LPs, stochastic games, or nonconvex optimization machinery to recover tractable policy synthesis (Rigter et al., 2021, Ahmadi et al., 2020, Ahmadi et al., 2021).

1. Conceptual scope and problem classes

The most direct risk-constrained formulation in the provided literature takes the form “optimize a nominal performance criterion subject to an explicit risk limit.” A canonical example is discounted-reward optimization with a hard bound on the probability of ever reaching a failure state, formalized as

$$\max_{\pi} \ \mathit{Payoff}(\pi,s_0) \quad\text{subject to}\quad \mathit{Risk}(\pi,s_0)\le \Delta,$$

where

$$\mathit{Risk}(\pi,s)=P^\pi_s\!\left(\bigcup_{i=0}^{\infty}\{S_i \in F_M\}\right).$$

This is a chance-constrained reachability-risk MDP rather than a standard constrained expected-cost model, because the constrained quantity is a trajectory-level event probability, not an additive auxiliary cost (Brazdil et al., 2020).

A broader constrained class uses dynamic coherent risk measures in both the objective and the constraints. In that setting, the discounted infinite-horizon objective is

$$J_{\gamma}(\kappa_0,\pi)=\rho^\gamma\big(c(q_0,\alpha_0),c(q_1,\alpha_1),\ldots\big),$$

with constraints

$$D_\gamma^i(\kappa_0,\pi)=\rho^\gamma\big(d^i(q_0,\alpha_0),d^i(q_1,\alpha_1),\ldots\big)\le \beta^i,\qquad i=1,\dots,n_c.$$

Here risk is not merely added as a penalty; both performance and feasibility are evaluated through nested coherent risk functionals of discounted cost streams (Ahmadi et al., 2020, Ahmadi et al., 2021).

A different but closely related tradition constrains the entire empirical distribution of a performance variable through stochastic dominance. In the average-reward setting, increasing concave stochastic dominance of the empirical distribution of a quantity $z(s_t,a_t)$ over a benchmark $Y$ is enforced through the continuum of inequalities

$$Z_{\eta}\left(\pi,\nu\right)\geq\mathbb{E}\left[\left(Y-\eta\right)_{-}\right], \quad \forall\eta\in\left[a,b\right],$$

where

$$Z_{\eta}\left(\pi,\nu\right)\triangleq\liminf_{T\rightarrow\infty}\frac{1}{T}\mathbb{E}_{\nu}^{\pi}\left[\sum_{t=0}^{T-1}\left(z\left(s_{t},a_{t}\right)-\eta\right)_{-}\right].$$

This constrains a policy against the preferences of all increasing concave utility functions rather than one chosen scalar risk functional (Haskell et al., 2012).

The literature also includes formulations that are not explicit risk-budget CMDPs but are central to the field’s taxonomy. One class directly minimizes a risk measure of total cost, such as AVaR/CVaR in transient total-cost MDPs,

$$\pi^* \in \arg\min_{\pi\in\Pi} \mathrm{AVaR}_\tau(c(\omega)),$$

without a separate expectation objective (Carpin et al., 2016). Another class uses a lexicographic objective that first attains the best achievable CVaR of total cost and then, within the set of CVaR-optimal policies, minimizes expected total cost (Rigter et al., 2021). These are best understood as risk-averse or lexicographically risk-constrained alternatives rather than generic solvers for arbitrary exogenous risk budgets.

A further family treats risk through exponential utility or multiplicative criteria. In finite-horizon Risk-CMDPs, the objective and the constraint may both be risk-sensitive: $$J_r(\pi,\alpha_r)=E^\pi_{\alpha_r}\!\left[e^{ \sum_{t=1}^{T-1}r_t(X_t,A_t,X_{t+1}) + r_T(X_T)}\right],$$

$$\mathit{Risk}(\pi,s)=P^\pi_s\!\left(\bigcup_{i=0}^{\infty}\{S_i \in F_M\}\right).$$0

This is neither CVaR-constrained nor chance-constrained; it is a constrained multiplicative/exponential-risk problem (Singh et al., 2022).

Finally, several papers are relevant because they show how risk can be attached to model uncertainty, verification objectives, or trajectory semantics rather than cumulative numeric cost. Bayesian Risk MDPs apply a nested risk functional to posterior uncertainty over unknown parameters (Lin et al., 2021, Lin et al., 2023). Formal-language constrained MDPs encode trajectory-level safety patterns as automata-derived costs (Quint et al., 2019). Risk-sensitive verification under cumulative prospect theory replaces linear probability evaluation by nonlinear utility and probability distortion (Cubuktepe et al., 2018). These formulations expand the meaning of “risk-constrained MDPs” beyond additive CMDPs.

2. Risk measures and semantic distinctions

A central distinction in the literature is between static risk of total return and dynamic, recursively composed one-step risk mappings. In stochastic shortest path MDPs, the CVaR of total cost is defined from the total-cost random variable $$\mathit{Risk}(\pi,s)=P^\pi_s\!\left(\bigcup_{i=0}^{\infty}\{S_i \in F_M\}\right).$$1 by

$$\mathit{Risk}(\pi,s)=P^\pi_s\!\left(\bigcup_{i=0}^{\infty}\{S_i \in F_M\}\right).$$2

$$\mathit{Risk}(\pi,s)=P^\pi_s\!\left(\bigcup_{i=0}^{\infty}\{S_i \in F_M\}\right).$$3

For continuous $$\mathit{Risk}(\pi,s)=P^\pi_s\!\left(\bigcup_{i=0}^{\infty}\{S_i \in F_M\}\right).$$4,

$$\mathit{Risk}(\pi,s)=P^\pi_s\!\left(\bigcup_{i=0}^{\infty}\{S_i \in F_M\}\right).$$5

In cost minimization, this evaluates the mean of the worst $$\mathit{Risk}(\pi,s)=P^\pi_s\!\left(\bigcup_{i=0}^{\infty}\{S_i \in F_M\}\right).$$6-fraction of cost outcomes (Rigter et al., 2021). The same Rockafellar–Uryasev variational form underlies several algorithms: $$\mathit{Risk}(\pi,s)=P^\pi_s\!\left(\bigcup_{i=0}^{\infty}\{S_i \in F_M\}\right).$$7 and

$$\mathit{Risk}(\pi,s)=P^\pi_s\!\left(\bigcup_{i=0}^{\infty}\{S_i \in F_M\}\right).$$8

These formulas are used for total-cost AVaR optimization, Bayesian CVaR Bellman recursions, and convex approximations (Carpin et al., 2016, Lin et al., 2021, Lin et al., 2023).

Dynamic coherent risk measures are instead defined recursively. In discounted infinite horizon, a nested coherent risk functional has the form

$$\mathit{Risk}(\pi,s)=P^\pi_s\!\left(\bigcup_{i=0}^{\infty}\{S_i \in F_M\}\right).$$9

where

$$J_{\gamma}(\kappa_0,\pi)=\rho^\gamma\big(c(q_0,\alpha_0),c(q_1,\alpha_1),\ldots\big),$$0

This time-consistent nesting is the basis for constrained risk-averse MDPs with coherent risk in the objective and constraints (Ahmadi et al., 2021, Ahmadi et al., 2020), as well as for general recursive-risk Bellman theory on Borel spaces (Bäuerle et al., 2020).

The exponential or multiplicative criterion is conceptually different. In finite-horizon Risk-CMDPs, risk sensitivity is induced by exponentiation of cumulative rewards or constraints rather than by tail averages or coherent duality: $$J_{\gamma}(\kappa_0,\pi)=\rho^\gamma\big(c(q_0,\alpha_0),c(q_1,\alpha_1),\ldots\big),$$1 with an analogous constraint functional. The paper explicitly distinguishes this from CVaR and coherent-risk formulations (Singh et al., 2022).

Behavioral risk is represented in the CPT-based verification work by

$$J_{\gamma}(\kappa_0,\pi)=\rho^\gamma\big(c(q_0,\alpha_0),c(q_1,\alpha_1),\ldots\big),$$2

This differs from both expectation and coherent risk because probabilities themselves are distorted nonlinearly, and gains and losses are transformed separately (Cubuktepe et al., 2018).

A separate semantic distinction concerns what uncertainty is risked over. In total-cost CVaR planning, risk is over trajectory-wise cumulative cost (Rigter et al., 2021, Carpin et al., 2016). In BR-MDPs, the outer risk is over the Bayesian posterior on unknown parameters, while aleatoric dynamics are averaged inside conditional expectations (Lin et al., 2021, Lin et al., 2023). In chance-constrained reachability formulations, risk is the probability of ever hitting a catastrophic state (Brazdil et al., 2020). In $$J_{\gamma}(\kappa_0,\pi)=\rho^\gamma\big(c(q_0,\alpha_0),c(q_1,\alpha_1),\ldots\big),$$3-regular control under unavoidable failure, “risk aversion” is defined through successive goal-reaching probabilities rather than a scalar return functional (Ehlers et al., 2016).

3. Bellman recursions, state augmentation, and structural consequences

Risk constraints and risk-aware objectives often destroy the sufficiency of the original Markov state unless one augments it with an additional statistic. One of the most basic mechanisms is cost-to-date augmentation. In total-cost AVaR minimization, the unbounded-horizon problem is approximated by timeout truncation, then cumulative cost and time are added to the state: $$J_{\gamma}(\kappa_0,\pi)=\rho^\gamma\big(c(q_0,\alpha_0),c(q_1,\alpha_1),\ldots\big),$$4 The augmented transition tracks discretized accumulated cost $$J_{\gamma}(\kappa_0,\pi)=\rho^\gamma\big(c(q_0,\alpha_0),c(q_1,\alpha_1),\ldots\big),$$5 and stage $$J_{\gamma}(\kappa_0,\pi)=\rho^\gamma\big(c(q_0,\alpha_0),c(q_1,\alpha_1),\ldots\big),$$6, enabling occupancy-measure optimization over cost distributions (Carpin et al., 2016).

A similar phenomenon appears for utility of cumulative reward. To recover Bellman optimality for

$$J_{\gamma}(\kappa_0,\pi)=\rho^\gamma\big(c(q_0,\alpha_0),c(q_1,\alpha_1),\ldots\big),$$7

the state is enlarged with cumulative reward $$J_{\gamma}(\kappa_0,\pi)=\rho^\gamma\big(c(q_0,\alpha_0),c(q_1,\alpha_1),\ldots\big),$$8, yielding augmented states $$J_{\gamma}(\kappa_0,\pi)=\rho^\gamma\big(c(q_0,\alpha_0),c(q_1,\alpha_1),\ldots\big),$$9 and Bellman operators

$$D_\gamma^i(\kappa_0,\pi)=\rho^\gamma\big(d^i(q_0,\alpha_0),d^i(q_1,\alpha_1),\ldots\big)\le \beta^i,\qquad i=1,\dots,n_c.$$0

$$D_\gamma^i(\kappa_0,\pi)=\rho^\gamma\big(d^i(q_0,\alpha_0),d^i(q_1,\alpha_1),\ldots\big)\le \beta^i,\qquad i=1,\dots,n_c.$$1

This is not a constrained formulation, but it illustrates a generic state-augmentation template that also underlies constrained-risk models (Wu et al., 2023).

Dynamic coherent-risk MDPs recover Bellman structure if the one-step risk measure is representable through a Markov risk transition mapping

$$D_\gamma^i(\kappa_0,\pi)=\rho^\gamma\big(d^i(q_0,\alpha_0),d^i(q_1,\alpha_1),\ldots\big)\le \beta^i,\qquad i=1,\dots,n_c.$$2

Under this assumption, the Bellman operator for a discounted risk-aware MDP becomes

$$D_\gamma^i(\kappa_0,\pi)=\rho^\gamma\big(d^i(q_0,\alpha_0),d^i(q_1,\alpha_1),\ldots\big)\le \beta^i,\qquad i=1,\dots,n_c.$$3

or equivalently

$$D_\gamma^i(\kappa_0,\pi)=\rho^\gamma\big(d^i(q_0,\alpha_0),d^i(q_1,\alpha_1),\ldots\big)\le \beta^i,\qquad i=1,\dots,n_c.$$4

This yields existence of stationary deterministic optimal policies in the risk-aware objective setting (Yu et al., 2017). In the more general Borel-space recursive-risk model, Bellman recursions of the form

$$D_\gamma^i(\kappa_0,\pi)=\rho^\gamma\big(d^i(q_0,\alpha_0),d^i(q_1,\alpha_1),\ldots\big)\le \beta^i,\qquad i=1,\dots,n_c.$$5

lead to optimal Markov policies in finite horizon and stationary optimal policies in infinite horizon under coherence, the Fatou property, and weighted contraction conditions (Bäuerle et al., 2020).

Risk-constrained discounted MDPs with coherent objective and coherent constraints produce a Bellman-style inequality system instead of a single fixed-point equation. The central optimization over value variables $$D_\gamma^i(\kappa_0,\pi)=\rho^\gamma\big(d^i(q_0,\alpha_0),d^i(q_1,\alpha_1),\ldots\big)\le \beta^i,\qquad i=1,\dots,n_c.$$6 and Lagrange multipliers $$D_\gamma^i(\kappa_0,\pi)=\rho^\gamma\big(d^i(q_0,\alpha_0),d^i(q_1,\alpha_1),\ldots\big)\le \beta^i,\qquad i=1,\dots,n_c.$$7 is

$$D_\gamma^i(\kappa_0,\pi)=\rho^\gamma\big(d^i(q_0,\alpha_0),d^i(q_1,\alpha_1),\ldots\big)\le \beta^i,\qquad i=1,\dots,n_c.$$8

subject to

$$D_\gamma^i(\kappa_0,\pi)=\rho^\gamma\big(d^i(q_0,\alpha_0),d^i(q_1,\alpha_1),\ldots\big)\le \beta^i,\qquad i=1,\dots,n_c.$$9

This is a Bellman optimization, not a standard DP recursion, and in general it lower-bounds the true constrained risk-averse problem (Ahmadi et al., 2020, Ahmadi et al., 2021).

CVaR planning in SSP MDPs yields yet another structural transformation. The CVaR objective can be written as a two-player zero-sum SSP stochastic game on an augmented state space

$z(s_t,a_t)$0

where the extra variable $z(s_t,a_t)$1 tracks the adversary’s remaining perturbation budget. The Bellman equation becomes

$z(s_t,a_t)$2

This stochastic-game representation is central to both pure CVaR optimization and the lexicographic CVaR-then-expectation refinement (Rigter et al., 2021).

4. Linear programming, occupation measures, and nonconvex optimization

The principal tractable backbone for constrained MDPs remains occupation-measure LP. In stochastic dominance-constrained MDPs, the average-reward constrained problem becomes the infinite-dimensional LP

$z(s_t,a_t)$3

subject to

$z(s_t,a_t)$4

The dominance operator is

$z(s_t,a_t)$5

so the continuum of dominance constraints becomes linear in the occupation measure $z(s_t,a_t)$6. The dual reveals modified average dynamic programming equations with an additional pricing term $z(s_t,a_t)$7, where the dual variable corresponds to an increasing concave utility function (Haskell et al., 2012).

Occupation measures also underlie total-cost AVaR planning after timeout truncation and state augmentation. With augmented-state occupancy measure

$z(s_t,a_t)$8

the distribution of terminal discretized cost is recovered by

$z(s_t,a_t)$9

and the risk-averse objective becomes

$Y$0

subject to linear flow constraints (Carpin et al., 2016). This is not a budget-constrained formulation, but it demonstrates how distributional tail criteria can be embedded into occupation-measure optimization.

Risk-sensitive constrained MDPs with exponential criteria admit a markedly different optimization structure. There, the key objects are policy-dependent forward factors

$Y$1

and backward factors

$Y$2

These induce a policy-level fixed-point condition

$Y$3

where $Y$4 is the solution set of a linear program $Y$5. The condition is necessary but not sufficient; exact solution requires a global search over fixed points or equivalent reformulations (Singh et al., 2022).

Several risk-constrained or risk-aware formulations become difference-of-convex programs. In constrained coherent-risk MDPs, the Bellman inequality optimization is written as a DCP because the risk transition mapping is convex in the continuation value (Ahmadi et al., 2020, Ahmadi et al., 2021). In CPT verification, the nonlinear probability-weighting function is approximated by a posynomial

$Y$6

which is decomposed into convex and concave parts and solved by convex-concave programming (Cubuktepe et al., 2018). In BR-MDPs, Bellman inequalities with variationally represented convex risk measures lead to bilevel DCPs over belief states and risk-measure parameters, then to approximate finite-posterior DCPs solved by DCCP-style machinery (Lin et al., 2023).

The chance-constrained planning algorithm RAlph replaces global LP solution by local LPs on a sampled search tree. The root action distribution is obtained from an LP over tree-flow variables $Y$7 maximizing predicted reward subject to a leaf-risk budget: $Y$8 This is not an occupation-measure LP on the full MDP but a local flow LP embedded inside MCTS (Brazdil et al., 2020).

5. Representative algorithmic paradigms

A concise taxonomy of algorithmic patterns emerges from the papers.

Paradigm	Core idea	Representative use
Occupation-measure LP	Linearize flow and risk constraints in measure space	Stochastic dominance constraints (Haskell et al., 2012); discretized AVaR total-cost planning (Carpin et al., 2016)
Bellman inequality optimization	Optimize value functions and multipliers under risk-transition constraints	Constrained coherent-risk MDPs (Ahmadi et al., 2020, Ahmadi et al., 2021)
Stochastic-game transformation	Encode CVaR as adversarial transition perturbation	CVaR planning and lexicographic CVaR-EV (Rigter et al., 2021)
Policy-level fixed point	Re-optimize one-step deviations using policy-dependent forward/backward factors	Exponential Risk-CMDP (Singh et al., 2022)
DCP / DCCP / CCP	Solve nonconvex risk Bellman problems by convex-concave iteration	Constrained coherent-risk MDPs (Ahmadi et al., 2020); CPT verification (Cubuktepe et al., 2018); BR-MDP approximation (Lin et al., 2023)
Search + local LP	Plan online with risk-feasible randomized root actions	Failure-probability chance-constrained planning (Brazdil et al., 2020)
Approximate dynamic programming	Replace exact risk backups by sample-based estimates	Large-scale coherent-risk MDPs (Yu et al., 2017)

The lexicographic CVaR-EV algorithm is especially notable because it isolates the slack left by pure CVaR optimization. It first solves the CVaR stochastic game to obtain a CVaR-optimal policy $Y$9, then builds a second augmented MDP with state $$Z_{\eta}\left(\pi,\nu\right)\geq\mathbb{E}\left[\left(Y-\eta\right)_{-}\right], \quad \forall\eta\in\left[a,b\right],$$0, where $$Z_{\eta}\left(\pi,\nu\right)\geq\mathbb{E}\left[\left(Y-\eta\right)_{-}\right], \quad \forall\eta\in\left[a,b\right],$$1 is accumulated cost, and constrains actions by

$$Z_{\eta}\left(\pi,\nu\right)\geq\mathbb{E}\left[\left(Y-\eta\right)_{-}\right], \quad \forall\eta\in\left[a,b\right],$$2

The resulting policy optimizes expectation among all continuations guaranteed to remain below the VaR cap, and execution switches to this second policy once the adversarial CVaR tail no longer places probability mass on the realized branch (Rigter et al., 2021).

The finite-horizon exponential Risk-CMDP algorithm GRC combines stochastic-approximation local updates

$$Z_{\eta}\left(\pi,\nu\right)\geq\mathbb{E}\left[\left(Y-\eta\right)_{-}\right], \quad \forall\eta\in\left[a,b\right],$$3

where $$Z_{\eta}\left(\pi,\nu\right)\geq\mathbb{E}\left[\left(Y-\eta\right)_{-}\right], \quad \forall\eta\in\left[a,b\right],$$4, with random restarts occurring with probability

$$Z_{\eta}\left(\pi,\nu\right)\geq\mathbb{E}\left[\left(Y-\eta\right)_{-}\right], \quad \forall\eta\in\left[a,b\right],$$5

The method searches globally over self-consistent policy fixed points because the fixed-point equation alone is not sufficient for optimality (Singh et al., 2022).

The approximate value iteration framework for coherent-risk MDPs uses empirical risk backups

$$Z_{\eta}\left(\pi,\nu\right)\geq\mathbb{E}\left[\left(Y-\eta\right)_{-}\right], \quad \forall\eta\in\left[a,b\right],$$6

followed by function fitting. This extends simulation-based ADP from expected-cost backups to coherent one-step risk backups, with finite-sample error decomposition into approximation and statistical terms (Yu et al., 2017).

In BR-MDPs with continuous belief spaces, ABDCP restricts the posterior set to a finite $$Z_{\eta}\left(\pi,\nu\right)\geq\mathbb{E}\left[\left(Y-\eta\right)_{-}\right], \quad \forall\eta\in\left[a,b\right],$$7, interpolates reachable posteriors using convex weights $$Z_{\eta}\left(\pi,\nu\right)\geq\mathbb{E}\left[\left(Y-\eta\right)_{-}\right], \quad \forall\eta\in\left[a,b\right],$$8, solves an approximate DCP over $$Z_{\eta}\left(\pi,\nu\right)\geq\mathbb{E}\left[\left(Y-\eta\right)_{-}\right], \quad \forall\eta\in\left[a,b\right],$$9, and returns a finite state controller with performance bounds

$$Z_{\eta}\left(\pi,\nu\right)\triangleq\liminf_{T\rightarrow\infty}\frac{1}{T}\mathbb{E}_{\nu}^{\pi}\left[\sum_{t=0}^{T-1}\left(z\left(s_{t},a_{t}\right)-\eta\right)_{-}\right].$$0

terminating when the controller-evaluation gap is below $$Z_{\eta}\left(\pi,\nu\right)\triangleq\liminf_{T\rightarrow\infty}\frac{1}{T}\mathbb{E}_{\nu}^{\pi}\left[\sum_{t=0}^{T-1}\left(z\left(s_{t},a_{t}\right)-\eta\right)_{-}\right].$$1 (Lin et al., 2023).

6. Empirical patterns, interpretive themes, and limitations

A consistent empirical theme is that expectation-optimal policies often achieve the best mean performance but noticeably worse tail or failure behavior, whereas strongly risk-averse methods improve safety or tail metrics at a computational or nominal-performance cost. In the lexicographic CVaR study, pure expected-value policies usually deliver the lowest expected cost but much worse tail risk; CVaR-only policies achieve the best CVaR but are conservative in expectation; the lexicographic CVaR-EV method closely matches CVaR-optimal tail risk while improving expected cost across inventory control, betting, deep sea treasure, and autonomous navigation domains (Rigter et al., 2021).

The total-cost AVaR study reports that a robot deployment policy optimized for $$Z_{\eta}\left(\pi,\nu\right)\triangleq\liminf_{T\rightarrow\infty}\frac{1}{T}\mathbb{E}_{\nu}^{\pi}\left[\sum_{t=0}^{T-1}\left(z\left(s_{t},a_{t}\right)-\eta\right)_{-}\right].$$2 of travel time reduced deadline-exceeding runs from 61 to fewer than 30 out of 1000 executions, compared with a risk-neutral policy. The result illustrates that direct tail-risk minimization can materially reshape the completion-time distribution, not merely its mean (Carpin et al., 2016).

In the rover navigation experiments for constrained coherent-risk MDPs, expectation-based policies are fastest to compute but incur larger empirical failure rates under terrain uncertainty, while CVaR and especially EVaR policies choose safer routes around obstacle-dense regions. The reported failure rates fall from $$Z_{\eta}\left(\pi,\nu\right)\triangleq\liminf_{T\rightarrow\infty}\frac{1}{T}\mathbb{E}_{\nu}^{\pi}\left[\sum_{t=0}^{T-1}\left(z\left(s_{t},a_{t}\right)-\eta\right)_{-}\right].$$3, $$Z_{\eta}\left(\pi,\nu\right)\triangleq\liminf_{T\rightarrow\infty}\frac{1}{T}\mathbb{E}_{\nu}^{\pi}\left[\sum_{t=0}^{T-1}\left(z\left(s_{t},a_{t}\right)-\eta\right)_{-}\right].$$4, and $$Z_{\eta}\left(\pi,\nu\right)\triangleq\liminf_{T\rightarrow\infty}\frac{1}{T}\mathbb{E}_{\nu}^{\pi}\left[\sum_{t=0}^{T-1}\left(z\left(s_{t},a_{t}\right)-\eta\right)_{-}\right].$$5 under expectation to $$Z_{\eta}\left(\pi,\nu\right)\triangleq\liminf_{T\rightarrow\infty}\frac{1}{T}\mathbb{E}_{\nu}^{\pi}\left[\sum_{t=0}^{T-1}\left(z\left(s_{t},a_{t}\right)-\eta\right)_{-}\right].$$6, $$Z_{\eta}\left(\pi,\nu\right)\triangleq\liminf_{T\rightarrow\infty}\frac{1}{T}\mathbb{E}_{\nu}^{\pi}\left[\sum_{t=0}^{T-1}\left(z\left(s_{t},a_{t}\right)-\eta\right)_{-}\right].$$7, and $$Z_{\eta}\left(\pi,\nu\right)\triangleq\liminf_{T\rightarrow\infty}\frac{1}{T}\mathbb{E}_{\nu}^{\pi}\left[\sum_{t=0}^{T-1}\left(z\left(s_{t},a_{t}\right)-\eta\right)_{-}\right].$$8 under CVaR, and to $$Z_{\eta}\left(\pi,\nu\right)\triangleq\liminf_{T\rightarrow\infty}\frac{1}{T}\mathbb{E}_{\nu}^{\pi}\left[\sum_{t=0}^{T-1}\left(z\left(s_{t},a_{t}\right)-\eta\right)_{-}\right].$$9, $$\pi^* \in \arg\min_{\pi\in\Pi} \mathrm{AVaR}_\tau(c(\omega)),$$0, and $$\pi^* \in \arg\min_{\pi\in\Pi} \mathrm{AVaR}_\tau(c(\omega)),$$1 under EVaR on $$\pi^* \in \arg\min_{\pi\in\Pi} \mathrm{AVaR}_\tau(c(\omega)),$$2, $$\pi^* \in \arg\min_{\pi\in\Pi} \mathrm{AVaR}_\tau(c(\omega)),$$3, and $$\pi^* \in \arg\min_{\pi\in\Pi} \mathrm{AVaR}_\tau(c(\omega)),$$4 grids, respectively (Ahmadi et al., 2020).

RAlph demonstrates that chance-constrained planning based on failure reachability can scale to very large spaces, including a hallway benchmark with about $$\pi^* \in \arg\min_{\pi\in\Pi} \mathrm{AVaR}_\tau(c(\omega)),$$5 states, by combining MCTS, learned reward/risk prediction, and local constrained LP selection. The paper’s practical message is that approximate empirical constraint satisfaction can be obtained in large MDPs where exact risk-constrained LPs are infeasible, albeit without formal guarantees once learned predictors are introduced (Brazdil et al., 2020).

The BR-MDP studies show a different pattern: risk is over epistemic model uncertainty rather than trajectory tail events. With small data, BR-MDP with CVaR risk in the objective often improves both expected and CVaR performance relative to nominal and distributionally robust baselines, while with large data the methods converge as posterior uncertainty shrinks (Lin et al., 2021, Lin et al., 2023). This suggests that the conservatism of robust MDPs can sometimes be replaced by posterior-weighted risk in a time-consistent way, although this remains an objective-based alternative rather than a hard constrained guarantee.

Several limitations recur. Static CVaR of total cost is powerful but not the same as a dynamic time-consistent risk measure, and methods built around it may require history dependence or augmented continuous states (Rigter et al., 2021, Carpin et al., 2016). Dynamic coherent-risk methods preserve Bellman structure but often yield nonconvex Bellman inequality programs or rely on strong Markov-risk assumptions (Ahmadi et al., 2020, Ahmadi et al., 2021, Bäuerle et al., 2020). Occupation-measure LPs can express rich constraints such as stochastic dominance, but they are infinite-dimensional in general Borel models (Haskell et al., 2012). Approximate implementations frequently depend on interpolation, discretization, or local DCCP solutions without global optimality guarantees (Rigter et al., 2021, Cubuktepe et al., 2018, Lin et al., 2023). Offline risk evaluation from logged data is statistically difficult, with error fundamentally controlled by the second moment of importance weights in finite-horizon MDPs (Huang et al., 2022).

A plausible implication is that there is no single dominant mathematical template for “risk-constrained MDPs.” The field instead comprises a layered hierarchy: explicit feasibility formulations such as chance, dominance, and coherent-risk constraints; risk-aware objectives such as CVaR, AVaR, recursive coherent risk, exponential utility, and Bayesian posterior risk; and specification- or semantics-driven constructions where risk is defined through verification, automata, or catastrophic-event structure. What unifies them is not one universal Bellman equation, but the repeated need to lift classical MDP machinery so that rare events, tail distributions, or uncertainty sets become first-class decision variables rather than residual properties of the expected return.