Constrained Risk-Averse MDPs

• Constrained risk-averse MDPs are extensions of classical MDPs that use coherent risk measures (e.g., CVaR, VaR) and explicit constraints to guard against severe, low-probability events.
• The framework relies on DCP, LP, and MIP formulations along with recursive risk mappings to enable tractable optimization and robust policy synthesis.
• Applications in robotic deployment, inventory management, and autonomous navigation demonstrate significant reductions in failure rates and enhanced risk profiles under uncertainty.

Constrained risk-averse Markov decision processes (MDPs) extend the classical MDP paradigm by incorporating risk-sensitive objectives and explicit constraints to account for the possibility of severe, low-probability events that can lead to catastrophic costs or failures. This is achieved by replacing or augmenting the traditional expectation-based performance criteria with risk measures—such as value-at-risk (VaR), conditional value-at-risk (CVaR), average value-at-risk (AVaR), entropic value-at-risk (EVaR), and cumulative prospect theory (CPT) distortions—and by enforcing constraints either on expected values, risk metrics, or failure probabilities. The resulting synthesis and verification problems span linear programming (LP), difference-of-convex programming (DCP), and mixed-integer programming (MIP), depending on the structure of the risk measures and constraints. Constrained risk-averse MDPs provide both provable risk profiles and practical guarantees about the distributional properties of random cost or reward, making them essential for mission-critical planning under uncertainty.

1. Formal Model and Problem Definition

A constrained risk-averse MDP is defined as a tuple $\mathcal{M} = (X, U, P, c)$, where:

• $X$ is the finite set of states,
• $U(x)$ is the set of admissible actions in $x$,
• $P(x' \mid x, u)$ specifies the transition probability from $x$ to $x'$ under $u$,
• $c: X \times U \to \mathbb{R}_{\ge 0}$ is the stage cost function.

A policy $\pi = (\pi_t)_{t \ge 0}$, possibly nonstationary and history-dependent, induces a cost random variable $X$0. Risk-averse optimization replaces or complements the standard $X$1 objective with a coherent risk measure $X$2, resulting in constraints and objectives such as minimization of AVaR, CVaR, or even more general measures (e.g., those induced by CPT or distributional robustness over ambiguity sets). Constraints may include bounds on additional risk quantifications or on probabilities of reaching failure states (Carpin et al., 2016, Ahmadi et al., 2020, Ahmadi et al., 2021, Brazdil et al., 2020, Křetínský et al., 2018).

2. Dynamic Risk Measures and Markov Compatibility

Risk measures employed in risk-averse MDPs are typically required to be coherent—convex, monotone, translation-invariant, and positively homogeneous—and to admit recursive, time-consistent nested composition. Formally, a dynamic coherent risk measure is constructed through nesting:

$X$3

with discount factor $X$4. Markov compatibility (existence of a risk transition mapping $X$5) enables recursive Bellman-style formulations:

$X$6

where, for example, the CVaR risk mapping is realized as

$X$7

This structure is crucial for obtaining tractable DCP or LP reformulations and is highlighted in modern frameworks for constrained risk-averse MDPs (Ahmadi et al., 2020, Ahmadi et al., 2021).

3. Optimization-Based Solution Methods

Risk-averse MDPs with constraints are solved via the following core methodologies:

Difference-of-Convex Programming (DCP) and Disciplined Convex-Concave Programming (DCCP)

When risk objectives and constraints can be represented via Markov transition risk mappings, the optimization reduces to a DCP:

$X$8

DCCP proceeds by linearizing the concave part $X$9 at each iteration, yielding a sequence of convex programs until convergence to a saddle point. This captures both policy synthesis (primal) and constraint satisfaction/enforcement (dual via $U(x)$0). When the risk measure is expectation, this reduces to classical LP for constrained expected-cost MDPs (Ahmadi et al., 2020, Ahmadi et al., 2021).

Mixed-Integer and Linear/Bilinear Programming

For VaR, CVaR, or AVaR constraints on total cost or mean payoff, LP and MIP formulations are employed. For example, when minimizing AVaR of the total cost,

$U(x)$1

the problem can be recast as a bilinear program (linear for fixed $U(x)$2) using state-augmentation and surrogate timeouts (Carpin et al., 2016). Chance-constraint enforcement via VaR for model parameter uncertainty leads to mixed-integer LPs with big-$U(x)$3 constraints to encode scenario quantile satisfaction (Merakli et al., 2019).

SOCP and Proximal Algorithms for Ambiguity

Distributionally robust (Wasserstein) ambiguity over rewards or transitions leads to SOCP formulations. For instance, maximizing a weighted combination of worst-case mean and percentile under reward ambiguity reduces to:

$U(x)$4

enabling large-scale optimization via first-order methods such as AD-LPMM (Ruan et al., 2023).

4. Complexity and Memory Requirements

Single-dimensional, expectation/CVaR/VaR-constrained reachability and mean-payoff problems admit polynomial-time LP solutions, with memoryless randomized strategies often sufficient due to the convexity properties of the objective and constraints. Multi-dimensional versions become NP-hard or even EXPSPACE-complete when coupling between constraints or non-linearities from risk operators (such as CVaR in several dimensions) are introduced. Infinite memory and randomization may be required for mean-payoff problems with multiple objectives (Křetínský et al., 2018). Heuristic and structural simplifications (such as monotonicity assumptions) can yield significant computational improvements in large-scale or scenario-based optimization (Merakli et al., 2019).

5. Applications and Empirical Evaluation

Constrained risk-averse MDPs have been applied to domains where tail risk control is critical:

• Robotic rapid deployment: AVaR-minimizing policies drastically reduce $U(x)$5 (e.g., over 50% reduction relative to expectation-optimal policies) while controlling overall completion time (Carpin et al., 2016).
• Inventory management for humanitarian relief: VaR-optimized policies mitigate the risk of high long-term cost under uncertain demand/supply parameters, with deterministic and randomized MIP formulations used in large-scale problem instances (Merakli et al., 2019).
• Autonomous rover navigation: Policies synthesized under CVaR or EVaR constraints display lower empirical failure rates than those computed via risk-neutral methods, at the expense of increased nominal cost (Table below; EVaR policies achieve zero or near-zero collision rates) (Ahmadi et al., 2020).

Problem	Risk Measure	Value $U(x)$6	Failure Rate
Grid 10×10	Expected	5.10	9%
Grid 10×10	CVaR$U(x)$7	≥7.76	1%
Grid 10×10	EVaR$U(x)$8	≥7.99	0%

This approach also enables full risk profiling (extraction of the entire cost distribution) and explicit visualization of $U(x)$9 for all $x$0 (Carpin et al., 2016).

6. Extensions: Distributional and Temporal Logics

Distributionally robust and parameter-ambiguous models extend the classical risk-averse MDP by considering ambiguity sets for reward and transition law (e.g., Wasserstein balls). Optimal policies hedge against both mean and tail under worst-case distributions, with tractable SOCP or MISOCP formulations and scalable first-order solvers (Ruan et al., 2023).

Policies with temporal logic constraints and risk-sensitive value functions (e.g., with CPT distortion) are synthesized by translating temporal logic formulas to automata, forming product MDPs, and casting the resulting policy optimization as a difference-of-convex program solvable via the convex–concave procedure (CCP). This enables specification and enforcement of probability or risk bounds on complex, temporally extended objectives (Cubuktepe et al., 2018).

7. Theoretical Guarantees and Open Challenges

The risk-averse, constraint-based Bellman programs provide explicit suboptimality bounds (e.g., via surrogate timeouts), error control via discretization, and convergence of the optimization process to locally optimal, feasible solutions (Carpin et al., 2016, Ahmadi et al., 2020). The LP and DCP frameworks accommodate classical results as special cases.

Open research directions include tractable incorporation of general coherent or utility-based risk measures under ambiguity; online/model-free extensions compatible with RL (policy gradient, actor-critic); multi-agent and partially observable generalizations; and refined finite-sample statistical calibration for robust, high-confidence policy synthesis (Ruan et al., 2023, Ahmadi et al., 2021).

---

References:

• (Carpin et al., 2016) "Risk Aversion in Finite Markov Decision Processes Using Total Cost Criteria and Average Value at Risk"
• (Ahmadi et al., 2020) "Constrained Risk-Averse Markov Decision Processes"
• (Ahmadi et al., 2021) "Risk-Averse Decision Making Under Uncertainty"
• (Ruan et al., 2023) "Risk-Averse MDPs under Reward Ambiguity"
• (Merakli et al., 2019) "Risk Aversion to Parameter Uncertainty in Markov Decision Processes with an Application to Slow-Onset Disaster Relief"
• (Brazdil et al., 2020) "Reinforcement Learning of Risk-Constrained Policies in Markov Decision Processes"
• (Křetínský et al., 2018) "Conditional Value-at-Risk for Reachability and Mean Payoff in Markov Decision Processes"
• (Cubuktepe et al., 2018) "Verification of Markov Decision Processes with Risk-Sensitive Measures"