Quantifying Risk Under Evolving Uncertainty: Belief-Dependent Robustness for Safe Sequential Decision Making

Abstract: How cautious should an agent be while it is still learning its environment? We propose RATTL (Risk-Adversarial Total-Reward Learning), which ties caution to epistemic uncertainty: the agent holds a Bayesian posterior over unknown dynamics and plans against a Wasserstein ambiguity set whose radius is a monotone function of that posterior. The radius contracts with evidence, so behaviour interpolates continuously between worst-case robustness and risk-neutral total-reward maximization. The design follows the duality underlying the Entropic Value-at-Risk, which converts the choice of a risk level into the choice of an ambiguity radius. We show the resulting planning problem is well posed under transience and compactness conditions, and prove a Safety Sandwich: the RATTL value lies between the uninformed robust value and the full- knowledge optimum, with a gap that vanishes as the posterior concentrates. In a canonical binary-hazard instance, the induced criterion reduces to Conditional Value-at-Risk at a level set by the posterior entropy. A worked example shows the agent deferring the efficient action until a sharp identification threshold. RATTL targets runtime safety for agents, including LLM-based systems, acting under uncertainty.

• Background: What is Robust Markov Decision Processes?.
• Background: What are Bayesian RL?.
• Definition: Robust Markov Decision Processes (RMDPs) are a class of decision-making frameworks where the agent operates under inherent uncertainty, particularly epistemic uncertainty, concerning the environment's dynamics. These processes extend classical MDPs by incorporating a set of possible models that capture this uncertainty. In a robust setting, the agent maximizes the minimum expected reward across these models, ensuring robustness against the worst-case scenario. To quote Wikipedia: Markdown and MathJax support is assisted.
• The paper proposes RATTL (Risk-Adversarial Total-Reward Learning) framework which examines risk attitude in decision-making scenarios where perceptions evolve during tasks
• The RATTL methodology effectively balances conservatism through a belief-dependent approach to uncertainty, adapting conservatism based on the evolving knowledge of the environment. The Wasserstein measure is more conservative than KL divergence and has stronger properties, ensuring better tail risk management in scenarios like the cliff transition.
• This approach is characterized by an ambiguity set that adjusts based on the entropy of the agent's beliefs, enabling more finely tuned risk management.
• The method includes robust MDP guarantees such as convergence, contractivity, and safety sandwich. These help approximate the expected hitting time across certain uncertainties.
• RATTL demonstrates explicit risk measures like how adversaries can make this model conservative.
• The Entropy-modulated Wass${\text{Kantorovich-Rubinstein}}$-ball ambiguity sets efficiently update as new information arrives, dynamically adjusting to the current level of uncertainty, and optimizing expected cumulative reward.
• The definition matches CVaR for discrete outcomes; ambiguity is belief-dependent and connected to normalized probabilities.
• Use-feedback algorithm: RATTL's feasibility for small state spaces and its adaptation to epistemics makes it potentially practical for real-time systems such as autonomous vehicles or risk management in financial markets, especially when uncertainty needs to be dynamically evaluated and acted upon.

Motivation: risk attitude as a function of what is still unknown

The paper addresses a gap between robust MDPs and Bayesian RL: an agent acting while identifying its environment needs a risk attitude that varies with residual epistemic uncertainty. Uniform worst-case reasoning is overly conservative once the environment is largely known, while planning against the belief average can be unsafe when uncertainty is high. The authors propose RATTL (Risk-Adversarial Total-Reward Learning), which makes conservatism endogenous: the size of the agent's adversarial ambiguity set shrinks as the Bayesian belief over environment types sharpens. The conceptual bridge is the Entropic Value-at-Risk (EVaR), whose KL-distributionally-robust dual recasts "how cautious should I be?" as "how large should my ambiguity ball be?" (2608.17574). RATTL departs from the RMDP-to-stochastic-game reduction of prior work by compressing a partially observed game into a subjective robust MDP whose uncertainty set is non-stationary and belief-dependent.

Framework

The setting is a partially observed turn-based stochastic game (PO-TBSG) with finite states, actions, opponent types $\mathcal{Z}$, per-type transition kernels, bounded rewards, and the undiscounted total-reward (stochastic shortest path) criterion. The agent maintains a belief $b\in\Delta(\mathcal{Z})$ with the standard Bayes update. The core object is the entropy-modulated Wasserstein ambiguity set:

$$U(b\mid s,a)=\{Q\in\Delta(\mathcal{S}) : W_1(Q, P_b(\cdot\mid s,a)) \le \beta H(b)\},$$

where $P_b$ is the belief-averaged kernel, $H(b)$ the Shannon entropy, and $\beta>0$ a sensitivity constant. The induced Nash-robust Bellman operator has the adversary minimize expected next value inside this budget, with the belief update applied inside the expectation so that kernel selection couples to future information.

The choice of Wasserstein rather than KL balls is argued on safety grounds, and supported numerically: on a five-state cliff transition where the catastrophe carries zero nominal mass ($\varepsilon=0.25$, nominal value $+57.5$), the KL adversary places zero mass on the off-support catastrophe and reports a worst case of only $-3.4$; total variation reaches $-217.5$; Wasserstein transports mass onto the catastrophe at finite cost and reports $b\in\Delta(\mathcal{Z})$0. This "support catastrophe" is the paper's strongest empirical argument: precisely when the nominal model becomes confident, KL-based hedging loses its ability to price tail events, whereas optimal transport does not.

Three assumptions carry the theory beyond bare properness: properness (a.s. termination under all policies and kernels), uniform reachability (termination within $b\in\Delta(\mathcal{Z})$1 steps with probability at least $b\in\Delta(\mathcal{Z})$2 uniformly over beliefs and adversaries), and identifiability (types are distinguishable). The authors note that a.s. reachability alone does not yield a uniformly bounded expected hitting time over the belief continuum—Assumption 2 is a genuine strengthening forced by the belief augmentation, not boilerplate.

Guarantees: contraction, Safety Sandwich, convergence

Contractivity. Under properness plus uniform reachability, the worst-case expected hitting time satisfies $b\in\Delta(\mathcal{Z})$3, and the Bellman operator is a contraction of modulus $b\in\Delta(\mathcal{Z})$4 in the weighted sup-norm on the affine space fixing terminal values. This yields a unique fixed point and geometric value iteration despite the non-stationary, belief-coupled ambiguity sets—an extension of standard SSP contraction arguments that must handle the belief continuum.

Safety Sandwich. For all $b\in\Delta(\mathcal{Z})$5,

$b\in\Delta(\mathcal{Z})$6

where $b\in\Delta(\mathcal{Z})$7 uses the radius frozen at $b\in\Delta(\mathcal{Z})$8 and $b\in\Delta(\mathcal{Z})$9 is the fully informed type-$$U(b\mid s,a)=\{Q\in\Delta(\mathcal{S}) : W_1(Q, P_b(\cdot\mid s,a)) \le \beta H(b)\},$$0 optimum. The lower bound follows from set inclusion; the upper bound from the fact that the nominal kernel is feasible in the ambiguity set, combined with the tower property showing the nominal-with-posterior process equals the mixture process. Notably, the authors explicitly refute the stronger ceiling $$U(b\mid s,a)=\{Q\in\Delta(\mathcal{S}) : W_1(Q, P_b(\cdot\mid s,a)) \le \beta H(b)\},$$1 at the realized true type: the true kernel generally lies outside the ambiguity ball for non-point-mass beliefs. The sandwich brackets value but provides no runtime behavioral certificate—a point the paper concedes directly in its applied discussion.

Convergence to best response. Under persistent identification (each type pair's distinguishing action visited infinitely often), the belief converges a.s. to a point mass on the true type, ambiguity collapses in Hausdorff distance, and $$U(b\mid s,a)=\{Q\in\Delta(\mathcal{S}) : W_1(Q, P_b(\cdot\mid s,a)) \le \beta H(b)\},$$2 converges to the informed optimum. With persistent excitation, $$U(b\mid s,a)=\{Q\in\Delta(\mathcal{S}) : W_1(Q, P_b(\cdot\mid s,a)) \le \beta H(b)\},$$3, giving a price-of-robustness rate of $$U(b\mid s,a)=\{Q\in\Delta(\mathcal{S}) : W_1(Q, P_b(\cdot\mid s,a)) \le \beta H(b)\},$$4. The delicate step—continuity of $$U(b\mid s,a)=\{Q\in\Delta(\mathcal{S}) : W_1(Q, P_b(\cdot\mid s,a)) \le \beta H(b)\},$$5 at point-mass beliefs, since no per-belief fixed-point operator exists due to the posterior shift—is handled via an invariant class of Lipschitz-in-belief value functions, contingent on the Bayes normalizer being bounded below near $$U(b\mid s,a)=\{Q\in\Delta(\mathcal{S}) : W_1(Q, P_b(\cdot\mid s,a)) \le \beta H(b)\},$$6; the paper flags that a belief-modulus contraction condition enters only here.

Tractability. By Kantorovich–Rubinstein duality, the inner infimum reduces to a one-dimensional dual over the transport multiplier $$U(b\mid s,a)=\{Q\in\Delta(\mathcal{S}) : W_1(Q, P_b(\cdot\mid s,a)) \le \beta H(b)\},$$7, so value iteration over a belief grid solves $$U(b\mid s,a)=\{Q\in\Delta(\mathcal{S}) : W_1(Q, P_b(\cdot\mid s,a)) \le \beta H(b)\},$$8 LPs of size $$U(b\mid s,a)=\{Q\in\Delta(\mathcal{S}) : W_1(Q, P_b(\cdot\mid s,a)) \le \beta H(b)\},$$9 per sweep. The continuous-state case is left open.

Risk-measure identity: coherent, Lipschitz-regularized, and exactly CVaR in two points

The paper characterizes what risk measure a Wasserstein ball induces. In general finite-state form, the inner worst case $P_b$0 is coherent in the Artzner sense and equals a Lipschitz-regularized expectation via inf-convolution duality, sharpening the Derman–Mannor regularization connection into an explicit coherent-risk identity.

On the canonical two-point catastrophe instance, the identity is exact and, notably, yields CVaR rather than EVaR: with good/catastrophe values, nominal catastrophe mass $P_b$1, and ground distance $P_b$2, the worst-case catastrophe mass is $P_b$3, and

$P_b$4

Since $P_b$5, the entropy dial is literally a CVaR-tail dial: as belief confidence rises, $P_b$6 increases toward risk-neutrality. Two caveats are stated plainly: for three or more states the worst case spreads mass to nearest low-value states, producing a $P_b$7-weighted "transport-CVaR" rather than ordinary CVaR; and no closed-form matching EVaR level exists, so the EVaR comparison is only one-sided ($P_b$8). The general transport-CVaR-to-EVaR comparison for $P_b$9 is identified as the headline open problem.

Worked example: the Ambiguous Bridge

A diagnostic with states {bridge, goal, fall}, actions {Sprint, Crawl}, and two types (benign/adversarial) admits fully closed-form robust Q-values. Sprint costs $H(b)$0 but risks a $H(b)$1 fall under the adversarial type; Crawl costs $H(b)$2 and always succeeds. The safety switch occurs at

$H(b)$3

with the $H(b)$4 terms canceling analytically. The threshold is determined purely by the reward asymmetry and is $H(b)$5-independent in this symmetric environment—$H(b)$6 shifts the threshold only when safe and risky actions face different ambiguity. Well-posedness requires $H(b)$7. The example instantiates the Safety Sandwich concretely: $H(b)$8 rises monotonically from the never-shrinking floor of $H(b)$9 toward the best-response ceiling of $\beta>0$0, staying provably between them. The authors are careful to state these are closed-form evaluations, not learning runs.

Position relative to prior work

Relative to static rectangular RMDPs, RATTL's contribution is the non-stationary, belief-driven radius. Against closest competitors—Russel & Petrik's policy-adapted sets, Choi & Li's contracting interval credible sets, Nakao et al.'s DR-POMDPs with static distance-based ambiguity—the claimed novelty is coupling a Wasserstein radius specifically to Shannon belief entropy, together with the Safety Sandwich and the two-point CVaR identity, neither of which appears in prior work. Ni & Bhat's result that stationary policies suffice for EVaR total-reward MDPs justifies RATTL's policy class, though at a fixed risk level rather than a belief-selected one.

Limitations and open questions

The paper is explicit about scope. The exact algorithm discretizes the belief simplex and is feasible only for small $\beta>0$1; there is no large-scale empirical validation. The guarantees require more than bare properness—uniform reachability for contraction, persistent identification/excitation for convergence and rate—so degenerate exploration is excluded. The exact CVaR identity holds only for the two-point catastrophe. Calibration requires $\beta>0$2, and the $\beta>0$3-independence of the switch threshold is specific to symmetric environments. Open problems include the general transport-CVaR/EVaR comparison for $\beta>0$4, hybrid Wasserstein-plus-KL ambiguity, alternative information measures (Rényi/Tsallis) and whether the Sandwich survives them, sample complexity of belief-adaptive robust RL, scaling the dual to continuous spaces via Lipschitz critics, and multi-agent composition under private beliefs. The stated application target—runtime safety for agentic GenAI systems invoking tools under latent-context uncertainty—currently receives a value-level bracket rather than an actionable runtime certificate; converting the sandwich into such a certificate and validating it on real LLM-agent pipelines remains untested.

Conclusion

RATTL formalizes the thesis that robustness should decrease with epistemic certainty by tying a Wasserstein ambiguity radius to Shannon belief entropy. Its contributions are a contraction theorem over the belief-augmented SSP space, a Safety Sandwich bracketing value between maximin and Bayesian-best-response ceilings with explicit conditions, an almost-sure convergence guarantee with an $\beta>0$5 price of robustness, and a partial risk-measure identification—coherence and Lipschitz regularization in general, exact entropy-controlled CVaR on the canonical catastrophe—with the Ambiguous Bridge providing a sharp, interpretable safety switch at $\beta>0$6.