Distributional Risk-Sensitive Reinforcement Learning

• Distributional risk-sensitive RL is a framework that models full return distributions, rather than only expectations, to enable risk-aware decision-making using measures like CVaR and distorted expectations.
• It integrates various risk measures and objective classes into algorithmic designs such as actor-critic methods and Bellman recurrences for balancing safety and performance.
• Empirical studies demonstrate that these methods enhance robustness in applications ranging from robotics and autonomous driving to complex stochastic control environments.

Searching arXiv for recent and foundational papers on distributional risk-sensitive reinforcement learning. arxiv_search("distributional risk-sensitive reinforcement learning") Distributional risk-sensitive reinforcement learning is the branch of reinforcement learning in which an agent learns the full distribution of long-term return, rather than only its expectation, and then optimizes a risk-sensitive functional of that distribution. In this formulation, the central object is the random return $Z^\pi$, and risk is expressed through statistics or functionals of its distribution, such as lower-tail criteria, distortion-based expectations, entropic utility, or broader static risk functionals. The field has developed along two intertwined lines: practical algorithm design, where a learned return distribution is used to induce safer or more adaptive action selection, and theoretical work, where the validity of Bellman recursions, policy improvement, regret bounds, and time consistency are examined for specific risk criteria and control formulations (Théate et al., 2022, Liang et al., 2022, Zhou et al., 2023, Moghimi et al., 5 Jul 2025).

1. Distributional formulation of risk-sensitive control

Classical RL optimizes the expected discounted return, typically through a scalar action-value function $Q^\pi(s,a)$. Distributional RL replaces this scalar with a random return. A representative formulation is

$$Z^\pi(s,a)\stackrel{D}{=}\sum_{t=0}^{\infty}\gamma^t R(s_t,a_t),$$

with a corresponding distributional Bellman equation

$$Z^{\pi}(s,a)\stackrel{D}{=}R(s,a)+\gamma P^\pi Z^\pi(s,a).$$

In this view, the agent learns a distribution over possible consequences of each action, and risk-sensitive control becomes the problem of applying a risk functional to that learned distribution rather than collapsing it immediately to its mean (Théate et al., 2022).

This formulation is attractive because many risk criteria are distributional by construction. CVaR, VaR, distorted expectations, spectral risk measures, and entropic risk all depend on the shape of the return distribution, not merely on $\mathbb{E}[Z]$. Several works therefore treat distributional RL as the natural substrate for risk-sensitive RL: it yields access to the lower tail, multimodality, and dispersion of returns, and it permits action selection through a chosen risk functional instead of through expectation alone (Ma et al., 2020, Xiao et al., 2024).

A further development concerns static trajectory-wise risk objectives. In static risk-sensitive RL, the goal is to optimize the risk of the entire accumulated trajectory return from the initial state distribution, rather than to apply a local risk criterion at each intermediate state. Because such objectives are generally non-Markovian in the original MDP, several papers introduce augmented state constructions that track accumulated reward or a stock variable so that distributional dynamic programming can still be carried out in an augmented MDP (Chen et al., 2024, Moghimi et al., 4 Feb 2026).

2. Risk measures and objective classes

A prominent family of objectives is based on the lower tail of the return distribution. In one practical formulation, risk is taken to be the “worst part” of $Z^\pi$, selected by a tail parameter $\rho\in(0,1)$, with

$$\mathrm{VaR}_{\rho}(Z^{\pi}) = \inf \left\{ z \in \mathbb{R} : F_{Z^\pi}(z)\ge \rho \right\},$$

and

$$\mathrm{CVaR}_{\rho}(Z^\pi)=\mathbb{E}\!\left[z \mid z \le \mathrm{VaR}_{\rho}(Z^\pi)\right].$$

In this line of work, the learned return distribution is converted into a risk-sensitive action score by combining expectation and a tail statistic: $$U^{\pi}(s,a) = \alpha\,\mathbb{E}[Z^\pi(s,a)] + (1-\alpha)\,\mathcal{R}_\rho[Z^\pi(s,a)],$$ where $Q^\pi(s,a)$0 interpolates between risk-neutral and fully risk-averse behavior (Théate et al., 2022).

Another large family uses distortion functions on quantiles. In quantile-based methods such as IQN-style and DSAC-style formulations, one defines a distorted expectation by sampling quantile fractions $Q^\pi(s,a)$1, transforming them through a distortion $Q^\pi(s,a)$2 or $Q^\pi(s,a)$3, and integrating the resulting quantiles. This covers CVaR, Wang distortion, CPW, and related objectives. In DSAC, for example, a general risk map $Q^\pi(s,a)$4 acts on the soft return distribution $Q^\pi(s,a)$5 through $Q^\pi(s,a)$6, with concrete instantiations including percentile/VaR, mean-variance, and distorted expectation (Ma et al., 2020). Related navigation and locomotion methods use coherent distortion measures such as CVaR and power-law or Wang distortion, with a runtime risk parameter $Q^\pi(s,a)$7 controlling the policy’s risk attitude (Choi et al., 2021, Schneider et al., 2023).

A distinct theoretical line focuses on risk measures with structural properties that support dynamic programming or policy gradient analysis. One such case is the entropic risk measure

$Q^\pi(s,a)$8

with $Q^\pi(s,a)$9 risk-seeking, $$Z^\pi(s,a)\stackrel{D}{=}\sum_{t=0}^{\infty}\gamma^t R(s_t,a_t),$$0 risk-averse, and $$Z^\pi(s,a)\stackrel{D}{=}\sum_{t=0}^{\infty}\gamma^t R(s_t,a_t),$$1 recovering the risk-neutral mean. Another line studies static Lipschitz risk measures, defined through Lipschitz continuity in the CDF sup-norm, so that distributional estimation error directly controls risk error (Liang et al., 2022, Chen et al., 2024).

More recently, static Spectral Risk Measures have been used as a unifying class: $$Z^\pi(s,a)\stackrel{D}{=}\sum_{t=0}^{\infty}\gamma^t R(s_t,a_t),$$2 where $$Z^\pi(s,a)\stackrel{D}{=}\sum_{t=0}^{\infty}\gamma^t R(s_t,a_t),$$3 is a left-continuous, non-increasing risk spectrum integrating to $$Z^\pi(s,a)\stackrel{D}{=}\sum_{t=0}^{\infty}\gamma^t R(s_t,a_t),$$4. This formulation explicitly includes CVaR and Mean-CVaR as special cases and is used to distinguish static episode-level risk optimization from iterative, per-state insertion of a risk measure into a critic (Moghimi et al., 5 Jul 2025).

3. Algorithmic design patterns

One practical pattern is to keep the distributional learning machinery unchanged and change only how decisions are derived from the learned distribution. In “Risk-Sensitive Policy with Distributional Reinforcement Learning,” the critic continues to learn $$Z^\pi(s,a)\stackrel{D}{=}\sum_{t=0}^{\infty}\gamma^t R(s_t,a_t),$$5 exactly as in the base distributional RL algorithm, while action selection and Bellman target action choice are switched from $$Z^\pi(s,a)\stackrel{D}{=}\sum_{t=0}^{\infty}\gamma^t R(s_t,a_t),$$6 to $$Z^\pi(s,a)\stackrel{D}{=}\sum_{t=0}^{\infty}\gamma^t R(s_t,a_t),$$7. In the concrete RS-UMDQN-C instantiation, the distributional loss is unchanged, and $$Z^\pi(s,a)\stackrel{D}{=}\sum_{t=0}^{\infty}\gamma^t R(s_t,a_t),$$8 is computed from the learned CDF using expectation and $$Z^\pi(s,a)\stackrel{D}{=}\sum_{t=0}^{\infty}\gamma^t R(s_t,a_t),$$9 (Théate et al., 2022).

A second pattern integrates risk directly into actor-critic architectures. DSAC models the distribution of entropy-augmented returns,

$$Z^{\pi}(s,a)\stackrel{D}{=}R(s,a)+\gamma P^\pi Z^\pi(s,a).$$0

using quantile regression, while the actor is trained with SAC-style updates and can optimize risk-neutral or risk-sensitive criteria derived from the distribution (Ma et al., 2020). RC-DSAC extends this idea by conditioning both actor and critic on a risk parameter $$Z^{\pi}(s,a)\stackrel{D}{=}R(s,a)+\gamma P^\pi Z^\pi(s,a).$$1, so that the same trained navigation policy can be switched at runtime across a range of risk measures without retraining (Choi et al., 2021). DPPO follows an analogous principle for PPO: it trains a QR-DQN-style value distribution, applies a distortion risk metric such as CVaR or Wang to obtain $$Z^{\pi}(s,a)\stackrel{D}{=}R(s,a)+\gamma P^\pi Z^\pi(s,a).$$2, and uses this risk-sensitive value estimate in advantage computation and policy optimization (Schneider et al., 2023).

A third pattern concerns online adaptation of risk levels. ARA argues that fixed CVaR levels are suboptimal and uses Random Network Distillation error $$Z^{\pi}(s,a)\stackrel{D}{=}R(s,a)+\gamma P^\pi Z^\pi(s,a).$$3 as a state-dependent uncertainty signal, mapping it to a CVaR risk level through $$Z^{\pi}(s,a)\stackrel{D}{=}R(s,a)+\gamma P^\pi Z^\pi(s,a).$$4. In the reported locomotion experiments, this produced reduced failure rates by up to a factor of $$Z^{\pi}(s,a)\stackrel{D}{=}R(s,a)+\gamma P^\pi Z^\pi(s,a).$$5 and improved generalization performance by up to $$Z^{\pi}(s,a)\stackrel{D}{=}R(s,a)+\gamma P^\pi Z^\pi(s,a).$$6 relative to both risk-aware and risk-agnostic baselines (Schubert et al., 2021). DRL-ORA addresses the same issue through online adaptation driven by epistemic uncertainty, represented by an ensemble of $$Z^{\pi}(s,a)\stackrel{D}{=}R(s,a)+\gamma P^\pi Z^\pi(s,a).$$7 IQN-style heads, and selects the risk level by minimizing total variation of epistemic risk over time with a Follow-The-Leader / Follow-The-Perturbed-Leader style update (Wu et al., 2023).

A fourth pattern explicitly separates aleatory and epistemic uncertainty. SENTINEL-K uses an ensemble of categorical distributional Q-networks to represent epistemic uncertainty across learners and aleatory uncertainty within each learner’s return distribution, and then composes coherent risk measures across the two levels through a composite risk. The ensemble is aggregated with Follow The Regularised Leader before the composite risk is used for action selection (Eriksson et al., 2021). This suggests a broader tendency in the literature: distributional RL supplies the aleatory component, while ensembles or model uncertainty machinery supply an epistemic layer that can itself be risk-assessed (Wu et al., 2023).

4. Time consistency, Bellman optimality, and model validity

A central controversy is whether standard distributional Bellman optimality updates actually optimize the intended risk-sensitive objective. One line of work studies “conventional” CVaR in sequential settings and shows that naïvely applying the same local CVaR criterion at every stage can be time-inconsistent. That paper contrasts fixed CVaR with two time-consistent alternatives: precommitted CVaR, which anchors evaluation at the start state, and nested CVaR, which recursively composes one-step conditional risk measures (Gagne et al., 2021).

A stronger critique is made in “Is Risk-Sensitive Reinforcement Learning Properly Resolved?” The paper argues that existing RSRL methods based on risk-sensitive Bellman optimality operators do not achieve unbiased optimization and can not guarantee optimality or even improvements regarding risk measures over accumulated return distributions. Its proposed remedy, Trajectory Q-Learning, replaces state-only value distributions with history-dependent return distributions $$Z^{\pi}(s,a)\stackrel{D}{=}R(s,a)+\gamma P^\pi Z^\pi(s,a).$$8, restoring the trajectory information needed for the static objective and yielding policy improvement guarantees for general distortion risk measures (Zhou et al., 2023).

The same issue reappears in static SRM optimization. “Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learning” argues that the naïve definition $$Z^{\pi}(s,a)\stackrel{D}{=}R(s,a)+\gamma P^\pi Z^\pi(s,a).$$9 is not equivalent to optimizing the episode-level objective $\mathbb{E}[Z]$0 and may produce time-inconsistent or suboptimal policies. Its solution is to optimize a static risk functional of the initial return distribution through a bi-level actor-critic formulation in an extended state space (Moghimi et al., 5 Jul 2025).

Model learning introduces an additional validity question. “Distributional Model Equivalence for Risk-Sensitive Reinforcement Learning” proves that proper value equivalence, which suffices for optimal planning in the risk-neutral case, is not sufficient in the risk-sensitive case because many risk measures depend on the shape of the return distribution rather than its expectation. The paper therefore introduces distribution equivalence, which preserves the full return distribution and supports planning for any risk measure, and a practical statistical functional equivalence, which preserves chosen statistics of the distribution and thereby supports the corresponding risk measures (Kastner et al., 2023).

Time preference interacts with risk as well. “Decoupling Time and Risk: Risk-Sensitive Reinforcement Learning with General Discounting” argues that discounting should not be treated as a fixed background parameter when rich risk-sensitive objectives are optimized. By using general discount functions, stock augmentation, and non-stationary policies, it gives a time-consistent framework for optimizing functionals of return distributions under non-exponential discounting (Moghimi et al., 4 Feb 2026).

5. Theoretical frameworks and convergence results

For finite-horizon tabular MDPs, entropic risk has been a particularly important case because it admits a principled bridge between distributional RL and risk-sensitive dynamic programming. “Bridging Distributional and Risk-sensitive Reinforcement Learning with Provable Regret Bounds” shows that the entropic risk measure satisfies the independence property, which is sufficient to recover a valid backward recursion over return distributions. On that basis it proposes RODI-MF and RODI-MB, two optimistic distributional algorithms, and proves regret upper bounds of

$\mathbb{E}[Z]$1

together with a tighter minimax lower bound for the $\mathbb{E}[Z]$2 case (Liang et al., 2022).

For general coherent risk measures, a policy-gradient route has also been developed. “Policy Gradient Methods for Risk-Sensitive Distributional Reinforcement Learning with Provable Convergence” derives an analytical gradient of the probability measure $\mathbb{E}[Z]$3 associated with the cumulative cost distribution, then turns this into gradients of coherent risk measures such as CVaR and mean-semideviation. Its practical categorical distributional policy gradient algorithm, CDPG, uses a finite-support approximation and comes with a finite-support optimality guarantee and a finite-iteration convergence guarantee under exact policy evaluation (Xiao et al., 2024).

A more general regret-based theory is given by “Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximation.” That framework covers static Lipschitz risk measures and introduces two meta-algorithms: RS-DisRL-M for model-based function approximation and RS-DisRL-V for value-function approximation. By coupling augmented MDP constructions with distributional estimation via Least Squares Regression or Maximum Likelihood Estimation, the paper derives the first $\mathbb{E}[Z]$4 dependency of the regret upper bound for this general setting (Chen et al., 2024).

Static episode-level risk optimization with actor-critic methods has recently been formalized for Spectral Risk Measures. The AC-SRM, TD3-SRM, TD3BC-SRM, and OAC-SRM family optimize a bi-level objective in which the outer step updates a concave risk function $\mathbb{E}[Z]$5 induced by the current return distribution and the inner step optimizes the policy for fixed $\mathbb{E}[Z]$6. In the finite state-action setting, the paper proves inner-loop convergence under Natural Policy Gradient updates and monotonic improvement of the full alternating scheme toward a stationary point (Moghimi et al., 5 Jul 2025).

6. Empirical domains and observed behavior

Empirical work has concentrated on settings where expected return is a poor summary of decision quality. In toy benchmarks designed so that risk-neutral and risk-sensitive optima differ, RS-UMDQN-C learns policies with slightly lower mean return but much lower tail risk than DQN, and its learned return distributions remain well-structured and multimodal (Théate et al., 2022). In stochastic Cliffwalk, distributional policy-gradient optimization of CVaR converges to the safe path with zero failures, whereas the risk-neutral policy converges to the shorter but riskier route (Xiao et al., 2024).

Robotics and navigation are recurring application areas. RC-DSAC was developed for partially observed robot navigation and reported fewer collisions than SAC, while allowing runtime changes of the risk parameter $\mathbb{E}[Z]$7 without retraining (Choi et al., 2021). DRL-ORA was evaluated on Nano Drone Navigation and Knapsack, where online adaptation of epistemic risk outperformed fixed-risk IQN and manually designed schedules, especially in higher-uncertainty settings (Wu et al., 2023). In legged robotics, DPPO produced emergent risk-sensitive locomotion in simulation and on ANYmal: risk-averse policies descended stairs reliably and refused dangerous obstacles, while risk-seeking policies attempted more aggressive traversals without reward retuning (Schneider et al., 2023).

Autonomous driving work has emphasized aleatoric uncertainty induced by other agents. In intersection-crossing scenarios, QRDQN combined with online risk assessment through CVaR or Wang distortion reduced collisions relative to DQN while maintaining an active driving style. In the reported averages over all mixed-uncertainty scenarios, DQN had $\mathbb{E}[Z]$8 collisions, QRDQN had $\mathbb{E}[Z]$9, and QRDQN + CVaR had $Z^\pi$0, with CVaR identified as the more suitable risk measure for that setting (Bernhard et al., 2021).

Operations research and robustness under shift constitute another active domain. A risk-sensitive discrete SAC algorithm based on the entropic risk measure was proposed for contextual multi-stage stochastic combinatorial optimization under distribution shifts. In a $Z^\pi$1 grid environment with twelve shifted test distributions, moderate risk-aversion around $Z^\pi$2 improved robustness and often also improved performance on the training distribution, outperforming risk-neutral SAC and two benchmark approaches for robust deep RL (Enders et al., 2024). This suggests that distributional risk-sensitive objectives can serve not only as safety devices but also as robustness mechanisms under realistic train-test mismatch.

Taken together, the literature portrays distributional risk-sensitive RL as a family of methods rather than a single recipe. One strand uses the learned return distribution as a post hoc decision object; another embeds risk directly into actor-critic or policy-gradient updates; a third addresses state-dependent or online adaptation of risk; and a growing theoretical literature distinguishes static episode-level objectives from iterative Bellman-style surrogates. The unifying idea is stable across these variants: risk is treated as a functional of the full return distribution, and distributional RL provides the representation needed to make that functional operational in sequential decision-making (Théate et al., 2022, Zhou et al., 2023, Moghimi et al., 5 Jul 2025, Moghimi et al., 4 Feb 2026).