Auditing the Risk Claims of Distributional Reinforcement Learning

Abstract: Distributional reinforcement learning agents learn full return distributions that are increasingly read at face value: for interpretability, risk-sensitive control, and safety monitoring. We ask a question theory anticipates but that has not been measured directly: are the risk claims of a trained distributional agent true? Our audit combines a decision-relevant screening metric (the excess Wasserstein gap between the top two actions, which equals the mass by which first-order stochastic dominance is violated), ground truth from snapshot-restart Monte Carlo, and a statistical harness (permutation nulls, bootstrap refutation, FDR control) without which the audit itself manufactures false conclusions. Across QR-DQN, C51, and IQN on MinAtar (33 runs), 40-95% of the strongest claimed risk trade-offs are refuted at 95% confidence, the placement of the strongest claims is statistically indistinguishable from truth-blind, and essentially no claim is confirmable: for these agents, the learned "risk" reflects a training artifact rather than environment stochasticity. The artifact is structural (fully formed early in training, uncorrelated with final score, idiosyncratic to each seed) and appears unchanged at full-Atari scale, with every top Breakout claim of a pretrained near-state-of-the-art QR-DQN refuted. Positive controls of known magnitude confirm 96-100% of real claims (correlation 0.89-0.92): the reading measures the agents, not the audit. Acting on the heads' CVaR advice at their most-flagged states ranges from beneficial to significantly worse than chance. Neither training for risk nor ensembling removes the artifact, and recalibration passes the audit only by nullifying the claims: the head is uninformative, not merely miscalibrated. We release the toolkit and document two silent pitfalls that produced convincing but wrong audits of our own.

• The paper introduces the excess Wasserstein gap metric to assess whether predicted risk differences align with real-world decision scenarios.
• It finds that between 40–95% of strong risk claims in popular models are statistically unsubstantiated, highlighting systematic overclaiming.
• Empirical tests across MinAtar and Atari environments reveal that the calibration failure of risk estimates undermines their use in safety-critical applications.

Auditing Risk Claims in Distributional Reinforcement Learning

Introduction

Distributional reinforcement learning (RL) architectures such as C51, QR-DQN, and IQN have seen widespread deployment, driven by their capacity to model return distributions rather than merely expectations. Such models are frequently interpreted as providing reliable, actionable risk estimates for interpretability, risk-sensitive control, and safety monitoring. Despite theoretical skepticism regarding their calibration and decision-relevance, the truthfulness of their risk-related claims has not been systematically audited in situ. This paper provides a decisive, decision-level evaluation of whether and when the risk differentials reported by distributional heads correspond to empirically verifiable risk structures in the environment (2607.11607).

Audit Metric: The Excess Wasserstein Gap

The paper centers its evaluation on a principled, decision-relevant statistic: the excess Wasserstein gap between the agent's top two actions at a state. This is defined as the $W_1$ distance (Wasserstein-1) between their predicted return distributions, minus their absolute mean difference. By Proposition~1, this value is strictly positive if and only if neither action first-order stochastically dominates the other—i.e., exactly when a risk-sensitive policy may rationally deviate from a mean-greedy policy. The audit metric thus filters out irrelevant errors, targeting only risk trade-offs that would substantively influence downstream decisions.

Methodology

The core audit procedure is as follows:

• States are sampled and ranked by the head’s excess Wasserstein gap.
• For each, ground truth return distributions are obtained via massive snapshot-restart Monte Carlo rollouts (thousands per action), capturing the full aleatoric variability under greedy policies.
• Statistical significance of risk claims is assessed by comparing the agent’s predicted gap to a bootstrap-based null calibrated precisely per state. Confirmation or refutation is controlled via permutation testing and FDR correction. Crucially, the procedure identifies not only error magnitudes, but also whether purported risk differences actually exist—mitigating pitfalls seen in prior measurement practices such as miscalibration due to RNG cloning (P1) or empirical Wasserstein estimation noise (P2).

Figure 1: Measurement of Pitfall P2: the empirical $W_1$ floor for two $n$-samples from the same distribution scales as $n^{-1/2}$ and can substantially inflate apparent risk claims.

Empirical Findings

Audits on QR-DQN, C51, and IQN were conducted over 33 MinAtar runs on Breakout, Seaquest, and Asterix, as well as on an off-the-shelf, near-state-of-the-art full-scale Atari agent. The findings are unambiguous:

• 40–95% of the top 2% most-confident risk trade-offs asserted by distributional RL agents are statistically refuted; that is, the claimed difference is far larger than any realized in ground-truth empirical rollouts.
• The placement of the strongest claims is statistically indistinguishable from random allocation (i.e., no informative power beyond claim magnitude).
• No claim is confirmable after multiple testing correction, and the learned-truth correlation for these risk scores is effectively zero.

Figure 2: MinAtar agent training curves indicate robust baseline performance—audit results are within-agent, agnostic to cross-method score differences.

Figure 3: Top-stratum refutation rates are highest at the strongest claims, mirroring what would be expected from a truth-blind predictor.

Figure 4: Scatter of learned versus MC ground-truth excess risk: absence of calibration, especially among high-magnitude claims.

The artifact is observable from early in training and is persistent as performance increases Figure 5, showing full formation by 0.5M steps and no attenuation as reward nearly doubles.

Figure 5: Overclaiming is fully formed early in training and persists regardless of score improvements.

Robustness and Controls

Several controls and variations were employed to test for audit or agent failures:

• Positive controls with environments containing known, tunable risk trade-offs yielded nearly perfect confirmation rates and high correlation, validating both audit power and architectural capacity.

Figure 6: On a controlled environment with explicit risk, the ranking tracks ground truth ($r=0.92$), in contrast to MinAtar.

• The pathological overclaiming signature was replicated under categorical (C51) and implicit quantile (IQN) projections, as well as against high-performance pretrained agents, invalidating explanations based on representation family, agent weakness, or low effect sizes.

Figure 7: Top-strata refutation rates hold across C51 and IQN.

• Post hoc repairs (risk-sensitive training, ensembling, monotone recalibration) did not restore calibration or informativeness; recalibration succeeded only by nullifying the claims wholesale.

Decision-Level Impact

The agent's own CVaR-based policy advice at its most-flagged states was tested:

• In some environments (Breakout), acting on the head's advice is beneficial; in others (Seaquest), it is significantly anti-predictive, leading to higher regret than simply ignoring risk altogether.
• There is no state-wise or environment-independent signal indicating which regime is present—posing severe risks to practical users seeking actionable risk guidance.

Structural Failure Mode

The artifact is not stochastic noise nor uniformly distributed—overclaims concentrate at states exhibiting highest predicted risk edge (winner’s curse), where practitioners are most likely to consult or act upon the head. The artifact is seed-idiosyncratic, uncorrelated both with score and across seeds, confirming its origin in the learning dynamics (bootstrapped distributional targets under function approximation) rather than environmental uncertainty or representational limitations.

Implications and Theoretical Perspective

The findings significantly qualify the utility of distributional RL heads for interpretability, risk-sensitive policy design, and safety guarantees unless their most confident claims can be empirically validated per deployment. Architectures that build uncertainty estimates or post hoc correctives upon raw distributional outputs should be reevaluated in light of these results. The audit methodology itself provides a concrete benchmark for future work on calibrated distributional RL—an audit-passing head would confirm its top claims.

Theoretically, the results empirically instantiate the theoretical pathologies articulated in the QTD literature regarding non-uniqueness and environment-irrelevance of fixed points [rowland2023qtd], as well as the conflation of epistemic and aleatoric uncertainty [clements2019estimating]. Practically, the study supplies a clear call for critical reinterpretation of distributional heads as actionable summaries of environmental risk.

Conclusion

The paper delivers a comprehensive, statistically rigorous audit of risk claims made by modern distributional RL agents, exposing a structural artifact that undermines their practical trustworthiness at exactly the points where they are most likely to be relied upon. Ground-truth validation of risk claims should be considered mandatory prior to interpretation or deployment of distributional RL risk features in any safety- or robustness-critical context. The provided audit toolkit establishes a foundation for future research targeting truly calibrated and decision-relevant risk estimation in RL.