RS-Diffuser: Risk-Sensitive Diffusion Planning with Distributional Value Guidance

Abstract: Offline reinforcement learning enables policy learning from fixed datasets without additional environment interaction, making it appealing for safety-critical applications where online exploration is costly or unsafe. Diffusion-based decision-making methods have recently achieved strong performance in offline RL by modeling rich, multimodal trajectory distributions. However, existing diffusion planners are typically risk-neutral and therefore may overlook rare but catastrophic outcomes that are crucial in real-world deployment. In this work, we propose RS-Diffuser, a risk-sensitive offline diffusion planning framework that combines diffusion-based trajectory generation with distributional value critics. RS-Diffuser learns a diffusion planner over future state trajectories, a separate inverse dynamics model for action decoding, and a Monte Carlo distributional critic that estimates the full return distribution of candidate plans through quantile regression. At sampling time, we incorporate a risk-sensitive guidance signal into the denoising process, using gradients computed from tail-aware objectives such as Conditional Value at Risk to steer generation toward desired risk profiles. As a result, a single trained model can flexibly produce risk-averse, risk-neutral, or risk-seeking behaviors by changing only the inference-time risk parameter. Extensive experiments on risk-sensitive D4RL and risky robot navigation benchmarks demonstrate that RS-Diffuser achieves state-of-the-art performance, improving both overall return and worst-case robustness while reducing safety violations.

• The paper introduces RS-Diffuser, which embeds explicit risk-sensitivity into diffusion planning to optimize tail-aware performance in offline RL.
• It employs a Monte Carlo distributional value critic and an inverse dynamics action decoder to generate risk-adjusted trajectories without requiring retraining.
• Empirical evaluations demonstrate superior performance in mean returns and worst-case risk metrics, outperforming state-of-the-art models in various benchmarks.

RS-Diffuser: Risk-Sensitive Diffusion Planning with Distributional Value Guidance

Motivation and Background

Offline reinforcement learning (RL) is essential for decision-making in safety-critical domains where online experimentation is costly or dangerous, such as healthcare, robotics, and autonomous driving. Recent advances have leveraged diffusion models to represent complex, multimodal trajectory distributions, leading to superior performance across diverse offline RL benchmarks. However, these diffusion-based approaches are inherently risk-neutral, optimizing expected returns while neglecting rare but disastrous outcomes that are critical in real-world scenarios.

Risk-sensitive RL, particularly distributional RL, addresses this gap by modeling the entire distribution of returns, enabling optimization of tail-aware criteria like Conditional Value at Risk (CVaR). Existing risk-sensitive offline RL methods predominantly rely on actor-critic constructs and lack the expressivity and flexibility of diffusion-based planners. RS-Diffuser introduces a structured mechanism to embed explicit risk-sensitivity into diffusion planning, enabling flexible inference-time control over risk preferences.

Methodological Contributions

RS-Diffuser augments the offline diffusion planning paradigm with a distributional value critic for tail-aware policy synthesis. The architecture consists of three key components:

1. Diffusion Planner over State Trajectories: Models future state sequences conditioned on the current state, decoupling long-horizon planning from low-level control.
2. Inverse Dynamics Action Decoder: Recovers actions from planned state transitions through supervised regression, leveraging offline data for consistent mapping.
3. Monte Carlo Distributional Value Critic: Predicts the full return distribution of candidate trajectories using quantile regression, supporting metrics such as VaR and CVaR.

Unlike prior works, RS-Diffuser leverages the critic’s quantile-based return distribution at inference to inject risk-sensitive gradients into the denoising process. This approach steers the generative trajectory sampling toward specified risk profiles without retraining, enabling the same model to realize risk-averse, risk-neutral, or risk-seeking behavior by varying a single risk parameter.

Theoretical and Practical Implications

The integration of distributional RL concepts directly within the generative trajectory planning offers several advantages. By using Monte Carlo supervision instead of Bellman bootstrapping, RS-Diffuser reduces bias and stabilizes critic training. The critic’s quantile predictions yield differentiable risk measures, whose gradients are efficiently incorporated into each diffusion step, guiding the model to select trajectories with desired tail characteristics.

The planner’s training is decoupled from risk preference, allowing post-hoc adjustment at inference. This yields broad practical utility: practitioners in safety domains can deploy a single model and dynamically switch risk profiles based on contextual requirements, policy audits, or changing operational constraints.

Empirical Evaluation

RS-Diffuser was evaluated on risk-sensitive D4RL (MuJoCo benchmarks with stochastic penalties) and risky robot navigation tasks (Risky PointMass and Risky Ant). The key metrics were mean return, CVaR$_{0.1}$ (worst-case returns), and safety violations (time spent in hazardous states).

Numerical summary highlights:

• On risk-sensitive D4RL, RS-Diffuser surpassed all baselines (including CQL, Diffusion-QL, OWCPG, ORAAC, CODAC, UDAC) in both mean and CVaR$_{0.1}$, especially in environments with significant tail risk (e.g., Half-Cheetah Mixed: CVaR$_{0.1}$=301, Mean=539; Hopper Expert: CVaR$_{0.1}$=1475, Mean=1623).
• In risky robot navigation, RS-Diffuser achieved zero violations in Risky PointMass and lowest violations in Risky Ant (269), while maintaining highest returns across mean, median, and CVaR$_{0.1}$.
• Ablation studies on risk measures evidenced that smaller $\alpha$ yields more risk-averse policies with superior CVaR but lower mean return, while larger $\alpha$ was more aggressive. CVaR guidance consistently promoted robust tail performance.

Contradictory and Strong Claims

RS-Diffuser produces flexible risk-sensitive behaviors without retraining. Unlike actor-critic risk-sensitive models (UDAC, CODAC), which optimize for a fixed risk profile, RS-Diffuser enables dynamic adaptation of risk preference post training, evidenced by superior results in both average and worst-case performance metrics.

Another strong claim supported by the empirical results is that distributional guidance within diffusion planning not only improves robustness but also decreases safety violations compared to state-of-the-art baselines, including those explicitly designed for risk-aware policy synthesis.

Broader Implications and Future Directions

The RS-Diffuser framework suggests a paradigm shift in offline RL policy deployment: risk preference modulation via inference parameters rather than retraining. This enhances operational adaptability, especially in environments with variable risk tolerance. The underlying methodology is extendable to domains requiring compliance with multi-objective constraints, richer uncertainty-aware critics, and high-dimensional robotics control.

Future work may focus on expanding RS-Diffuser to real-world robotics domains with richer dynamical structure, integrating constraint inference (e.g., ICRL methodologies), and investigating the interplay between distributional policy evaluation and generative planning in multi-agent and multi-task settings.

Conclusion

RS-Diffuser demonstrates that explicit distributional guidance within diffusion planning decisively improves risk-sensitive decision-making in offline RL. It achieves state-of-the-art performance in both mean and worst-case returns across diverse safety-critical benchmarks, while enabling flexible post-hoc control over risk appetite. The approach merges the expressive trajectory modeling of diffusion with principled distributional value evaluation, offering a robust foundation for deploying adaptive, safe, and high-performance RL agents (2606.27766).