Risk-Aware Multimodal Actor-Critic (RAMAC)

• The paper introduces a novel offline RL framework that integrates an expressive generative actor with a distributional critic to optimize lower-tail risk (CVaR) while achieving strong returns.
• It employs a composite actor objective that combines behavior cloning with risk minimization, using diffusion and flow-matching methods for robust multimodal policy generation.
• Experimental results on Stochastic-D4RL tasks show that RAMAC reduces out-of-distribution errors and consistently outperforms baselines on CVaR and mean return metrics.

Risk-Aware Multimodal Actor-Critic (RAMAC) is an offline reinforcement learning framework for safety-critical domains in which online data collection is infeasible and policies must deliver high returns without incurring catastrophic lower-tail risk. The framework couples an expressive generative actor with a distributional critic, and it differentiates a composite objective combining distributional risk and behavior cloning (BC) loss through the generative path. RAMAC is instantiated with diffusion and flow-matching actors, and it is reported to achieve consistent gains in $\mathrm{CVaR}_{0.1}$ while maintaining strong returns on most Stochastic-D4RL tasks (Fukazawa et al., 3 Oct 2025).

1. Problem formulation and conceptual scope

RAMAC is motivated by a specific gap in offline RL. Prior work on risk-averse offline RL achieves safety at the cost of value conservatism and restricted policy classes, whereas expressive policies are only used in risk-neutral settings. RAMAC addresses this gap in complex multimodal scenarios by combining lower-tail risk optimization with an expressive policy class and explicit behavior regularization (Fukazawa et al., 3 Oct 2025).

The framework is designed for settings in which multimodality is not incidental but structural. The actor is intended to capture multimodal behaviors, while the critic models the full conditional return distribution rather than only its mean. This pairing is central to the RAMAC formulation: risk sensitivity is not added as a post hoc penalty to a conventional critic, but is driven by a distributional representation of returns and propagated through the actor’s generative mechanism.

A common misconception is that risk-aware offline RL must rely on pessimism or conservative value estimation. RAMAC is explicitly positioned against that assumption: it does not rely on pessimism or conservative value estimation, and it does not force the policy to remain close to a unimodal anchor. Instead, it regularizes the deployed generative policy directly with BC, while using a lower-tail risk objective derived from the critic.

2. Generative actor and distributional critic

The RAMAC actor is an expressive generative model with reparameterized action generation,

$$a = \psi_\theta(s, z), \quad z \sim \mathcal{N}(0, I),$$

where $\psi_\theta$ is a differentiable mapping from state $s$ and latent noise $z$ to action $a$ (Fukazawa et al., 3 Oct 2025).

Two actor instantiations are given. The diffusion-policy variant, called RADAC, generates actions by evolving a reverse-time stochastic differential equation conditioned on the state,

$$\mathrm{d}_t \mathbf{a}_t = f_\theta(t, \mathbf{a}_t, s)\,\mathrm{dt} + g(t)\,\mathrm{d}\mathbf{w}_t.$$

The flow-matching variant, called RAFMAC, generates actions by integrating a deterministic ordinary differential equation,

$$\frac{\mathrm{d}\mathbf{a}_t}{\mathrm{dt}} = v_\theta(t, \mathbf{a}_t, s).$$

Both classes enable differentiable trajectories so that critic signals, including risk and reward gradients, can be backpropagated through the entire action-generation process.

The critic is distributional. RAMAC uses an IQN critic $Z_\phi$ that parameterizes the return quantile function $Z_\phi(s,a;\tau)$ with $$a = \psi_\theta(s, z), \quad z \sim \mathcal{N}(0, I),$$0. The critic follows the distributional Bellman equation

$$a = \psi_\theta(s, z), \quad z \sim \mathcal{N}(0, I),$$1

where $$a = \psi_\theta(s, z), \quad z \sim \mathcal{N}(0, I),$$2 and $$a = \psi_\theta(s, z), \quad z \sim \mathcal{N}(0, I),$$3. Training uses a quantile Huber objective,

$$a = \psi_\theta(s, z), \quad z \sim \mathcal{N}(0, I),$$4

The architectural distinction of RAMAC lies in the coupling of these two components. The actor is not merely stochastic; it is expressive and multimodal. The critic is not merely scalar-valued; it is distributional and therefore directly supports lower-tail risk measurement. This suggests that RAMAC treats multimodality and risk sensitivity as jointly coupled design requirements rather than separate enhancements.

3. CVaR optimization and the composite actor objective

RAMAC focuses on Conditional Value-at-Risk (CVaR) as its risk measure. In integral form,

$$a = \psi_\theta(s, z), \quad z \sim \mathcal{N}(0, I),$$5

For actor gradients, the paper uses a Monte Carlo estimate derived from critic quantiles,

$$a = \psi_\theta(s, z), \quad z \sim \mathcal{N}(0, I),$$6

The corresponding risk term for policy optimization is

$$a = \psi_\theta(s, z), \quad z \sim \mathcal{N}(0, I),$$7

The negative sign means maximizing the lower-tail return (Fukazawa et al., 3 Oct 2025).

The actor is trained with a composite objective,

$$a = \psi_\theta(s, z), \quad z \sim \mathcal{N}(0, I),$$8

Here, $$a = \psi_\theta(s, z), \quad z \sim \mathcal{N}(0, I),$$9 is the BC loss between policy and dataset actions, $\psi_\theta$0 is the negative CVaR signal from the distributional critic, and $\psi_\theta$1 balances imitation and risk aversion.

This objective is described as fully differentiable end-to-end along the generative trajectory. That property is operationally important: it allows direct risk shaping of the expressive policy through the generative path, rather than relying on surrogate action-selection heuristics or separate post-processing. The learning loop is correspondingly simple at a high level: the critic is fit by quantile Bellman error, and the actor is updated by minimizing the composite BC-plus-CVaR objective.

4. Behavior regularization and out-of-distribution control

Behavior regularization is a central element of RAMAC because offline RL is vulnerable to out-of-distribution (OOD) actions that can induce unsafe behavior and erroneous value estimates. RAMAC applies BC loss directly to the deployed generative policy; no anchor or prior mixing is needed. The paper states that this regularizes the expressive actor to remain on or near the behavioral manifold observed in the data, with forward KL minimization as a practical outcome (Fukazawa et al., 3 Oct 2025).

A theoretical statement in the appendix connects OOD control to KL divergence. The per-state OOD probability is bounded as

$\psi_\theta$2

Under this bound, minimizing BC loss, or forward KL, directly reduces the OOD action rate.

RAMAC contrasts this mechanism with prior-anchored perturbation methods such as ORAAC. According to the paper, prior-anchored approaches constrain the policy as residual perturbations to a behavior prior, often with a norm bound, and these methods struggle with support leakage in multimodal or nonconvex action spaces, especially around thin or nonconvex regions. RAMAC instead regularizes globally using the actual data distribution. The reported implication is better mode coverage and avoidance of geometric OOD failure modes that can arise when a local anchor is treated as sufficient.

A further practical component is empirical OOD monitoring. The implementation uses efficient OOD action detection via a 1-NN metric and robust thresholding as part of evaluation. In the reported experiments, RAMAC achieves the lowest empirical rate of OOD actions, denoted $\psi_\theta$3.

5. Experimental setting, reported performance, and implementation

RAMAC is evaluated on Stochastic-D4RL benchmarks, described as modified D4RL MuJoCo datasets with rare, heavy-tailed penalties injected, including velocity- or angle-dependent hazards in half-cheetah, walker, and hopper. Baselines include value-conservative methods such as CQL, distributionally conservative methods such as CODAC, prior-anchored risk-averse methods such as ORAAC, and expressive risk-neutral methods such as DiffusionQL and FlowQL. The principal metrics are mean return and $\psi_\theta$4 (Fukazawa et al., 3 Oct 2025).

The main reported finding is that RAMAC variants, RADAC and RAFMAC, consistently outperformed baselines on $\psi_\theta$5 on most tasks while maintaining highly competitive or state-of-the-art mean return. The paper further reports low OOD error, qualitative safety improvements, and robustness across risk distortions, with appendix ablations indicating that CVaR is robust for lower-tail control while other risk distortions are less reliable.

The qualitative interpretation of the results is tied to mode allocation. Diffusion-based RADAC is described as maintaining mass on diverse in-distribution modes and as being strong in lower-tail hazards in halfcheetah and hopper tasks. RAFMAC is described as achieving strong mean and CVaR when task geometry favors flow transport. On a toy bandit task, the paper reports that RAMAC avoids unsafe inter-modal gaps and successfully targets safe, high-value modes, unlike anchor-perturb methods or expressive risk-neutral baselines.

Implementation details reported in the paper include diffusion actors with 5 steps and an MLP of size 256-256, flow actors with 10 steps, and a double IQN critic with quantile Huber loss and 32 quantiles. Shared hyperparameters include Adam, batch size 256, discount 0.99, and risk level $\psi_\theta$6. The reported BC–risk balancing coefficients are $\psi_\theta$7 for RADAC and $\psi_\theta$8 for RAFMAC. The codebase is released at the project repository linked in the paper.

6. Relation to other risk-aware actor-critic methods

RAMAC belongs to a broader family of risk-aware actor-critic methods, but it occupies a distinct point in that landscape. Earlier work studied model-free RL with dynamic convex risk measures and an actor-critic style algorithm using neural networks, emphasizing time-consistent dynamic programming and policy gradient updates under dynamic convex risk measures (Coache et al., 2021). Other work used a risk-conditioned distributional soft actor-critic framework in navigation, conditioning actor and critic networks on a risk-measure parameter $\psi_\theta$9 so that a single agent can adapt online to a range of risk measures without retraining (Choi et al., 2021). In motion planning, risk-conditioned SAC added a risk critic and a penalty term involving a continuous risk bound $s$0, allowing the user to adjust the risk-averse level of the agent on the fly (Huang et al., 2021). More recently, static spectral risk measures have been optimized in online and offline actor-critic settings through a unified SRM framework based on bi-level optimization and a distributional critic (Moghimi et al., 5 Jul 2025).

Relative to those formulations, RAMAC is specialized to offline RL with multimodal expressive policies. Its central differentiation is not merely the use of a risk measure, but the coupling of a distributional risk objective with a generative actor and direct BC regularization on the deployed policy. This suggests that RAMAC extends the scope of risk-aware actor-critic design from risk-conditioned value shaping and constrained optimization toward expressive multimodal policy classes in offline settings.

The literature also clarifies what RAMAC is not. It is not formulated around time-consistent dynamic convex risk measures, as in dynamic risk-to-go methods. It is not described as a runtime risk-conditioning method over a user-specified parameter such as $s$1 or $s$2. It is not a pessimistic or conservative-value method in the style of value-conservative offline RL. Instead, the paper frames it as a risk-aware multimodal offline RL framework in which lower-tail safety is optimized directly through CVaR gradients propagated along the generative path, while OOD behavior is controlled by direct BC regularization.

Taken together, these properties define RAMAC as a particular synthesis within risk-aware RL: expressive generative policies, distributional lower-tail optimization, and offline behavior regularization are treated as mutually dependent components rather than independent modules.