Diffusion Actor-Critic (DAC) Overview

• Diffusion Actor-Critic (DAC) is a reinforcement learning approach that integrates reverse-time diffusion processes with actor-critic methods to generate complex, multimodal action distributions.
• DAC leverages Q-gradient guidance and adaptive entropy-regularization to refine policies, reduce diffusion steps, and achieve state-of-the-art performance in continuous control tasks.
• Efficient implementations like DACER2 demonstrate faster inference (<1 ms/action) and superior benchmarks on tasks such as Humanoid, Ant, HalfCheetah, and Walker2d.

Diffusion Actor-Critic (DAC)

Diffusion Actor-Critic (DAC) encompasses a family of reinforcement learning (RL) methodologies that integrate score-based diffusion models into the policy or value-learning components of actor-critic algorithms. Leveraging reverse-time stochastic differential equations (SDEs), diffusion policies are trained either in online or offline RL regimes to enable expressive, multimodal action distributions and more robust policy improvement. DAC variants address inference and training efficiency, theoretical optimality under entropy-regularized or KL-constrained objectives, and establish state-of-the-art (SOTA) performance across a range of continuous control and multimodal tasks.

1. Fundamental Principles and Mathematical Foundations

In DAC, the agent’s policy is parameterized as a reverse diffusion process over actions, conditioned on the state. At each timestep, an action is generated by unrolling a reverse-time SDE (or discrete Markov process) from a noise sample, with learned denoiser (score network) $$S_\theta(s,a_t,t) \approx \nabla_{a_t} \log p_t(a_t \mid s)$$, drift $f$, and diffusion coefficient $g$: $$da_t = [f(a_t, t) - g(t)^2 S_\theta(s, a_t, t)]\,dt + g(t)\,d\omega(t),$$ with $a_T\sim\mathcal{N}(0, I)$. The resulting policy does not admit a tractable density; actions are sampled via this generative process.

The canonical actor objective implements “max-Q” or maximum expected reward, possibly under entropy or KL constraints. The entropy-regularized form seeks a Boltzmann policy $\pi^*(a\mid s) \propto \exp(Q(s,a)/\alpha)$, rendering diffusion sampling particularly effective for capturing complex, multimodal action-value landscapes (Wang et al., 29 May 2025, Wang et al., 2024, Fang et al., 2024).

2. Policy Learning with Reverse Diffusion and Q-Gradient Guidance

Unlike conventional Gaussian policies, DAC employs an expressive diffusion-based generator for the policy, allowing it to represent arbitrarily complex action distributions. The score network is trained to guide denoising towards high-$Q$ actions by introducing a Q-gradient auxiliary objective. The theoretical underpinning is that for the max-entropy or KL-constrained optimal policy,

$$\nabla_a \log\pi^*(a|s) = \frac{1}{\alpha} \nabla_a Q(s,a) \quad \text{or} \quad \nabla_a \log\pi^*(a|s) = \nabla_a \log\pi_\beta(a|s) + \frac{1}{\eta}\nabla_a Q(s,a),$$

thus score-matching to the Q-gradient aligns the diffusion policy with the soft-optimal solution (Fang et al., 2024, Dong et al., 17 Feb 2025, Wang et al., 29 May 2025).

In enhanced variants such as DACER2, the Q-gradient field is normalized and temporally weighted: $$\mathcal{L}_g(\theta) = \mathbb{E}_{s,t,a_t} \left\| w(t)\nabla_{a_t} Q_{\rm norm}(s,a_t) - S_\theta(s, a_t, t) \right\|_2^2,$$ with $w(t)=\exp(0.4\,t-1.8)$, to provide strong coarse guidance during early denoising steps and refined alignment late, enabling accurate action refinement with few diffusion steps (Wang et al., 29 May 2025).

3. Actor, Critic, and Entropy-Augmented Objectives

The general actor loss in DAC aggregates the policy-gradient (max-Q or soft-Q) term, the Q-gradient auxiliary guidance, and (for some variants) an entropy regularizer: $f$0 Entropy is estimated via a Gaussian Mixture Model fitted to sampled actions, as the diffusion policy’s density is intractable (Wang et al., 2024, Wang et al., 29 May 2025). Adaptive entropy coefficients can be tuned by gradient ascent/descent to maintain a target entropy.

Critic learning typically employs double-$f$1 or ensemble lower-confidence-bound (LCB) updates: $f$2

$f$3

for stability and bias reduction (Fang et al., 2024, Wang et al., 29 May 2025).

4. Training Algorithms and Pseudocode

Actor and critic updates alternate in a standard off-policy actor-critic loop. Key steps include:

• Sampling environment transitions $f$4.
• Critic update via TD bootstraps from double/ensemble Q.
• Actor/score update by reverse-diffusion sampling, Q-gradient matching at randomly sampled diffusion steps, and entropy regularization (if present).
• Target networks are updated via Polyak EMA.

A condensed DACER2 training pseudocode (Wang et al., 29 May 2025): $a_T\sim\mathcal{N}(0, I)$1 A fast inference algorithm achieves <1 ms/action with only 5 diffusion steps.

5. Theoretical Properties and Justifications

The Q-gradient auxiliary loss in DACER2 and similar methods exploits the correspondence between the optimal soft policy gradient and the $f$5 gradient, providing dense intermediate supervision throughout the diffusion process and preventing mode collapse. Temporal weighting calibrates the impact of the Q-gradient based on the denoising stage, empirically enabling significant reduction in diffusion steps ($f$6 from 20 to 5) without performance degradation (Wang et al., 29 May 2025). In Q-constrained settings, soft Q-guidance supresses out-of-distribution actions and fosters stable iterative improvement (Fang et al., 2024).

Empirical ablations show that the omission of entropy regularization or Q-gradient guidance degrades stability and expressiveness, while ensemble LCB critics avoid both over-pessimism and overestimation.

6. Empirical Results and Performance Benchmarks

DAC algorithms demonstrate SOTA performance on standard continuous-control suites. For example, DACER2 achieves:

• Humanoid: $f$7
• Ant: $f$8
• HalfCheetah: $f$9
• Walker2d: $g$0 with only 5 diffusion steps, outperforming both SAC and the original DACER across all high-dimensional tasks. Inference time is reduced by 60.6%, and training time by 41.7%, relative to DACER (Wang et al., 29 May 2025). In multimodal toy tasks, DACER2 stably samples all goal modes, while other diffusion policies may miss modes.

A summary table of final returns:

Task	SAC	DACER	DACER2
Humanoid	$g$1	$g$2	$g$3
Ant	$g$4	$g$5	$g$6
HalfChee.	$g$7	$g$8	$g$9
Walker2d	$$da_t = [f(a_t, t) - g(t)^2 S_\theta(s, a_t, t)]\,dt + g(t)\,d\omega(t),$$0	$$da_t = [f(a_t, t) - g(t)^2 S_\theta(s, a_t, t)]\,dt + g(t)\,d\omega(t),$$1	$$da_t = [f(a_t, t) - g(t)^2 S_\theta(s, a_t, t)]\,dt + g(t)\,d\omega(t),$$2

7. Implementation and Hyperparameter Details

Strongest-performing DAC variants share the following hyperparameters (Wang et al., 29 May 2025):

• Diffusion steps $$da_t = [f(a_t, t) - g(t)^2 S_\theta(s, a_t, t)]\,dt + g(t)\,d\omega(t),$$3.
• Actor/Critic learning rate $$da_t = [f(a_t, t) - g(t)^2 S_\theta(s, a_t, t)]\,dt + g(t)\,d\omega(t),$$4.
• Q-gradient loss weight $$da_t = [f(a_t, t) - g(t)^2 S_\theta(s, a_t, t)]\,dt + g(t)\,d\omega(t),$$5 (Humanoid/Ant/HalfCheetah), 1.0 (Walker2d).
• Temporal weighting $$da_t = [f(a_t, t) - g(t)^2 S_\theta(s, a_t, t)]\,dt + g(t)\,d\omega(t),$$6.
• Batch size 256; replay capacity $$da_t = [f(a_t, t) - g(t)^2 S_\theta(s, a_t, t)]\,dt + g(t)\,d\omega(t),$$7.
• Discount $$da_t = [f(a_t, t) - g(t)^2 S_\theta(s, a_t, t)]\,dt + g(t)\,d\omega(t),$$8.
• Adam optimizer ($$da_t = [f(a_t, t) - g(t)^2 S_\theta(s, a_t, t)]\,dt + g(t)\,d\omega(t),$$9, $a_T\sim\mathcal{N}(0, I)$0).

Networks typically use MLPs for actor/critic; score functions may be implemented as multi-layer MLPs or U-Nets with timestep embeddings. Entropy is estimated using GMM fitting on batches of action samples.

• "Enhanced DACER Algorithm with High Diffusion Efficiency" (Wang et al., 29 May 2025)
• "Diffusion Actor-Critic with Entropy Regulator" (Wang et al., 2024)
• "Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning" (Fang et al., 2024)
• "Maximum Entropy Reinforcement Learning with Diffusion Policy" (Dong et al., 17 Feb 2025)