Diffusion Policy Gradient: Methods & Insights

• The paper introduces diffusion policy gradient by generalizing policy-gradient methods to denoising diffusion policies, enabling versatile multimodal action modeling.
• It replaces traditional unimodal (e.g., Gaussian) policies with multi-step diffusion processes using techniques such as Q–score matching, pathwise gradient, and REINFORCE estimators.
• Empirical results across continuous control, robotics, language, and molecular design highlight improved sample efficiency, stability, and robust exploration with solid theoretical backing.

Diffusion Policy Gradient refers to a collection of methodologies that generalize policy-gradient reinforcement learning to the class of stochastic policies parameterized by denoising diffusion models, both in continuous and discrete domains. The central paradigm replaces unimodal action distributions (e.g., Gaussian) with highly expressive, multi-step Markovian diffusion processes, facilitating the modeling and optimization of multimodal and structured policy classes. Over the last three years, a rigorous mathematical and algorithmic foundation for “Diffusion Policy Gradient”—with major variants such as Q–score matching, pathwise and REINFORCE estimators, maximal-entropy RL objectives, and reweighted score matching frameworks—has been developed and empirically validated across control, robotics, language, molecular, and combinatorial domains (Psenka et al., 2023, Chi et al., 2023, Li et al., 2024, Ren et al., 2024, Zekri et al., 3 Feb 2025, Yang et al., 15 May 2025, Zhang et al., 2023, Qi et al., 2 Oct 2025, Liu et al., 2024, Mathur et al., 2023, Kang et al., 4 Dec 2025, Ma et al., 1 Feb 2025, Chen et al., 2023, Sanokowski et al., 1 Dec 2025, Oba et al., 6 Feb 2026, Lattimore, 10 Mar 2026).

1. Diffusion Policies: Parameterization and Expressivity

Diffusion policies, in the canonical setting, parameterize a stochastic policy $\pi_\theta(a|s)$ as the marginal of a Markov chain involving $K$ steps of forward diffusion (noising) and $K$ reverse steps of learned denoising. For state $s$, action $a$ is sampled as $$a = \text{ReverseDiffusion}_\theta(a^K, \ldots, a^0 | s)$$ with $a^K \sim N(0, I)$ and each transition

$$p_\theta(a^{k-1} | a^k, s) = \mathcal{N}(a^{k-1};\, \mu_\theta(a^k, s, k),\, \Sigma_k)$$

A learned score network $s_\theta(a, k, s)$ approximates $\nabla_a \log p_{\theta,k}(a | s)$. This framework generalizes classical unimodal stochastic actors and underpins expressive, multi-modal action distributions, crucial in high-dimensional or task-structured environments (Psenka et al., 2023, Chi et al., 2023, Li et al., 2024).

Discrete action spaces and masked sequence denoising are handled by analogous categorical diffusion chains, with neural “score” networks trained to predict transition rates: $s_\theta(x, t)_y \approx p_t(x)/p_t(y)$ on Hamming neighbors (Zekri et al., 3 Feb 2025, Oba et al., 6 Feb 2026). For combinatorial or graph-valued outputs, each diffusion step may act over nodes/edges by categorical kernels (Liu et al., 2024).

2. Policy-Gradient Formulation with Diffusion Policies

For RL, the goal is to optimize the expected return: $$J(\theta) = \mathbb{E}_{\pi_\theta}\left[ \sum_{t=0}^T \gamma^t r(s_t, a_t) \right]$$ The gradient is (policy-gradient theorem): $$\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\bigg[ \sum_t A_t \nabla_\theta \log \pi_\theta(a_t|s_t) \bigg]$$ For a diffusion policy, the marginal action log-likelihood decomposes as: $$\log \pi_\theta(a^0 | s) = \sum_{k=1}^K \log p_\theta(a^{k-1} | a^k, s)$$ Thus, the gradient becomes: $$\nabla_\theta J(\theta) = \mathbb{E}\left[ \sum_{t=0}^{T} \sum_{k=1}^{K} A_t \nabla_\theta \log p_\theta(a_t^{k-1} | a_t^{k}, s_t) \right]$$ where $A_t$ is an advantage estimate (e.g., from GAE) (Ren et al., 2024, Psenka et al., 2023, Yang et al., 15 May 2025).

The central issue is that for direct end-to-end RL, log-marginals and their gradients involve intractable nested integrations through the diffusion reverse chain (especially for large $K$). Multiple algorithms have been developed to address this, as detailed below.

3. Major Algorithmic Variants

Q–Score Matching (QSM)

Introduced in (Psenka et al., 2023), Q–Score Matching derives a regression-based update exploiting a key insight: under ideality, the diffusion score field aligns with the action-gradient of the Q-function, $s^*(s,a) \propto \nabla_a Q^*(s,a)$. For a mini-batch $(s, a, r, s')$ and time-step $t$, sample a noised $a_t$ via $t$ forward steps. The loss is: $$L_{QSM}(\theta) = \mathbb{E}_{(s,a), t} \left[ \|\, s_\theta(a_t, t | s) + \nabla_a Q_\phi(s, a_t) \|^2 \right]$$ Minimizing this sidesteps backpropagation through the diffusion sampler and preserves the multimodality and exploration capacity of the policy (Psenka et al., 2023).

Denoising Score Matching and Behavior Cloning

For behavior cloning, the score network is trained via denoising score matching: $$L_{\text{DSM}}(\theta) = \mathbb{E}_{(s, a^0, t, \epsilon)} \left[ \|\epsilon - \epsilon_\theta(s, a^k, t)\|^2 \right]$$ where $$a^k = \sqrt{\bar\alpha_k} a^0 + \sqrt{1-\bar\alpha_k} \epsilon$$ and $\epsilon \sim \mathcal{N}(0, I)$. The learned score plays a “policy-gradient role” during inference through Langevin dynamics, implementing stochastic gradient ascent in action-space (Chi et al., 2023). This mechanism provides efficient multimodal imitation and fits a broad class of real-world action distributions.

Policy Gradient through Reverse Chain (Pathwise and REINFORCE)

Several works attempt to differentiate through the reverse diffusion chain:

• Pathwise/backprop-through-timesteps: NCDPO (Yang et al., 15 May 2025) treats all reverse steps as deterministic functions given sampled noises, making $a^0 = f_\theta(s; a^K, z^{1:K})$ fully differentiable. This enables backpropagation through all denoising steps for policy gradient/PPO updates, achieving improved sample efficiency and tractable optimization (Yang et al., 15 May 2025).
• Likelihood-sum approach: DPPO (Ren et al., 2024) implements RL by summing the Gaussian log-probabilities at each reverse step, yielding closed-form gradients suitable for PPO. This is generalizable to DiffSAC/DiffWPO (Sanokowski et al., 1 Dec 2025).
• REINFORCE estimators: For discrete or combinatorial settings where pathwise gradients are infeasible, unbiased score-function estimators (REINFORCE) are used (Zekri et al., 3 Feb 2025, Liu et al., 2024, Oba et al., 6 Feb 2026).

Reweighted Score Matching for Online RL

Naive backpropagation through diffusion steps is computationally infeasible for large $K$. Reweighted Score Matching (RSM) (Ma et al., 1 Feb 2025) avoids this by reweighting the score-matching objective with importance weights, preserving the correct fixed-point solution while allowing learning from policy-improved targets or Boltzmann-weighted Q-values. This underlies methods such as Diffusion Policy Mirror Descent (DPMD) and Soft Diffusion Actor Critic (SDAC).

Max-Entropy and KL-Regularized Objectives

Diffusion policy gradients are readily extended to maximize expected reward plus an entropy or KL-regularization term. Reverse KL divergence is often upper-bounded and tractably estimated through the forward/reverse diffusion process, permitting rigorous connection to maximum entropy RL and its Boltzmann policies. Algorithms such as SQDF (Kang et al., 4 Dec 2025) and DiffSAC/DiffPPO (Sanokowski et al., 1 Dec 2025) use KL or entropy bonuses to encourage diversity and stability, with gradients computed via a combination of pathwise backprop, soft-Q surrogates, or the REINFORCE estimator.

4. Algorithmic Properties, Efficiency, and Implementation

Approach	Gradient Estimator	Backprop Path	Sample Efficiency
QSM (Psenka et al., 2023)	Regression to $\nabla_a Q$	No diffusion chain	High (avoids full chain)
DPPO (Ren et al., 2024)	Sum-logprob of chain	Yes, per-step	High (on-manifold stability)
NCDPO (Yang et al., 15 May 2025)	Pathwise (noise-cond.)	Yes, through chain	High (matches MLP+PPO)
RSM (Ma et al., 1 Feb 2025)	Weighted score-matching	No chain	Very high
SEPO (Zekri et al., 3 Feb 2025)	REINFORCE	No chain	High for discrete models
DiffSAC/PPO (Sanokowski et al., 1 Dec 2025)	As in PPO/SAC, via per-step chain	Yes	Efficient; minor overhead
SRPO (Chen et al., 2023)	Behavior score reg.	No chain	High; O(1) inference time

Key algorithmic details include truncating or sharing scores among diffusion steps, using “mode-conditioned” embeddings for explicit control of discovered behaviors (Li et al., 2024), and integrating off-policy replay buffers for improved mode coverage (Kang et al., 4 Dec 2025, Li et al., 2024). For graph or discrete domains, adaptation of categorical SDEs and marginalization tricks are necessary for tractability (Liu et al., 2024, Zekri et al., 3 Feb 2025).

5. Empirical Results and Benchmarks

Diffusion policy gradient methods have demonstrated superior or competitive performance across a diverse range of domains:

• Continuous Control: On MuJoCo and DeepMind Control Suite tasks, QSM, DPPO, NCDPO, DPMD, and SDAC consistently match or outperform standard Gaussian-baseline RL algorithms (TD3, SAC, PPO), with notable advantages in sample efficiency and exploration, and impressive stability in high-dimensional action spaces (Psenka et al., 2023, Ren et al., 2024, Yang et al., 15 May 2025, Ma et al., 1 Feb 2025).
• Robotics and Manipulation: Diffusion Policy (Chi et al., 2023) and DPPO achieve high-fidelity visuomotor control and manipulation, especially with vision inputs and compositional policies (Ren et al., 2024, Chi et al., 2023).
• Multimodal RL and Exploration: DDiffPG (Li et al., 2024) maintains and discovers multiple behavioral modes, with explicit clustering and mode-specific Q-learning, enabling robust multimodal behavior under sparse rewards and dynamic replanning scenarios.
• Discrete and Combinatorial: SEPO (Zekri et al., 3 Feb 2025) and GDPO (Liu et al., 2024) facilitate RL fine-tuning of masked LLMs, sequence generators, and molecular/graph diffusion models with non-differentiable objectives.
• 3D Generation: RL-based policy-gradient approaches (e.g., DDPO3D) effectively align 3D score-distillation pipelines to downstream rewards and aesthetics.
• Theoretical Analysis: SDE/diffusion-limit analyses clarify drift, noise, and regret in bandit policy-gradient settings (Lattimore, 10 Mar 2026).

6. Theoretical Properties and Convergence

Rigorous analysis underpins the diffusion policy gradient framework:

• Fixed-point and monotonicity: Q–Score Matching provably aligns the diffusion score with $\nabla_a Q$, ensuring that regression steps strictly improve $J$ under standard actor-critic coupling (Psenka et al., 2023).
• Variance and Credit Assignment: Pathwise estimators (NCDPO) enjoy low-variance updates and stable gradients through exact backprop. Score-matching and REINFORCE estimators maintain unbiasedness with proper baseline subtraction and importance weighting (Zekri et al., 3 Feb 2025, Oba et al., 6 Feb 2026).
• Global Convergence: Proximal PG for SDEs with control-dependent diffusion achieves linear convergence under strong convexity assumptions (Davey et al., 23 May 2025).
• Exploration and Multimodality: Diffusion policies maintain on-manifold exploration in both the action and state space, supporting better coverage and structural robustness compared to traditional parameterizations (Li et al., 2024, Sanokowski et al., 1 Dec 2025).
• Limitations and Open Problems: Sampling efficiency, implicit bias of one-step Q-approximations, and stability of high-step chains remain research frontiers (Kang et al., 4 Dec 2025, Zekri et al., 3 Feb 2025, Psenka et al., 2023).

7. Extensions and Broader Impact

The diffusion policy gradient methodology has been extended and rigorously validated across several axes:

• Graph and Structure Generation: Fine-tuning graph diffusion policies for molecular design, combining categorical SDEs and eager policy gradients (Liu et al., 2024).
• Language and Masked Models: Scalable discrete diffusion RL (SEPO), masked LLMs with intermediate-state credit assignment (DiSPO) (Zekri et al., 3 Feb 2025, Oba et al., 6 Feb 2026).
• Maximum Entropy and KL-RL: Unified frameworks for Boltzmann-weight and KL-regularized RL, with efficient algorithmic instantiations for SAC, PPO, and WPO (Sanokowski et al., 1 Dec 2025, Kang et al., 4 Dec 2025).
• Efficiency and Deterministic Extraction: Score-regularized policy optimization replaces expensive sampling with single-shot inference, directly using diffusion-based score functions to regularize regression-based actors (Chen et al., 2023).
• Guided Policy Gradients: Diffusion-inspired conditional/unconditional guidance boosts controllability and sample efficiency across settings (Qi et al., 2 Oct 2025).
• 3D, Vision, and Beyond: Policy-gradient aligned diffusion enables reward-driven control of complex generative models, image and asset synthesis, and cross-domain transfer (Mathur et al., 2023).

The methodological diversity and empirical advances of diffusion policy gradient methods have established them as foundational for high-dimensional, multimodal policy learning, spanning continuous control, robotics, combinatorial optimization, sequence generation, molecular design, and beyond. Their theoretical tractability and flexibility continue to accelerate progress at the intersection of generative modeling and reinforcement learning.