Diffusion Policy Optimization

• Diffusion policy optimization is a framework that uses a reverse denoising generative process to produce expressive, multimodal actions in continuous control and language models.
• It employs tractable surrogate objectives and specialized updates, like DPPO and CPPO, to overcome intractable likelihoods and stabilize reinforcement learning.
• These methods have demonstrated significant improvements in robotics, masked diffusion language models, and efficient planning while mitigating challenges like representation collapse.

Diffusion policy optimization refers to methods that optimize policies parameterized by diffusion or related score-based or flow generative models. In these methods, an action, action chunk, trajectory, or token sequence is produced by a reverse denoising chain rather than by a single Gaussian policy or a standard autoregressive factorization, which makes exact policy likelihoods, entropy terms, KL regularization, and credit assignment difficult or expensive to evaluate. The literature therefore centers on tractable surrogates that preserve the expressive, multimodal structure of diffusion policies while making reinforcement learning, preference alignment, or post-training feasible in continuous control, robotics, masked diffusion LLMs, and RLHF for diffusion generators (Ren et al., 2024, Ding et al., 2024, Zhang et al., 19 Mar 2026, Jiang et al., 11 Jun 2026).

1. Generative formulation and optimization problem

In continuous-control RL, a diffusion policy is a conditional generative model over actions or action chunks. A standard formulation uses a forward noising process

$$q(a_t\mid a_{t-1})=\mathcal{N}(a_t;\sqrt{1-\beta_t}\,a_{t-1},\beta_t I),$$

with closed form

$$q(a_t\mid a_0)=\mathcal{N}(a_t;\sqrt{\bar{\alpha}_t}a_0,(1-\bar{\alpha}_t)I),$$

and a learned reverse process

$$p_\theta(a_{t-1}\mid a_t,s)=\mathcal{N}(a_{t-1};\mu_\theta(a_t,t,s),\sigma_t^2 I).$$

The policy sample is obtained by starting from Gaussian noise and iteratively denoising to the executable action. This representation is attractive because diffusion policies can represent multimodal, complex action distributions with richer expressiveness and exploration in continuous spaces (Ren et al., 2024, Yang et al., 15 May 2025).

For masked diffusion LLMs, the optimization object is a reverse denoising trajectory over partially masked token states. With diffusion states $\tau_t\in\bar{\mathcal V}^L$ and reverse policy $\pi_\theta(\tau_{t-1}\mid \tau_t,t)$, the full trajectory probability is

$$\pi_\theta(\boldsymbol{\tau})=\prod_{t=1}^{T}p_\theta(\tau_{t-1}\mid \tau_t,t,\mu_\theta).$$

Unlike autoregressive LLMs, which factorize sequence probability into per-token conditionals computable with one causal forward pass, masked diffusion models require probabilities over multi-step trajectories of partially masked states, making policy optimization dependent on intermediate states that differ from the final clean sequence (Zhang et al., 19 Mar 2026).

This generative structure creates the central optimization difficulty. In continuous control, the marginal action likelihood $\pi_\theta(a\mid s)$ is generally intractable, so standard on-policy methods that depend on exact log-likelihoods or policy ratios become cumbersome (Yang et al., 15 May 2025, Liu et al., 5 Mar 2026). In masked diffusion models and variationally trained diffusion policies, the same problem appears as a gap between the true likelihood and tractable surrogates such as ELBO-style objectives; once that gap grows during RL updates, the resulting proxy gradients can become misaligned with the true policy gradient (Jiang et al., 11 Jun 2026). A large part of the field can be understood as designing update rules that avoid, estimate, or neutralize this intractability without sacrificing the multimodal advantages of diffusion parameterizations.

2. On-policy and online reinforcement learning

One major line of work adapts policy-gradient methods to the denoising chain. "Diffusion Policy Policy Optimization" (DPPO) formulates a two-layer diffusion MDP in which denoising steps are treated as an inner MDP linked to the outer environment MDP, so that per-step Gaussian likelihoods become tractable. DPPO then applies PPO-style clipping over denoising steps, uses environment-step advantages at $k=0$, applies a denoising discount $\gamma_d^k$ to weight earlier denoising steps less, and targets a value function over $s_t$ only rather than $$q(a_t\mid a_0)=\mathcal{N}(a_t;\sqrt{\bar{\alpha}_t}a_0,(1-\bar{\alpha}_t)I),$$0. The paper reports that DPPO achieves the strongest overall fine-tuning performance and stability on several robotic benchmarks and demonstrates zero-shot sim-to-real transfer on a long-horizon manipulation task (Ren et al., 2024).

A closely related but more likelihood-economical formulation appears in "Diffusion Policy through Conditional Proximal Policy Optimization". CPPO aligns the policy iteration with the diffusion process by representing the improved policy as

$$q(a_t\mid a_0)=\mathcal{N}(a_t;\sqrt{\bar{\alpha}_t}a_0,(1-\bar{\alpha}_t)I),$$1

with a conditional Gaussian residual

$$q(a_t\mid a_0)=\mathcal{N}(a_t;\sqrt{\bar{\alpha}_t}a_0,(1-\bar{\alpha}_t)I),$$2

The PPO ratio is then evaluated only in this simple Gaussian space, so the method requires only evaluating a simple Gaussian probability rather than the full diffusion-policy likelihood. The same formulation yields a tractable entropy lower bound,

$$q(a_t\mid a_0)=\mathcal{N}(a_t;\sqrt{\bar{\alpha}_t}a_0,(1-\bar{\alpha}_t)I),$$3

which allows entropy regularization to be handled naturally in a setting where diffusion-policy entropy is otherwise difficult to compute (Liu et al., 5 Mar 2026).

A different route is to convert the denoising chain into a deterministic computation conditioned on stored noise. "Fine-tuning Diffusion Policies with Backpropagation Through Diffusion Timesteps" introduces NCDPO, which treats the denoising process as a noise-conditioned deterministic policy and appends a simple output distribution over the final action,

$$q(a_t\mid a_0)=\mathcal{N}(a_t;\sqrt{\bar{\alpha}_t}a_0,(1-\bar{\alpha}_t)I),$$4

This makes the interactive action likelihood tractable and allows PPO gradients to backpropagate through all diffusion timesteps. The paper reports that NCDPO achieves sample efficiency comparable to MLP+PPO when training from scratch and is robust to the number of denoising timesteps (Yang et al., 15 May 2025).

Online RL has also been approached through variational and score-matching surrogates. QVPO introduces a Q-weighted variational loss and proves that it is a tight lower bound of the policy objective in online RL under certain conditions; it complements this with Q-weight transformations, a diffusion-specific entropy surrogate, and an efficient behavior policy for variance reduction (Ding et al., 2024). "Efficient Online Reinforcement Learning for Diffusion Policy" instead generalizes denoising score matching by reweighting the loss function, producing Reweighted Score Matching and two online algorithms: Diffusion Policy Mirror Descent and Soft Diffusion Actor-Critic. The paper emphasizes that these preserve the optimal solution and low computational cost of denoising score matching while eliminating the need to sample from the target distribution (Ma et al., 1 Feb 2025).

Optimization of the denoiser itself has also been treated as a first-class problem. ADPO introduces an Adam-based framework, with an ADAPG update that interpolates between current and momentum gradients and adds a Katyusha-style momentum term. Across Hopper, Walker2D, HalfCheetah, Lift, Can, and Square, the ADPO variants are reported to achieve better or comparable performance than six diffusion-based RL baselines, and the paper systematically analyzes the sensitivity of $$q(a_t\mid a_0)=\mathcal{N}(a_t;\sqrt{\bar{\alpha}_t}a_0,(1-\bar{\alpha}_t)I),$$5 and the momentum parameter $$q(a_t\mid a_0)=\mathcal{N}(a_t;\sqrt{\bar{\alpha}_t}a_0,(1-\bar{\alpha}_t)I),$$6 (Jiang et al., 13 May 2025). D2PPO addresses a distinct failure mode, diffusion representation collapse, by adding a dispersive loss regularizer that treats all hidden representations within a batch as negative pairs. On RoboMimic, D2PPO reports an average improvement of $$q(a_t\mid a_0)=\mathcal{N}(a_t;\sqrt{\bar{\alpha}_t}a_0,(1-\bar{\alpha}_t)I),$$7 in pre-training and $$q(a_t\mid a_0)=\mathcal{N}(a_t;\sqrt{\bar{\alpha}_t}a_0,(1-\bar{\alpha}_t)I),$$8 after fine-tuning, with early-layer regularization favoring simpler tasks and late-layer regularization favoring more complex manipulation (Zou et al., 4 Aug 2025).

3. Offline, regularized, and preference-driven optimization

Offline diffusion RL has focused heavily on regularization, because expressive diffusion actors can exploit out-of-distribution actions if unconstrained. BDPO develops a behavior-regularized framework in which the KL penalty between the learned diffusion policy and a behavior diffusion model is computed analytically as an accumulated discrepancy of reverse-time kernels:

$$q(a_t\mid a_0)=\mathcal{N}(a_t;\sqrt{\bar{\alpha}_t}a_0,(1-\bar{\alpha}_t)I),$$9

With isotropic fixed variance, each step reduces to a quadratic distance between reverse-process means. BDPO couples this pathwise KL with a two-time-scale actor-critic algorithm and reports state-of-the-art or near-state-of-the-art results on D4RL locomotion and AntMaze tasks (Gao et al., 7 Feb 2025).

DIPOLE reinterprets KL-regularized RL as bounded, weighted diffusion regression and introduces a greedified policy regularization scheme that decomposes the optimal policy into two dichotomous policies,

$$p_\theta(a_{t-1}\mid a_t,s)=\mathcal{N}(a_{t-1};\mu_\theta(a_t,t,s),\sigma_t^2 I).$$0

whose scores are combined at inference as

$$p_\theta(a_{t-1}\mid a_t,s)=\mathcal{N}(a_{t-1};\mu_\theta(a_t,t,s),\sigma_t^2 I).$$1

Because the weighting is bounded by sigmoid factors rather than unbounded exponentials, the method is designed to avoid the instability of direct value maximization and the burden of Gaussian likelihood approximations. The reported evaluations on ExORL, OGBench, and NAVSIM emphasize controllable greediness and stable large-policy training (Liang et al., 31 Dec 2025).

Preference-based optimization introduces another form of regularization. FKPD first trains a reference diffusion policy by behavior cloning and then aligns it to paired preferences by a DPO-style loss on denoising MSE differences, together with a forward-KL regularizer implemented through an ELBO or denoising-score-matching surrogate. The forward KL term is intended to keep the aligned policy close to the behavior distribution and avoid generating out-of-distribution actions. The paper reports superior alignment with preferences and improvements over previous methods on MetaWorld and D4RL tasks (Shan et al., 2024).

These offline and preference-based approaches share a common pattern. Rather than treating the diffusion policy as an unconstrained high-capacity actor, they bind it to a behavior policy, reference policy, or bounded weighting rule. This suggests that regularization to an explicit or implicit prior is not an auxiliary detail but a structural element of successful diffusion policy optimization in offline settings (Gao et al., 7 Feb 2025, Liang et al., 31 Dec 2025, Shan et al., 2024).

4. Diffusion LLMs and generative alignment

In masked diffusion LLMs, the policy object is not a continuous action but a denoising trajectory over partially masked token sequences. dTRPO addresses the cost of optimizing such models by proving two reduction results. First, under reference-policy regularization, the probability ratio of newly unmasked tokens is an unbiased estimate of the ratio for intermediate diffusion states. Second, the log-probability of the full trajectory can be estimated by sampling one time step per block and evaluating all sampled blocks in a single forward pass over a re-masked final state with block attention. These results yield a DPO-style objective that is offline and single-forward in nature. On 7B diffusion LLMs, dTRPO reports gains of up to $$p_\theta(a_{t-1}\mid a_t,s)=\mathcal{N}(a_{t-1};\mu_\theta(a_t,t,s),\sigma_t^2 I).$$2 on STEM tasks, up to $$p_\theta(a_{t-1}\mid a_t,s)=\mathcal{N}(a_{t-1};\mu_\theta(a_t,t,s),\sigma_t^2 I).$$3 on coding tasks, and up to $$p_\theta(a_{t-1}\mid a_t,s)=\mathcal{N}(a_{t-1};\mu_\theta(a_t,t,s),\sigma_t^2 I).$$4 on instruction-following tasks (Zhang et al., 19 Mar 2026).

Credit assignment inside the denoising trajectory is the focus of DiSPO. Instead of broadcasting a terminal reward uniformly across all diffusion steps, DiSPO selects intermediate masked states, branches by resampling fillings for currently masked positions from rollout-cached logits, scores the resulting completions, and updates only the newly filled tokens without additional multi-step diffusion rollouts. The method defines a fixed-state objective and a policy-gradient estimator over state-wise surrogate likelihoods, then combines this with terminal-feedback optimization. On LLaDA-8B-Instruct, DiSPO consistently improves over diffu-GRPO on math and planning benchmarks under matched rollout compute and optimizer steps (Oba et al., 6 Feb 2026).

Variational stabilization has become equally important. DiPOD identifies a double-drift phenomenon in diffusion policy-gradient methods: optimizing a variational surrogate can let the ELBO separate from the true log-likelihood, and that discrepancy then misaligns the proxy policy gradient with the true policy gradient of expected return. Its remedy is simple: augment each policy-gradient update with an on-policy ELBO regularizer. In the reported dLLM experiments, FPO+DiPOD improves GSM8K from $$p_\theta(a_{t-1}\mid a_t,s)=\mathcal{N}(a_{t-1};\mu_\theta(a_t,t,s),\sigma_t^2 I).$$5 to $$p_\theta(a_{t-1}\mid a_t,s)=\mathcal{N}(a_{t-1};\mu_\theta(a_t,t,s),\sigma_t^2 I).$$6 and Countdown from $$p_\theta(a_{t-1}\mid a_t,s)=\mathcal{N}(a_{t-1};\mu_\theta(a_t,t,s),\sigma_t^2 I).$$7 to $$p_\theta(a_{t-1}\mid a_t,s)=\mathcal{N}(a_{t-1};\mu_\theta(a_t,t,s),\sigma_t^2 I).$$8, while SPG+DiPOD raises Sudoku from $$p_\theta(a_{t-1}\mid a_t,s)=\mathcal{N}(a_{t-1};\mu_\theta(a_t,t,s),\sigma_t^2 I).$$9 to $\tau_t\in\bar{\mathcal V}^L$0 (Jiang et al., 11 Jun 2026).

A separate line removes policy ratios altogether. wd1 reformulates the objective as a weighted likelihood that requires only a single approximation for the current policy likelihood, replacing ratio-based importance sampling with symmetric positive and negative weights derived from group-relative advantages. The method is explicitly motivated by the observation that approximation errors in denominator likelihoods can induce large bias and variance in diffusion GRPO-style methods. On reasoning benchmarks, wd1 is reported to outperform existing RL methods for dLLMs without supervised fine-tuning or any supervised data, achieving up to $\tau_t\in\bar{\mathcal V}^L$1 higher accuracy, along with reduced training time and fewer function evaluations per gradient step (Tang et al., 7 Jul 2025).

The same denoising perspective has been extended beyond language to RLHF for diffusion and flow generators. AEGPO keeps the GRPO loss unchanged but reallocates compute globally across prompts according to the relative change in attention entropy and locally across denoising timesteps according to peaks of absolute attention entropy. On FLUX.1-dev and SD3.5-M, it improves HPS, PickScore, ImageReward, and GenEval, while the paper reports a $\tau_t\in\bar{\mathcal V}^L$2 speedup on DanceGRPO and a $\tau_t\in\bar{\mathcal V}^L$3 speedup on DiffusionNFT to reach baseline final reward (Li et al., 6 Feb 2026).

5. World models, planning, and systems efficiency

A distinct branch of diffusion policy optimization operates inside learned world models. MBDPO argues that the main scaling bottleneck is not only model bias or error compounding but also a structural misalignment between search and value learning: search policies query value functions trained under different state-action distributions. To address this, it reformulates policy optimization as a diffusion process over searched trajectories in latent world models, introduces a KL-constrained objective around a behavior policy, and learns an implicit energy function from data to anchor policy updates. The paper reports consistent and monotonic performance gains as model capacity increases from $\tau_t\in\bar{\mathcal V}^L$4M to $\tau_t\in\bar{\mathcal V}^L$5M parameters in multi-task offline pretraining, and it evaluates the method across 121 tasks spanning DMControl, MetaWorld, ManiSkill2, and MyoSuite (Cheng et al., 25 May 2026).

MODIP takes a more explicitly model-based route. Rather than applying RL directly to the diffusion actor, it trains a world model, performs MPC in latent space with a terminal state value,

$\tau_t\in\bar{\mathcal V}^L$6

and then distills the resulting optimized action chunks into the diffusion policy through the standard denoising loss. Two efficiency choices are central: using a terminal $\tau_t\in\bar{\mathcal V}^L$7 instead of $\tau_t\in\bar{\mathcal V}^L$8 during planning, and using policy-independent TD targets during critic training. On HalfCheetah, the paper reports an inference-time reduction from $\tau_t\in\bar{\mathcal V}^L$9 to $\pi_\theta(\tau_{t-1}\mid \tau_t,t)$0 seconds per $\pi_\theta(\tau_{t-1}\mid \tau_t,t)$1 environment steps and a training-time reduction from about $\pi_\theta(\tau_{t-1}\mid \tau_t,t)$2 hours to about $\pi_\theta(\tau_{t-1}\mid \tau_t,t)$3 hours (Asri et al., 9 Jun 2026).

Diffusion-policy optimization has also spread outside robot control. D3PG integrates a diffusion model into DDPG for Wi-Fi MAC optimization, jointly adjusting the contention window and aggregation frame length. In the reported dense 64-STA setting, it improves average throughput by $\pi_\theta(\tau_{t-1}\mid \tau_t,t)$4 over the standard Wi-Fi baseline, $\pi_\theta(\tau_{t-1}\mid \tau_t,t)$5 over PPO, and $\pi_\theta(\tau_{t-1}\mid \tau_t,t)$6 over DDPG, while maintaining more stable learning curves (Liu et al., 2024).

Adjacent work optimizes deployment cost rather than the training objective itself. The RTI scheme for diffusion policy warm-starts inference from the previous action chunk and runs only a few late denoising steps, yielding substantial reduction in inference time with comparable overall performance; the reported latency reduction is commonly $\pi_\theta(\tau_{t-1}\mid \tau_t,t)$7 to $\pi_\theta(\tau_{t-1}\mid \tau_t,t)$8 (Duan et al., 7 Aug 2025). LightDP compresses the denoising transformer and combines pruning with consistency distillation to reduce the number of sampling steps to four. On iPhone 13, the denoiser latency of DP-T drops from $\pi_\theta(\tau_{t-1}\mid \tau_t,t)$9 ms to $$\pi_\theta(\boldsymbol{\tau})=\prod_{t=1}^{T}p_\theta(\tau_{t-1}\mid \tau_t,t,\mu_\theta).$$0 ms and that of MDT-V from $$\pi_\theta(\boldsymbol{\tau})=\prod_{t=1}^{T}p_\theta(\tau_{t-1}\mid \tau_t,t,\mu_\theta).$$1 ms to $$\pi_\theta(\boldsymbol{\tau})=\prod_{t=1}^{T}p_\theta(\tau_{t-1}\mid \tau_t,t,\mu_\theta).$$2 ms, while maintaining competitive policy performance across PushT, RoboMimic, CALVIN, and LIBERO (Wu et al., 1 Aug 2025). Although these papers optimize inference rather than return, they address the same systems bottleneck created by multi-step denoising.

6. Empirical landscape, failure modes, and research directions

Across domains, diffusion policy optimization is most compelling where multimodality, long-horizon structure, or intermediate denoising decisions matter. In robotic fine-tuning, DPPO is reported as the first RL method to solve Transport to high success, while D2PPO shows especially large gains on Square and Transport when late-layer representations are regularized (Ren et al., 2024, Zou et al., 4 Aug 2025). In masked diffusion LLMs, dTRPO, DiSPO, DiPOD, and wd1 all report improvements that are largest on reasoning-intensive or planning-heavy benchmarks, where coarse terminal credit assignment and unstable likelihood surrogates are especially costly (Zhang et al., 19 Mar 2026, Oba et al., 6 Feb 2026, Jiang et al., 11 Jun 2026, Tang et al., 7 Jul 2025).

The main failure modes are remarkably consistent. Several papers identify intractable or poorly approximated likelihoods as the core technical obstacle, whether it appears as expensive trajectory probability calculation in dLLMs, full-chain likelihood estimation in on-policy RL, or ELBO-likelihood drift in variational policy gradients (Zhang et al., 19 Mar 2026, Liu et al., 5 Mar 2026, Jiang et al., 11 Jun 2026). Others emphasize data-support and critic-quality issues: offline methods can exploit out-of-distribution actions unless behavior regularization or forward-KL anchoring is explicit, and online methods that reweight score matching or variational objectives depend on sufficiently accurate Q estimates (Shan et al., 2024, Ma et al., 1 Feb 2025, Gao et al., 7 Feb 2025). Representation collapse in denoisers, estimator variance that scales with long diffusion horizons or large numbers of blocks, and world-model bias in imagined planning remain recurring problems (Zou et al., 4 Aug 2025, Zhang et al., 19 Mar 2026, Asri et al., 9 Jun 2026).

The field’s open problems are correspondingly concrete. Papers identify the need for better off-policy variants and variance reduction for score-matching objectives, general-purpose EUBO surrogates beyond absorbing-mask dLLMs, adaptive $$\pi_\theta(\boldsymbol{\tau})=\prod_{t=1}^{T}p_\theta(\tau_{t-1}\mid \tau_t,t,\mu_\theta).$$3 scheduling and layer-wise dispersion selection, uncertainty-aware planning in model-based diffusion control, longer-context reasoning in diffusion LLMs, and broader real-world deployment of large diffusion or vision-language-action policies (Ma et al., 1 Feb 2025, Jiang et al., 11 Jun 2026, Zou et al., 4 Aug 2025, Asri et al., 9 Jun 2026). This suggests an emerging synthesis: reliable diffusion policy optimization increasingly combines three ingredients—tractable surrogate objectives, explicit regularization to a behavior or reference distribution, and systems techniques that reduce the cost of multi-step denoising—rather than relying on any single policy-gradient or actor-critic template.