Diffusion-Augmented Reinforcement Learning

• Diffusion-Augmented Reinforcement Learning (DARL) is a family of frameworks that integrate diffusion processes into RL pipelines to generate trajectories, guide policy learning, and synthesize data with enhanced expressiveness and stability.
• DARL employs diffusion models in roles such as planners, policies, and data augmenters, enabling robust control and improved multimodal exploration in robotics, finance, and generative post-training.
• Key design patterns include diffusion-guided candidate generation with RL selection, adaptive inference scheduling, and embedding diffusion in representation learning to mitigate high online computational costs.

Searching arXiv for papers on diffusion-augmented reinforcement learning and related surveys. Diffusion-Augmented Reinforcement Learning (DARL) denotes a family of reinforcement-learning frameworks in which diffusion models are integrated into the RL pipeline as planners, policies, data synthesizers, world models, reward models, or post-training policies, with the aim of improving multimodal expressiveness, stability, robustness, long-horizon planning, or inference-time control of computation (Xu et al., 14 Oct 2025, Zhu et al., 2023). In current usage, the label is not fully standardized: some papers use “Diffusion-Augmented Reinforcement Learning” for robotics, autonomous underwater control, portfolio optimization, or diffusion-language-model post-training, while other papers use the acronym “DARL” for unrelated expansions such as “Dynamics-Agnostic Discriminator Ensemble” or “Denoising Autoregressive Representation Learning” (Liu et al., 15 Jul 2025, Choudhary et al., 8 Oct 2025, Kunde et al., 13 Mar 2026, Luo et al., 2022, Li et al., 2024). Within the diffusion-RL literature proper, however, the common structure is the use of a diffusion process to model or guide actions, trajectories, latent plans, or data distributions, combined with RL objectives, value functions, or policy-gradient updates (Xu et al., 14 Oct 2025, Zhu et al., 2023).

1. Terminological scope and historical placement

The phrase “Diffusion-Augmented Reinforcement Learning” is best understood as a broad methodological category rather than a single algorithm. A recent survey organizes the field along two orthogonal dimensions: a function-oriented taxonomy specifying the role of the diffusion model within the RL stack, and a technique-oriented taxonomy distinguishing online from offline regimes (Xu et al., 14 Oct 2025). An earlier survey presents a similar role-based taxonomy in which diffusion models act as planners, policies, data synthesizers, or auxiliary components for RL-related tasks (Zhu et al., 2023).

Within this broad category, several concrete instantiations have appeared. In robotics, diffusion models have been used as action priors or trajectory generators combined with RL residual policies or critics (Yang et al., 13 Jun 2025, Liu et al., 15 Jul 2025). In embodied agents, diffusion has been used to transform past observations into hindsight-consistent task data for transfer and sample efficiency (Palo et al., 2024). In finance, a conditional DDPM has been used to generate synthetic market-crash trajectories for PPO-based portfolio optimization under stress scenarios (Choudhary et al., 8 Oct 2025). In diffusion LLMs, RL has been formulated directly over denoising trajectories, with exact policy gradients decomposed over diffusion steps (Kunde et al., 13 Mar 2026). In generative post-training, diffusion policies themselves have been optimized by RL with forward-KL data regularization (Ye et al., 3 Dec 2025).

A source of ambiguity is that the acronym “DARL” is overloaded. “Transferable Reward Learning by Dynamics-Agnostic Discriminator Ensemble” explicitly states that its DARL is not “Diffusion-Augmented Reinforcement Learning” and contains no diffusion mechanism (Luo et al., 2022). “DARL: Encouraging Diverse Answers for General Reasoning without Verifiers” also states that its “diffusion” is conceptual rather than a continuous diffusion process (Huang et al., 21 Jan 2026). Conversely, “Ocean Diviner” uses “Diffusion-Augmented Reinforcement Learning” in the concrete sense of a diffusion model generating candidate trajectories for an RL controller (Liu et al., 15 Jul 2025). This suggests that encyclopedia treatment of DARL must distinguish the broad diffusion-RL paradigm from acronym collisions.

2. Architectural roles of diffusion within RL

The most systematic description of DARL is the function-oriented taxonomy in which diffusion models can appear in at least six roles: diffusion-based trajectory optimization, diffusion-based policy learning, diffusion-based imitation learning, diffusion-based exploration augmentation, diffusion-based environmental simulation, and diffusion-based reward modeling (Xu et al., 14 Oct 2025). The earlier survey emphasizes a closely related set of roles—planner, policy, data synthesizer, and auxiliary modules—and motivates them through four recurring RL challenges: restricted expressiveness in offline learning, data scarcity in experience replay, compounding error in model-based planning, and generalization in multitask learning (Zhu et al., 2023).

As planners, diffusion models denoise an entire trajectory or plan conditioned on goals, returns, or constraints, rather than rolling forward one-step dynamics models autoregressively. This trajectory-level denoising is intended to reduce compounding error and maintain temporal consistency over long horizons (Xu et al., 14 Oct 2025, Zhu et al., 2023). As policies, diffusion models replace unimodal Gaussian actors with expressive conditional generative models over actions or action chunks, which is particularly attractive in robotics where action distributions are often multimodal (Xu et al., 14 Oct 2025). As data synthesizers, diffusion models generate additional trajectories, market regimes, or visually transformed hindsight observations that expand training support beyond the original replay buffer (Palo et al., 2024, Choudhary et al., 8 Oct 2025). As post-training policies for generative diffusion models, RL acts directly on denoising transitions while diffusion loss provides regularization to the data manifold (Ye et al., 3 Dec 2025).

A useful distinction is between methods that sample from diffusion at deployment time and methods that use diffusion only during representation learning. Most trajectory-diffusion and diffusion-policy methods incur iterative reverse-denoising cost during acting or planning (Liu et al., 15 Jul 2025, Yang et al., 13 Jun 2025, Yu et al., 9 Aug 2025). By contrast, Diffusion Spectral Representation learns value-sufficient features from a diffusion/energy-based view of transition dynamics and then performs ordinary value-based control without reverse-diffusion sampling at inference time (Shribak et al., 2024). This suggests two broad design patterns within DARL: diffusion-as-online-generator and diffusion-as-offline-representation-learner.

3. Core mathematical formulations

A large fraction of DARL methods inherit the standard DDPM or DDIM formalism. A forward noising process gradually corrupts data, for example

$$q(x_t \mid x_{t-1}) = \mathcal{N}\big(\sqrt{1-\beta_t}\,x_{t-1}, \beta_t I\big),$$

with the associated marginal

$$x_t = \sqrt{\bar{\alpha}_t}x_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon, \quad \epsilon \sim \mathcal{N}(0,I),$$

while a learned reverse process predicts either noise or the clean sample to recover $x_0$ from noisy latents (Liu et al., 15 Jul 2025, Choudhary et al., 8 Oct 2025, Li et al., 2024, Ye et al., 3 Dec 2025). In RL settings, the denoised object may be an action chunk, a state-action trajectory, a sequence of returns, or a market-return path rather than an image (Liu et al., 15 Jul 2025, Choudhary et al., 8 Oct 2025, Xu et al., 14 Oct 2025).

When diffusion models act as trajectory generators or action priors, RL typically enters through one of three mechanisms. The first is value-guided sampling, in which a critic or return model scores candidate trajectories or provides gradients that bias denoising toward high-return regions (Liu et al., 15 Jul 2025, Xu et al., 14 Oct 2025). The second is policy-gradient fine-tuning of the diffusion policy itself, as in DPPO-style robotics methods or diffusion post-training for media generation, where denoising transitions become the action space of a higher-level RL problem (Yu et al., 9 Aug 2025, Ye et al., 3 Dec 2025). The third is hybridization, where diffusion proposes trajectories or actions and an RL policy, critic, or residual controller selects or corrects them (Liu et al., 15 Jul 2025, Yang et al., 13 Jun 2025).

In diffusion LLMs, the denoising chain can be written explicitly as a finite-horizon MDP. For a prompt $q$, state $s_t = (x_{T-t}, q)$, action $a_t = x_{T-t-1}$, deterministic environment transition over denoising time, and sparse terminal reward $r(x_0,q)$, the exact policy gradient decomposes over denoising steps: $$\nabla_\pi J(\pi) = \sum_{t=0}^{T-1} \mathbb{E}_{q,x\sim \pi} \big[ A_t^\pi(x_{t+1},x_0,q)\nabla_\pi \log \pi(x_t\mid x_{t+1}) \big],$$ with stepwise advantage

$$A_t^\pi(x_{t+1},x_0,q)=r(x_0,q)-V_{t+1}^\pi(x_{t+1},q).$$

This formulation avoids explicit sequence-likelihood evaluation and uses one-step denoising rewards and entropy-guided step selection for efficiency (Kunde et al., 13 Mar 2026).

In generative post-training, DDRL replaces the usual on-policy reverse-KL regularization with a forward-KL term anchored to an off-policy data distribution. The theoretical objective

$$\mathcal J_{\mathrm{DDRL}}(p_\theta) = \mathbb E_{p_\theta(x_0\mid c)} \Big[ \lambda\Big(\frac{r(x_0,c)-Z}{\beta}\Big) \Big] - \mathrm{KL}\big(\tilde p_{\mathrm{ref}}\Vert p_\theta\big)$$

is shown to be equivalent to maximizing the transformed reward while minimizing standard diffusion loss on an off-policy data sampler $$x_t = \sqrt{\bar{\alpha}_t}x_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon, \quad \epsilon \sim \mathcal{N}(0,I),$$0, yielding an optimal policy of the form

$$x_t = \sqrt{\bar{\alpha}_t}x_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon, \quad \epsilon \sim \mathcal{N}(0,I),$$1

(Ye et al., 3 Dec 2025). This is one of the clearest examples of DARL as a principled fusion of diffusion training and RL regularization.

4. Major design patterns

The literature described in the surveys and recent task-specific papers supports four especially prominent DARL patterns (Xu et al., 14 Oct 2025, Zhu et al., 2023).

The first is diffusion-guided proposal and RL selection. In “Ocean Diviner,” a diffusion model conditioned on a history-augmented AUV state

$$x_t = \sqrt{\bar{\alpha}_t}x_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon, \quad \epsilon \sim \mathcal{N}(0,I),$$2

with $$x_t = \sqrt{\bar{\alpha}_t}x_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon, \quad \epsilon \sim \mathcal{N}(0,I),$$3 generates $$x_t = \sqrt{\bar{\alpha}_t}x_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon, \quad \epsilon \sim \mathcal{N}(0,I),$$4 candidate actions or trajectory fragments, and a TD3 critic selects the argmax-Q candidate. The diffusion model uses a 6-layer U-Net with $$x_t = \sqrt{\bar{\alpha}_t}x_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon, \quad \epsilon \sim \mathcal{N}(0,I),$$5 diffusion steps and a linear variance schedule $$x_t = \sqrt{\bar{\alpha}_t}x_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon, \quad \epsilon \sim \mathcal{N}(0,I),$$6 (Liu et al., 15 Jul 2025). This makes diffusion a structured proposal distribution and the critic a planner over those proposals.

The second is frozen diffusion prior plus RL residual policy. In Multi-Loco, a morphology-agnostic diffusion model trained offline on multi-robot demonstrations samples an action in a unified padded action space, and a shared residual policy trained with PPO refines it: $$x_t = \sqrt{\bar{\alpha}_t}x_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon, \quad \epsilon \sim \mathcal{N}(0,I),$$7 The residual is regularized by

$$x_t = \sqrt{\bar{\alpha}_t}x_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon, \quad \epsilon \sim \mathcal{N}(0,I),$$8

to stay close to the diffusion prior (Yang et al., 13 Jun 2025). This pattern is especially natural when diffusion serves as a behavior prior learned from rich offline data and RL supplies embodiment-specific adaptation.

The third is diffusion-based data augmentation. In DAAG, a diffusion model edits trajectories or observations so that old experience visually aligns with new goals, a process called Hindsight Experience Augmentation. The transformed observations

$$x_t = \sqrt{\bar{\alpha}_t}x_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon, \quad \epsilon \sim \mathcal{N}(0,I),$$9

are used to create synthetic successful trajectories for new tasks, while a VLM serves as reward detector and an LLM orchestrates subgoal decomposition and swap selection (Palo et al., 2024). In portfolio optimization, a conditional DDPM similarly generates synthetic stress trajectories $x_0$0 for crash intensity $x_0$1, augmenting historical data for PPO training (Choudhary et al., 8 Oct 2025).

The fourth is adaptive control of diffusion inference itself. D3P treats the number of denoising steps as an RL-controlled variable. A lightweight adaptor $x_0$2 observes the current environment state and noisy action chunk and outputs a stride $x_0$3 that determines how far to jump in the noise schedule: $x_0$4 The adaptor is trained by PPO with rewards that combine a task-level advantage estimate and a penalty for excessive denoising steps, allowing larger computational budgets for crucial actions and smaller budgets for routine ones (Yu et al., 9 Aug 2025). This suggests a broader DARL theme of RL over inference-time computation.

5. Domain-specific instantiations

Robotics and embodied control

Robotics is the most visible application area of DARL. “Ocean Diviner” targets robust AUV control in underwater data-collection tasks. Its diffusion model produces candidate multi-step trajectories, TD3 evaluates them, and an S-Surface controller tracks the chosen high-level plan. The environment is formulated as a control-affine MDP

$x_0$5

with $x_0$6, $x_0$7 candidate actions, and disturbances representing waves and currents (Liu et al., 15 Jul 2025).

In simulation, Ocean Diviner reports that Diffusion+RL converges in about 300 episodes, nearly twice as fast as standard RL, and that under extreme sea conditions the S-Surface version achieves $x_0$8 MBit/s SDR and $x_0$9 SSN in ES, and $q$0 MBit/s SDR and $q$1 SSN in VES (Liu et al., 15 Jul 2025). A plausible implication is that diffusion helps most when exploration needs temporal coherence and physical plausibility rather than i.i.d. action noise.

Multi-Loco extends diffusion-augmented control to multi-embodiment locomotion. Observations and actions are padded to unified spaces, invalid action dimensions are masked, and an EDM-based diffusion prior is trained on demonstrations from four robot morphologies. The full system, CR-DP+RA, combines this cross-robot diffusion prior with a shared RL residual actor and robot-specific critics (Yang et al., 13 Jun 2025). Quantitatively, it reports a 10.35% average return improvement over a standard PPO baseline, with gains up to 13.57% in wheeled-biped locomotion, and cross-robot diffusion improves over single-robot diffusion by 17.96% on average (Yang et al., 13 Jun 2025).

D3P addresses a different robotics bottleneck: diffusion-policy latency. It reports an average 2.2× inference speed-up over baselines in simulation while maintaining success, and a 1.9× acceleration on a physical Franka robot in the Square task (Yu et al., 9 Aug 2025). This is significant because it shows that RL can optimize not only the control behavior of a diffusion policy but also its deployment-time computational schedule.

Finance

In portfolio optimization, DARL takes the form of stress-scenario generation. A conditional DDPM models return sequences conditioned on a crash intensity $q$2, with the standard forward process

$q$3

and a reverse model $q$4 that predicts noise $q$5 (Choudhary et al., 8 Oct 2025). Synthetic crisis trajectories are then mixed with historical data during PPO training in a custom portfolio environment.

On Dow 30 data from 2011–2025, the proposed DARL portfolio method reports cumulative return $q$6, annualized return $q$7, Sharpe ratio $q$8, Calmar ratio $q$9, and max drawdown $s_t = (x_{T-t}, q)$0, outperforming the same PPO framework without augmentation, FinRL-PPO, OLMAR, Hybrid-GA, Markowitz, and the index benchmark (Choudhary et al., 8 Oct 2025). The comparison against the no-augmentation version isolates the effect of diffusion-based stress synthesis: cumulative return rises from $s_t = (x_{T-t}, q)$1 to $s_t = (x_{T-t}, q)$2, Sharpe from $s_t = (x_{T-t}, q)$3 to $s_t = (x_{T-t}, q)$4, and max drawdown improves from $s_t = (x_{T-t}, q)$5 to $s_t = (x_{T-t}, q)$6 (Choudhary et al., 8 Oct 2025).

Diffusion LLMs and generative post-training

A separate but increasingly important branch of DARL studies RL directly on diffusion generative models rather than using diffusion to aid a conventional control problem. In diffusion LLMs, denoising itself is treated as the control process. The “entropy-guided step selection and stepwise advantages” framework derives an exact policy gradient over denoising steps and uses one-step denoising rewards as approximate value estimates: $s_t = (x_{T-t}, q)$7 It then selects the top-$s_t = (x_{T-t}, q)$8 denoising steps by entropy because the gradient error from omitting steps can be bounded by the sum of entropies of omitted denoising distributions (Kunde et al., 13 Mar 2026).

Empirically, this framework reports strong results on logical reasoning and coding, including HumanEval best 44.5 for EGSPO-SA versus 37.8 for the base model and 37.8 for d1, and MBPP best 51.1 versus 41.2 for the base model and 44.7 for d1 (Kunde et al., 13 Mar 2026). This suggests that diffusion time provides a useful axis for RL credit assignment distinct from token time in autoregressive models.

DDRL studies RL for high-resolution image and video diffusion models with human-preference rewards. Its central claim is that reverse-KL regularization to a reference policy is unreliable because it is evaluated on on-policy trajectories in out-of-distribution regions, whereas forward-KL anchoring to an off-policy data sampler is equivalent to adding ordinary diffusion loss on data during RL (Ye et al., 3 Dec 2025). Across large-scale video experiments involving over a million GPU hours and ten thousand double-blind human evaluations, DDRL is reported to improve rewards while alleviating reward hacking and to achieve the highest human preference among compared methods (Ye et al., 3 Dec 2025). A plausible implication is that forward-KL data anchoring may become a standard stabilization principle for RL on diffusion generators.

6. Representation learning and alternative uses of diffusion

Not all diffusion-augmented RL methods use reverse diffusion at inference time. Diff-SR explicitly argues that iterative denoising is too costly for broad real-world deployment and therefore uses diffusion from a representation-learning perspective (Shribak et al., 2024). Starting from an energy-based transition model

$s_t = (x_{T-t}, q)$9

the method applies Gaussian corruption to next states, invokes Tweedie’s identity, and derives a regression objective

$a_t = x_{T-t-1}$0

for learning spectral features that are sufficient for value functions (Shribak et al., 2024). It then defines a finite-dimensional feature map $a_t = x_{T-t-1}$1 and a linear critic

$a_t = x_{T-t-1}$2

Across fully observable MuJoCo tasks, partially observable MuJoCo variants, and image-based Meta-World POMDPs, Diff-SR is reported to match or exceed competitive baselines while being about $a_t = x_{T-t-1}$3 faster in wall-clock time than PolyGRAD in MDP benchmarks and $a_t = x_{T-t-1}$4–$a_t = x_{T-t-1}$5 faster in POMDPs because no reverse-diffusion sampling is needed at action time (Shribak et al., 2024). This broadens the meaning of DARL: diffusion need not appear as a deployed policy or planner; it can instead provide sufficient statistics for downstream RL.

An even more unusual diffusion-oriented RL perspective appears in Maximum Diffusion Reinforcement Learning, where “diffusion” refers not to denoising generative models but to ergodic diffusion processes in state space (Berrueta et al., 2023). The method derives a maximum-entropy path distribution over continuous trajectories subject to continuity constraints, yielding a maximally diffusive target process and a trajectory-space KL objective

$a_t = x_{T-t-1}$6

It proves that MaxDiff RL generalizes MaxEnt RL and shows strong robustness and single-shot learning behavior under ergodicity assumptions (Berrueta et al., 2023). Although methodologically distant from DDPM-style models, it illustrates that diffusion-augmented RL can also denote physically grounded diffusion-process regularization.

7. Empirical themes across the literature

Several empirical patterns recur across the DARL literature.

A first theme is improved handling of multimodality. The surveys explicitly motivate diffusion in offline RL by the failure of unimodal Gaussian policies to fit heterogeneous behavior distributions (Zhu et al., 2023, Xu et al., 14 Oct 2025). In robotics, diffusion priors and proposal models exploit this by generating diverse, coherent candidate actions or gait patterns, which critics or residual policies then refine (Liu et al., 15 Jul 2025, Yang et al., 13 Jun 2025).

A second theme is robustness under distribution shift or rare events. Multi-Loco uses cross-embodiment data to build a morphology-agnostic locomotion prior that transfers to novel morphologies and hardware (Yang et al., 13 Jun 2025). The portfolio DARL framework improves out-of-sample robustness to the 2025 Tariff Crisis by exposing PPO to synthetic crash regimes during training (Choudhary et al., 8 Oct 2025). DDRL reduces reward hacking in video and image generation by anchoring RL updates to a stable data distribution (Ye et al., 3 Dec 2025).

A third theme is sample efficiency through structured exploration or augmentation. DAAG reduces the need for reward-labeled data and new task interaction by editing old trajectories into new-task successes and using those to train reward detectors and policies (Palo et al., 2024). Ocean Diviner uses diffusion-based candidate generation to replace unstructured action noise with temporally coherent exploration (Liu et al., 15 Jul 2025). A plausible implication is that diffusion adds the most value when exploration requires long-range consistency rather than local perturbation.

A fourth theme is computation as a first-class optimization target. D3P shows that diffusion-policy inference depth can be scheduled adaptively by RL (Yu et al., 9 Aug 2025). Diff-SR avoids online diffusion sampling entirely (Shribak et al., 2024). This theme addresses one of the most persistent practical criticisms of diffusion in RL: slow sampling at control time (Shribak et al., 2024, Yu et al., 9 Aug 2025).

8. Limitations and controversies

The most widely acknowledged limitation of DARL is inference cost. Diffusion policies and trajectory planners often require several denoising evaluations per decision, which is substantially slower than standard MLP or transformer policies (Shribak et al., 2024, Yu et al., 9 Aug 2025). D3P is an explicit response to this limitation, and Diff-SR avoids it by moving diffusion into representation learning. A plausible implication is that future DARL systems will either use few-step or adaptive samplers or distill diffusion models into cheaper surrogates.

A second limitation is dependence on data quality and coverage. Portfolio DARL can only generate crisis scenarios supported by historical crisis structure (Choudhary et al., 8 Oct 2025). Multi-Loco depends on observation-action datasets from existing controllers and cannot yet leverage action-free data such as motion capture (Yang et al., 13 Jun 2025). DAAG relies on diffusion edits being geometrically and temporally consistent enough that the resulting synthetic trajectories remain useful for control (Palo et al., 2024).

A third issue is reward reliability. DDRL is motivated precisely by reward hacking under weak regularization (Ye et al., 3 Dec 2025). In diffusion LLM RL, one-step denoising values and entropy-guided step selection are practical approximations rather than exact long-horizon evaluations (Kunde et al., 13 Mar 2026). More generally, when RL is used to optimize diffusion models with learned rewards, the standard failure modes of reward misspecification remain.

A fourth issue is terminological ambiguity. Because “DARL” is also used for non-diffusion methods and even non-RL methods, not every paper using the acronym belongs to the diffusion-RL family (Luo et al., 2022, Li et al., 2024, Huang et al., 21 Jan 2026). This can create confusion in bibliographic searches and literature reviews. The surveys instead recommend focusing on functional roles of diffusion within RL rather than on acronym usage (Xu et al., 14 Oct 2025, Zhu et al., 2023).

9. Relationship to adjacent research areas

DARL sits at the intersection of diffusion modeling, offline RL, model-based planning, imitation learning, and generative post-training. Relative to classical model-based RL, trajectory-diffusion planners replace one-step predictive rollouts with direct sequence generation (Zhu et al., 2023). Relative to behavior cloning and offline RL, diffusion offers a more expressive prior over actions and trajectories (Xu et al., 14 Oct 2025). Relative to standard maximum-entropy RL, Maximum Diffusion RL elevates the entropy principle from action distributions to path distributions (Berrueta et al., 2023). Relative to post-training of generative models, DDRL and diffusion-LLM RL show that denoising trajectories themselves can be treated as decision processes subject to RL optimization (Ye et al., 3 Dec 2025, Kunde et al., 13 Mar 2026).

The field also interacts with multi-agent and multi-embodiment learning. Multi-Loco shows how zero-padding, masking, and a shared residual actor can use diffusion to learn morphology-invariant locomotion patterns (Yang et al., 13 Jun 2025). The 2025 survey highlights joint-trajectory and decentralized multi-agent diffusion methods as an expanding branch of diffusion-based RL (Xu et al., 14 Oct 2025). This suggests that the most natural next extensions of DARL may lie in compositionality: across agents, embodiments, task families, and modalities.

10. Outlook

The available evidence suggests that Diffusion-Augmented Reinforcement Learning is converging toward a set of recurring principles rather than a single canonical algorithm. Diffusion can provide expressive priors over actions and trajectories, coherent synthetic data, denoising-time structure for policy gradients, or value-sufficient representations; RL can in turn guide diffusion sampling, fine-tune diffusion parameters, select among diffusion proposals, schedule denoising computation, or regularize generative policies against data (Liu et al., 15 Jul 2025, Yang et al., 13 Jun 2025, Choudhary et al., 8 Oct 2025, Kunde et al., 13 Mar 2026, Ye et al., 3 Dec 2025, Shribak et al., 2024).

The surveys identify several open directions: faster samplers and hardware-aware implementations, online stability, safety and constraint integration, partial observability, larger-scale multi-agent settings, and tighter theory for diffusion-based policies and planners (Xu et al., 14 Oct 2025, Zhu et al., 2023). The recent task-specific literature sharpens these into concrete agendas: adaptive computation for deployment-time latency (Yu et al., 9 Aug 2025), data-anchored regularization against reward hacking (Ye et al., 3 Dec 2025), cross-embodiment priors for robust control (Yang et al., 13 Jun 2025), and generative stress synthesis for tail-risk robustness (Choudhary et al., 8 Oct 2025). Taken together, these works indicate that DARL is evolving from a set of isolated demonstrations into a coherent research program on how denoising generative structure can be exploited within reinforcement learning.