Data-Regularized Diffusion RL (DDRL)

• Data-regularized Diffusion Reinforcement Learning (DDRL) is a method that integrates score-based diffusion modeling with RL via data-anchored KL regularization to align policies with empirical distributions.
• It employs a two-term loss combining diffusion denoising on off-policy data and RL reward on on-policy rollouts to mitigate issues like reward hacking and trajectory instability.
• DDRL has demonstrated state-of-the-art performance in generative modeling, robotics manipulation, and safe offline RL by balancing reward maximization with data fidelity.

Data-regularized Diffusion Reinforcement Learning (DDRL) is a class of algorithms that tightly couples score-based diffusion generative modeling with reinforcement learning (RL) via explicit or implicit regularization to empirical data distributions. DDRL mitigates reward hacking, trajectory instability, and reward/generalization failures associated with standard RL fine-tuning of diffusion models by introducing direct penalties—typically forward or reverse KL divergences—to remain close to high-quality, off-policy data distributions during optimization. This approach has been applied at scale to large generative models, robotics manipulation policy synthesis, and safety-critical offline RL domains, with theoretical analyses demonstrating unbiased integration of reward maximization and diffusion loss minimization. The core DDRL paradigm manifests as a simple, provably grounded two-term loss: diffusion-based denoising on off-policy data plus explicit RL reward on on-policy rollouts.

1. Mathematical Objective and Theoretical Foundations

DDRL training is structured around an objective that combines a standard diffusion model loss with a data-anchored KL divergence. In large-scale generative modeling, the canonical DDRL objective is

$$L_\mathrm{total}(\theta) = L_\mathrm{diffusion}(\theta; \tilde{p}_\mathrm{data}) + \lambda \, \mathbb{E}_{x \sim \pi_\theta}\left[ \log \pi_\theta(x) - \log p_\mathrm{data}(x) \right],$$

where $L_\mathrm{diffusion}$ is the denoising loss on off-policy data or synthetic samples, and the second term implements a forward-KL regularizer anchoring $\pi_\theta$ to $p_\mathrm{data}$ (Ye et al., 3 Dec 2025).

Theoretical guarantees (Theorem 3.1 in (Ye et al., 3 Dec 2025)) show that maximizing the corresponding reward-regularized objective,

$$J_\mathrm{DDRL}(p_\theta) = \mathbb{E}_{x_0 \sim p_\theta}[\lambda(r(x_0)/\beta)] - KL(\tilde{p}_\mathrm{data} \parallel p_\theta),$$

yields the unique Boltzmann optimal solution $$p^*_\theta(x_0) \propto p_\mathrm{data}(x_0) \exp(r(x_0)/\beta)$$, free from on-policy regularization bias. In continuous-time score-based frameworks, DDRL appears as an RL problem over the mean of a time-dependent Gaussian reverse SDE, with the reward composed of a fidelity penalty to the data manifold plus a reward functional over terminal (or intermediate) samples (Gao et al., 7 Sep 2024).

2. Algorithmic Components

A typical DDRL workflow includes:

• Diffusion Model Pretraining: The model is first fit on offline or synthetic data using the standard denoising MSE loss, ensuring high-quality, multi-modal initialization (Yang et al., 24 Sep 2025, Ye et al., 3 Dec 2025).
• Reward Augmentation and RL Fine-Tuning:
  • In on-policy settings, RL proceeds via rollout sampling from the current diffusion model, with rewards computed by human preference or task-specific reward models (Ye et al., 3 Dec 2025).
  • In trajectory generation domains (e.g., robotics), PPO or general actor-critic updates are performed over the diffusion chain, treating each reverse-sampling step as a sub-action in the policy (Yang et al., 24 Sep 2025).
• Regularization: The forward-KL (or reverse-KL in some offline variants (2502.12391)) between the model and data distributions is explicitly incorporated into the loss. In practical settings, this is often reduced to routine denoising loss computations with respect to (possibly synthetic) data samples.
• Score/ration Estimation (for explicit data regularization): Some DDRL formulations use ratio-estimators or diffusion scores computed from the empirical data law to enable direct gradient estimation of the data-regularization term (Gao et al., 7 Sep 2024).

A generic pseudocode template for DDRL is as follows (adapted from (Ye et al., 3 Dec 2025)):

for iter in range(M): # Sample off-policy (data) batch x_tilde_0 ~ p_data # Rollout N on-policy samples and compute rewards x_0^n ~ p_theta for n = 1...N r^n = reward(x_0^n) A^n = (r^n - mean(r^n)) / std(r^n) for t in T: # Compute diffusion loss on x_tilde_0 x_tilde_t = sqrt(1 - beta_t)*x_tilde_0 + sqrt(beta_t)*epsilon l_diff = ||epsilon_theta(x_tilde_t, t) - epsilon||^2 # RL loss on on-policy sample l_RL = -A^n * log p_theta(x_t^n | x_{t+1}^n) accumulate l_t^n = l_diff + l_RL update theta via gradient of sum_n,t l_t^n

3. Forms of Data Regularization: Forward-KL, Reverse-KL, and Implicit Mechanisms

Forward-KL Regularization: The canonical DDRL approach for large-scale generative model alignment is to anchor $p_\theta$ directly to $p_\mathrm{data}$ via a forward-KL, thus regularizing toward the empirical data distribution and mitigating reward hacking and out-of-distribution collapse (Ye et al., 3 Dec 2025). The forward-KL is efficiently computed via denoising losses on off-policy samples.

Reverse-KL Regularization: In offline RL and safety-critical domains, such as Diffusion-Regularized Constrained Offline Reinforcement Learning (DRCORL), a reverse-KL $D_{KL}(\pi_\theta || \mu_\psi)$ is employed between the learned policy and a diffusion model fit to the behavioral dataset. This discourages excessive deviation from safe or trusted behaviors (2502.12391).

Implicit Data Regularization via Denoising: In physical trajectory generation, the multi-step denoising process itself acts as an effective implicit regularizer. Training the score model to predict noise at each denoising step constrains the action manifold to be smooth and consistent, enforcing low-variance and reducing trajectory jitter compared with standard Gaussian RL (Yang et al., 24 Sep 2025).

4. Empirical Performance and Qualitative Properties

DDRL demonstrates superior quantitative and qualitative outcomes across diverse domains.

Generative Models (Video/Image):

• DDRL consistently increases both reward scores and human preference relative to baselines reliant on reverse-KL or reward-only optimization. For example, on high-resolution video with Cosmos2.5-2B, DDRL improves reward to 0.604 (VideoAlign T2V) with a Δ-vote preference of 0%, while DanceGRPO achieves higher raw reward (0.715*) but suffers negative human preference (–10.5%), reflecting reward hacking (Ye et al., 3 Dec 2025).
• In OCR-constrained image generation, DDRL achieves a balance: OCR = 0.823 with no decrease in human realism or OOD metrics, whereas baseline RL methods sacrifice human preference for raw metric gains.

Synthetic Data Generation (Robotics/Manipulation):

• On the LIBERO benchmark, diffusion RL-generated data achieves average success rate (SR) of 81.94%, outperforming both human demonstrations (76.64%) and Gaussian RL data (69.32%). Action variance and trajectory jerk are also dramatically reduced (Yang et al., 24 Sep 2025).
• Zero-shot OOD generalization improves from 1.47% (human-only) to 5.20% (human + diffusion RL), suggesting that data-regularized synthetic data aids in broad generalization.

Offline Safe RL:

• DRCORL achieves cost-normalized safety (C_norm ≈ 0) while matching or exceeding reward records set by prior algorithms. Inference speedups of 10× versus diffusion-only policies are achieved by extracting a Gaussian actor that is regularized by the underlying diffusion score model (2502.12391).

5. Algorithmic Variants and Practical Considerations

Variant	Domain	Regularizer
Forward-KL DDRL	Generative modeling	Forward-KL (data)
Reverse-KL DDRL (DRCORL)	Offline safe RL	Reverse-KL
Implicit denoising DDRL	Manipulation/trajectory	Denoising loss
Ratio-estimator DDRL	Model-free diffusion RL	Data score ratio

• Sampling and Sampler Types: Standard DDRL implementations use SDEs or their ODE analogs (e.g., probability-flow ODE, DDIM) for generation, with the loss formulation unchanged (Gao et al., 7 Sep 2024, Ye et al., 3 Dec 2025).
• Initialization: A warm-start phase via multimodal behavior cloning, or pretraining on synthetic/real data, is used to anchor the initial policy distribution (Yang et al., 24 Sep 2025, Ye et al., 3 Dec 2025).
• Reward Normalization and Control: Reward scores can be transformed or centered (e.g., via exponential or batch mean normalization) to stabilize optimization (Ye et al., 3 Dec 2025).
• Hyperparameters: Regularization strength (λ, β), diffusion steps, and learning rates control the tradeoff between data proximity and reward maximization. Empirically, square-root scheduling of regularization yields optimal reward–stability trade-offs in DRCORL (2502.12391).

6. Limitations, Pitfalls, and Mitigation of Reward Hacking

Reverse-KL regularization, as in many standard diffusion RL methods, uses on-policy samples and provides unreliable penalties, which leads to phenomena such as reward hacking, quality degradation, over-stylization, and mode collapse (Ye et al., 3 Dec 2025). DDRL’s forward-KL objective explicitly anchors optimization to empirical data, eliminating on-policy bias and resulting in empirically robust behavior, as evidenced by absence of reward hacking artefacts across over one million GPU-hours of video generation (Ye et al., 3 Dec 2025). DDRL retains high OOD fidelity, human realism, and overall alignment at scale.

In offline RL, care is required to balance the strength of the KL regularizer; too strong a penalty suppresses generalization, while a weak penalty may not prevent unsafe out-of-distribution actions. Integration of gradient manipulation schemes further improves monotonic safety-reward trade-offs (2502.12391).

7. Extensions and Domain-Specific Instantiations

• Conditional and ODE-based DDRL: The framework directly extends to conditional generation by modifying the running reward and score estimators to reflect conditioning variables (Gao et al., 7 Sep 2024).
• Trajectory Smoothing and Variance Reduction: In policy generation for sequential decision making, iterative denoising provides smoothness and variance reduction, with few (as low as 5) deterministic DDIM steps sufficient for high-quality samples (Yang et al., 24 Sep 2025).
• Safe Policy Extraction: In safety-critical domains, simplified Gaussian actors can be efficiently extracted from trained diffusion models without test-time diffusion sampling, enabling high-throughput, safe inference (2502.12391).

In sum, DDRL unifies score-based diffusion model training and explicit reward-based RL via direct, data-anchored regularization and has demonstrated state-of-the-art performance and scalability in generative modeling, robotics, and safe offline RL domains (Ye et al., 3 Dec 2025, Yang et al., 24 Sep 2025, 2502.12391, Gao et al., 7 Sep 2024).