---
title: Diffusion-based Reward Model (DRM)
url: https://www.emergentmind.com/topics/diffusion-based-reward-model-drm
type: topic
---

# Diffusion-based Reward Model (DRM)

Searching arXiv for the specified DRM papers and closely related work to ground the article.
{"query":"2508.01509 A Reward-Directed Diffusion Framework for Generative Design Optimization 2512.04985 Towards a unified framework for guided diffusion models 2605.25661 DRM: Diffusion-based Reward Model With Step-wise Guidance", "max_results": 10}
Searching arXiv for foundational and recent DRM-related papers.
{"query":"Diffusion-based Reward Model arXiv 2508.01509 2512.04985 2605.25661", "top_k": 10, "source": "arxiv"}
A diffusion-based reward model (DRM) is a diffusion-centered mechanism that uses a reward, value, preference, or evaluative signal to bias generation toward preferred outputs. In the recent literature, the term does not denote a single standardized construction. Instead, it appears in several closely related senses: a reward-directed diffusion generator whose reverse process is reweighted toward high-value states; a reward-reweighted score model used for guided sampling; a diffusion-based discriminator whose output is converted into a policy reward; a temporal critic defined over noisy diffusion states; and a diffusion backbone repurposed as a step-wise evaluator of intermediate latents [2508.01509], [2512.04985], [2605.25661]. What unifies these usages is that reward information is not merely an external post hoc selector, but is structurally coupled to diffusion training, sampling, or both.

## 1. Conceptual scope and definitions

In reward-directed generative optimization, a DRM is a denoising diffusion probabilistic model (DDPM) whose reverse process policy is shaped by a soft value function associated with a terminal reward \(R(x_0)\), so that intermediate marginals are reweighted toward high-value states and decoded samples concentrate on high-performance designs [2508.01509]. In the continuous-time guided-diffusion formulation, the same idea is expressed as a reward-reweighted data distribution \(p_{x_0^{r\text{-}\mathsf{wt}}}(x_0) \propto r(x_0)\,p_{x_0}(x_0)\), with guidance implemented by the score difference between the original and reward-reweighted marginals [2512.04985]. In diffusion alignment for image generation, DRM can also mean a pre-trained diffusion model used as an evaluative backbone that scores noisy intermediate latents \(x_t\) as well as final samples \(x_0\) [2605.25661].

A second family of usages treats the reward model itself as diffusion-based. In Diffusion-Reward Adversarial Imitation Learning, the DRM is a reward function derived from a diffusion discriminative classifier \(D_\phi(s,a)\), with reward given by the AIRL/GAIL logit \(r_\phi(s,a)=\log D_\phi(s,a)-\log(1-D_\phi(s,a))\) [2405.16194]. In offline preference-based reinforcement learning, DPR and C-DPR define rewards from diffusion-based preference distributions over state-action pairs rather than from Bradley–Terry models over trajectories [2503.01143]. In visual RL from expert videos, “Diffusion Reward” denotes a reward computed from the negative conditional entropy of a conditional video diffusion model trained on expert trajectories [2312.14134].

This suggests that DRM is best understood as a family resemblance term. The common structure is a diffusion process coupled to a reward-bearing object—terminal reward, value function, preference signal, discriminator output, or conditional entropy—so that the induced sampling or control law is tilted toward preferred behavior.

## 2. Reward-directed diffusion as a generative optimizer

A canonical DRM formulation appears in engineering design optimization. The design variable is a parametric vector \(x \in \mathbb{R}^d\), the scalar performance is \(R(x)\), and constraints are incorporated as penalties through \(r(x)=R(x)-g(x)\). The forward noising process uses the usual DDPM schedule,
\[
q(x_t\mid x_{t-1})=\mathcal{N}(x_t;\sqrt{\alpha_t}x_{t-1},\beta_t I),
\qquad
x_t=\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\,\epsilon,
\]
while the reverse model is parameterized by a noise predictor \(\epsilon_\theta(x_t,t)\) and mean \(\mu_\theta(x_t,t)\) [2508.01509].

The distinctive feature is the Markov decision process view of reverse diffusion. The reverse chain is treated as an MDP with states \(x_t\), actions \(x_{t-1}\), pretrained transition kernel \(p_t^{pre}(x_{t-1}\mid x_t)\), and terminal reward \(r(x_0)\). Under KL regularization to the pretrained policy, the soft-optimal reverse policy takes the conservative form
\[
p_t^*(x_{t-1}\mid x_t)=
\frac{\exp(v_{t-1}(x_{t-1})/\alpha)\,p_t^{pre}(x_{t-1}\mid x_t)}
{\exp(v_t(x_t)/\alpha)},
\]
with marginal reweighting
\[
p_t^*(x_t)\propto \exp(v_t(x_t)/\alpha)\,p_t^{pre}(x_t).
\]
The soft Bellman recursion further yields
\[
\exp(v_t(x_t)/\alpha)=\mathbb{E}_{p^{pre}}\!\left[\exp(r(x_0)/\alpha)\mid x_t\right],
\qquad v_0(x_0)=r(x_0).
\]
These expressions formalize the idea that reward enters not only at the terminal sample, but as a consistent reweighting of intermediate diffusion states [2508.01509].

Training and inference are both derivative-free with respect to the reward. Fine-tuning is performed by reward-weighted maximum likelihood, using per-sample weights \(w^{(i,t)}=\exp(r(x_0^{(i,t)})/\alpha)\), while inference uses soft-value importance sampling. At reverse step \(t\), one draws \(M\) proposals from the fine-tuned reverse kernel, computes a posterior-mean estimate
\[
\widehat{x}_0(x_t)=\frac{x_t-\sqrt{1-\bar\alpha_t}\,\epsilon_\theta(x_t,t)}{\sqrt{\bar\alpha_t}},
\]
assigns weights \(w_{t-1}^{\langle m\rangle}=\exp(r(\widehat{x}_0)/\alpha)\), and resamples one candidate categorically [2508.01509].

The reported engineering tasks are 2D airfoil design and 3D ship hull design. For airfoils, \(x \in \mathbb{R}^{384}\), the dataset contains 38,000 augmented UIUC airfoils, and reward is normalized \(C_l/C_d\) predicted by an XGBoost surrogate with \(R^2=0.994\). For hulls, \(x \in \mathbb{R}^{44}\), reward is reduction in calm-water resistance aggregated over 8 speeds and 4 drafts, predicted by an XGBoost surrogate with \(R^2=0.985\). The method shifts the \(C_l/C_d\) distribution above the training range with \(>10\%\) overall improvement and individual designs reaching \(C_l/C_d \approx 275.56\), while iterative soft-value guidance achieves \(>25\%\) reduction in total resistance for ship hulls. Practical inference used an A100 40GB GPU, 1000 parallel trajectories, and typical \(M \le 10\)–11 due to memory limits [2508.01509].

The significance of this formulation is narrow but concrete. It is designed for settings where rewards come from non-differentiable surrogates or costly simulations such as XGBoost, ensemble models, GNN surrogates, or CFD, and where direct gradient-based guidance is unavailable or too expensive [2508.01509].

## 3. Score reweighting, soft values, and theoretical guidance

A second DRM formulation is developed as a unified theory of guided diffusion. Here the reward is first exponentiated, typically as
\[
r(x)=\exp(\beta R(x)),\qquad \beta>0,
\]
which induces the reward-reweighted distribution \(p_{x_0^{r\text{-}\mathsf{wt}}}(x_0)\propto r(x_0)p_{x_0}(x_0)\). Let
\[
s(x,t)=\nabla_x \log p_{x_{1-t}}(x),\qquad
s_R(x,t)=\nabla_x \log p_{x_{1-t}^{r\text{-}\mathsf{wt}}}(x),
\]
and define the guidance term as the score difference
\[
\Delta s(x,t)=s_R(x,t)-s(x,t).
\]
The guided reverse-time SDE is then
\[
\mathrm{d}y_t^w=
\Big(\tfrac12 y_t^w+\nabla_x\log p_{x_{1-t}}(y_t^w)+w\,\Delta s(y_t^w,t)\Big)\frac{\mathrm{d}t}{t}
+\frac{1}{\sqrt{t}}\,\mathrm{d}B_t,
\]
with a DDPM-style discrete analogue using the mixture \((1-w)s_n+w s_n^{r\text{-}\mathsf{wt}}\) [2512.04985].

Within this framework, the improved quantity is not stated loosely but identified exactly. For early stopping level \(\delta>0\), define the posterior expected reward
\[
r_t(y)=\mathbb{E}[r(x_0)\mid x_t=y].
\]
The central theorem gives
\[
\mathbb{E}[r_\delta(y_{1-\delta}^{w})]-\mathbb{E}[r_\delta(y_{1-\delta})]
=
\int_0^{1-\delta}\frac{w}{t}\,
\mathbb{E}\!\left[
r_{1-t}(y_t^w)\,
\big\|\nabla\log p_{x_{1-t}}(y_t^w)-\nabla\log p_{x_{1-t}^{r\text{-}\mathsf{wt}}}(y_t^w)\big\|_2^2
\right]\mathrm{d}t.
\]
This implies nonnegative reward improvement in expectation, with strict improvement when the score difference is nonzero with positive probability and \(w>0\) [2512.04985].

The same paper also gives a specific characterization of classifier-free guidance. Under the unified formulation, CFG decreases the expected reciprocal classifier probability,
\[
\mathbb{E}\!\left[\frac{1}{p(c\mid x)}\right],
\]
providing what the authors describe as the first theoretical characterization of the specific performance metric that CFG improves for general target distributions [2512.04985].

Training of the reward-reweighted score can be reduced to single-step denoising score matching rather than full trajectory rollouts. The proposed objective is
\[
\widehat\theta^{r\text{-}\mathsf{wt}}
\in
\arg\min_\theta
\mathbb{E}_{t,x_0,\epsilon}
\left[
r(x_0)\,\|\epsilon-\mathrm{NN}_\theta(x_t,t)\|_2^2
\right],
\qquad
x_t=\sqrt{1-t}\,x_0+\sqrt{t}\,\epsilon.
\]
A practical implication is that DRM, in this sense, can be trained without full diffusion trajectories and without reward gradients, while still inheriting a theorem that quantifies reward improvement and a stability bound for the discretized sampler [2512.04985].

This theoretical line is closely related to earlier work on reward-directed conditional diffusion. In the semi-supervised setting of unlabeled data plus noisy reward labels, a reward model \(\hat f\) is learned, pseudo-labels are assigned to unlabeled examples, and a conditional score model is trained to approximate \(P(x\mid \hat y=a)\). The resulting theory decomposes the suboptimality of generated samples into an off-policy bandit regret term, an on-support diffusion error, and an off-support extrapolation term, thereby linking reward gain to label efficiency, distribution shift, and subspace recovery [2307.07055].

## 4. Diffusion models as reward estimators and discriminators in reinforcement learning

In reinforcement learning and imitation learning, the term DRM often refers not to reward-directed sampling of a generator, but to a reward function constructed from a diffusion model. In DRAIL, a class-conditional DDPM is trained as a discriminator over state-action pairs \(x=(s,a)\), with class labels \(c^+\) for expert and \(c^-\) for agent. The diffusion losses under the two labels are turned into a classifier
\[
D_\phi(s,a)=
\frac{\exp(-\mathcal{L}_{\text{diff}}(s,a,c^+))}
{\exp(-\mathcal{L}_{\text{diff}}(s,a,c^+))+\exp(-\mathcal{L}_{\text{diff}}(s,a,c^-))}
=
\sigma\!\big(\mathcal{L}_{\text{diff}}(s,a,c^-)-\mathcal{L}_{\text{diff}}(s,a,c^+)\big),
\]
and the reward is the logit
\[
r_\phi(s,a)=\log D_\phi(s,a)-\log(1-D_\phi(s,a)).
\]
The method uses a cosine noise schedule with \(T=1000\), single-step denoising per label for efficient discrimination, and PPO for policy optimization [2405.16194].

Empirically, DRAIL reports strong results across navigation, manipulation, locomotion, and driving. For example, Maze success reaches \(96.83\%\pm0.49\), FetchPush \(95.78\%\pm0.80\), Walker return \(5381.15\pm115.25\), and CarRacing return \(746.35\pm31.28\). The paper attributes part of this behavior to smoother reward landscapes produced by the diffusion-based discriminator [2405.16194].

A different RL usage appears in DRESS, where a conditional diffusion model over \((s,a)\) generates an auxiliary reward \(r^{(G)}\) rather than classifying expert versus agent behavior. The total reward becomes
\[
r_t^{\text{total}} = r_t^{(E)} + \beta\,r_t^{(G)},
\]
and the diffusion generator is trained with a DDPM-style denoising loss while a guidance network \(Q_\nu(s,a,r^{(G)})\) is trained by TD learning. In the MECLatency wireless benchmark, DRESSed-SAC achieves about \(1.5\times\) faster convergence than SAC; across seven DRL benchmarks, DRESSed-SAC is best in 6/7 tasks [2503.07433]. This is a DRM in the reward-shaping sense: diffusion does not model the policy directly, but produces informative shaping rewards under sparse or degraded environmental feedback.

Preference-based offline RL yields yet another construction. DPR defines a diffusion-based discriminator on single-step state-action pairs,
\[
D_\phi(s,a;\epsilon,t)=\exp\!\big(-\|\epsilon-\epsilon_\phi((s,a),\epsilon,t)\|^2\big),
\]
and derives reward as
\[
r_{\mathrm{D}}(s,a)= -\frac{1}{T}\sum_{t=1}^T \log(1-D_\phi(s,a;\epsilon,t)).
\]
C-DPR adds a conditional normalization between positive and negative preference contexts and defines
\[
r_{\mathrm{C}}(s,a)= -\frac{1}{T}\sum_{t=1}^T \log(1-D_\phi^c(s,a;\epsilon,t)).
\]
With \(T=10\), a 4-layer MLP of hidden dimension 128, and Adam at \(3\times10^{-4}\), diffusion-based rewards outperform MLP- and Transformer-based preference rewards across multiple MuJoCo settings. For example, average normalized scores on MuJoCo with TD3BC rise from 45.87 with MLP rewards and 51.77 with Transformer rewards to 79.53 with DPR and 81.12 with C-DPR [2503.01143].

These RL instantiations share a practical claim: diffusion is used not only as a trajectory or action generator, but as the mechanism by which reward information is estimated, regularized, or shaped.

## 5. Step-wise and temporal DRMs for image and video alignment

A recent image-generation line treats a pre-trained diffusion backbone itself as the reward model. In this DRM, a truncated SD3.5-Medium diffusion transformer is fine-tuned with a lightweight reward head to score noisy intermediate latents \(x_t\) and prompts \(c\). Pairwise preference training uses the Bradley–Terry loss
\[
L_{\mathrm{DRM}}=-\log \sigma(s_{\mathrm{win}}-s_{\mathrm{lose}}),
\]
where the preferred and dispreferred images are first encoded to latents and then noised to timestep \(t\). The defining claim is that a model capable of high-fidelity generation already contains representations of perceptual attributes such as aesthetics, composition, and visual harmony, and can therefore serve as an evaluative backbone rather than only as a generator [2605.25661].

This step-wise evaluative capacity is used in two ways. In Step-wise GRPO, the denoising process is treated as an MDP and, at each reverse step, \(k\) candidate next latents are sampled and scored by the DRM. The step-wise advantage is
\[
A_t^{(j)}=
\frac{R(x_{t-1}^{(j)},c,t-1)-\mathrm{mean}_j\,R(x_{t-1}^{(j)},c,t-1)}
{\mathrm{std}_j\,R(x_{t-1}^{(j)},c,t-1)}.
\]
In Step-wise Sampling, the same branching is used at inference, with greedy selection
\[
x_{t-1}=\arg\max_j R(x_{t-1}^{(j)},c,t-1).
\]
On SD3.5-Medium, baseline scores of ImageReward \(=1.01\), PickScore \(=16.76\), and HPSv3 \(=8.95\) increase to \(1.17\), \(17.04\), and \(10.28\) under DRM with Step-GRPO. The method is also reported to converge approximately \(2.5\times\) faster in steps and \(3.5\times\) faster in GPU hours than standard GRPO [2605.25661].

Temporal reward modeling has also been proposed as a remedy for reward overoptimization in diffusion alignment. TDPO-R introduces a critic \(R_\phi(x_t,c)\) over noisy intermediate states, defining an advantage-like signal
\[
T_\phi(x_t,c)=R(x_0,c)-R_\phi(x_t,c),
\]
with critic regression objective
\[
L_{\mathrm{critic}}(\phi)=\mathbb{E}\big[(R_\phi(x_t,c)-R(x_0,c))^2\big].
\]
The policy gradient then uses
\[
\nabla_\theta J(\theta)=
\mathbb{E}\!\left[
\sum_{t=1}^T
w_t\,
\nabla_\theta \log p_\theta(x_{t-1}\mid x_t,c)\,
\big(R(x_0,c)-R_\phi(x_t,c)\big)
\right].
\]
The paper argues that terminal-only reward assignment ignores the temporal inductive bias of diffusion and exacerbates reward hacking, while critic active-neuron reset mitigates primacy bias in the critic [2402.08552].

In video-conditioned RL from expert demonstrations, Diffusion Reward defines the reward from the negative conditional entropy of a conditional video diffusion model:
\[
-H(p_\theta(\cdot\mid z_c))=
\mathbb{E}_{p_\theta(\cdot\mid z_c)}[\log p_\theta(z_k\mid z_c)].
\]
The practical reward is estimated through an ELBO-based Monte Carlo surrogate and combined with RND and sparse task reward as
\[
r_k^{\mathrm{diff}}=(1-\alpha)\,\bar r_k^{ce}+\alpha\,r_k^{rnd}+r_k^{spar}.
\]
Using VQ-GAN plus conditional VQ-Diffusion, this framework improves average success by about \(38\%\) on MetaWorld and \(35\%\) on Adroit over the best baseline, and also shows zero-shot transfer to unseen MetaWorld tasks [2312.14134].

Taken together, these works shift DRM from a purely terminal evaluator to a step-wise or temporal object. Reward is attached to the denoising trajectory itself rather than only to the final decoded sample.

## 6. Relations, limitations, and recurrent points of confusion

One common misconception is that all DRM methods require reward gradients with respect to design or image variables. This is not generally true. The engineering reward-directed framework is explicitly derivative-free with respect to the reward in both training and inference, relying on reward-weighted MLE and importance sampling rather than \(\nabla_x R(x)\) [2508.01509]. The unified score-reweighting framework similarly learns the reward-reweighted score by weighted denoising score matching at single noise levels, without full trajectory training [2512.04985]. By contrast, direct reward fine-tuning methods such as DRaFT backpropagate differentiable reward gradients through the entire or truncated sampling chain; this is a related but distinct design point rather than a universal property of DRM [2309.17400].

A second point of confusion concerns whether DRM is a generator or a reward estimator. In some papers it is the generator itself, reward-directed during training and sampling [2508.01509]; in others it is the reward-producing module, as in diffusion discriminators, diffusion preference models, or conditional video diffusion rewards [2405.16194], [2503.01143], [2312.14134]. This suggests that the phrase is overloaded across subfields, and precise interpretation depends on where reward enters the diffusion pipeline.

The main limitations are also recurrent across formulations. Surrogate fidelity is a central constraint in engineering optimization: a biased XGBoost or CFD surrogate can misguide the generator [2508.01509]. In reward-reweighted score learning, large guidance strength or poorly estimated reward-weighted scores can tilt the model away from the original data distribution, with the theory guaranteeing reward improvement but not calibration of unrelated quality metrics [2512.04985]. In RL and preference learning, noisy or sparse labels can distort the induced reward; DPR explicitly notes that dividing preference data into two diffusion-modeled distributions may still deviate from the actual reward [2503.01143]. In image alignment, large reward pressure can induce reward overoptimization, stylization, or loss of diversity, motivating temporal critics, active-neuron reset, or style-preserving regularizers [2402.08552], [2412.03268].

A plausible implication is that DRM should be viewed less as a single algorithm and more as a design space organized by three decisions: what object supplies reward information, at which diffusion times reward is injected, and whether guidance is gradient-based or gradient-free. The literature already spans terminal soft-value guidance for non-differentiable engineering objectives, score-difference guidance with explicit reward-improvement theorems, diffusion-based discriminators and preference models for RL, entropy-based rewards from conditional video diffusion, and step-wise evaluators built from the diffusion backbone itself [2508.01509], [2512.04985], [2605.25661].

Source: https://www.emergentmind.com/topics/diffusion-based-reward-model-drm