Reward Score Matching (RSM) in Generative Models

• Reward Score Matching is a unifying framework that aligns pretrained diffusion and flow models to an exponential reward-induced target distribution.
• It decomposes the score into a reference part and a value-gradient guidance component, balancing bias and variance via different estimators.
• The method streamlines diverse fine-tuning approaches by optimizing reward guidance across timesteps and incorporating trust-region stabilization.

Reward Score Matching (RSM) is a unifying formulation for reward-based fine-tuning of pretrained diffusion and flow generative models in which alignment is written as score matching toward a reward-guided target distribution rather than as a heterogeneous collection of methods derived from Soft RL, GFlowNets, stochastic optimal control, direct regression, or PPO/GRPO-style policy optimization. In this formulation, the aligned model should match the score of an exponential tilt of the reference marginal, and most method variation reduces to two core choices: how the reward-induced value guidance is estimated, and how strongly that guidance is applied across timesteps (Lee et al., 19 Apr 2026). A closely related discrete counterpart appears in Target Concrete Score Matching, where reward fine-tuning is implemented by matching the concrete score of a reward-tilted target distribution in clean discrete space (Zhang et al., 23 Apr 2025).

1. Reward-guided target distributions and score decomposition

RSM starts from reward alignment of a pretrained model $p^{\mathrm{ref}}$. The terminal reward is $r(x_0)$, and the entropy-regularized fine-tuning objective is

$$\max_\theta \mathbb{E}_{x_0\sim p_0^\theta}[r(x_0)] - \alpha \sum_i \mathcal{D}_{\mathrm{KL}}\!\big( p^\theta(\cdot\mid x_{t_i}) \;\|\; p^{\mathrm{ref}}(\cdot\mid x_{t_i}) \big).$$

For affine conditional flows, a trajectory $x_t$ evolves under a flow ODE

$dx_t = v_t(x_t)\,dt,$

and a reverse-time flow SDE preserving the same marginals is

$$dx_t = \Big(v_t(x_t)-\frac{\tilde\sigma_t^2}{2}\nabla \log p_t(x_t)\Big)\,dt + \tilde\sigma_t\,dw_t.$$

The score is

$s_t \equiv s(x_t,t) := \nabla_{x_t}\log p_t(x_t).$

The central object is the soft value function $V_t(x_t)$. Under the reward-regularized objective, the optimal marginal satisfies

$$p_t^\star(x_t) = \frac{p_t^{\mathrm{ref}}(x_t)}{Z} \exp\!\Big(\frac{V_t(x_t)}{\alpha}\Big),$$

with $Z$ independent of $r(x_0)$0. Taking $r(x_0)$1 yields the target score

$r(x_0)$2

RSM denotes the reward-induced correction by

$r(x_0)$3

so that

$r(x_0)$4

This decomposition is the defining statement of RSM. It makes the aligned score equal to the reference score plus value-gradient guidance. The same target also emerges from a path-space KL-regularized control problem and from stochastic optimal control, where the optimal controlled marginal again takes the exponential-tilt form and the optimal control is proportional to $r(x_0)$5. The unification is therefore not merely terminological: different derivations recover the same score correction (Lee et al., 19 Apr 2026).

2. Common RSM loss and its canonical gradient form

RSM becomes operational once the reward-guided target score is turned into a regression objective. For discretized affine flow SDEs, the reverse transition density has the form

$r(x_0)$6

Because the transition mean depends affinely on the score, reward alignment can be written as score regression toward $r(x_0)$7.

The common RSM loss is

$r(x_0)$8

Here $r(x_0)$9 is the current model score, $$\max_\theta \mathbb{E}_{x_0\sim p_0^\theta}[r(x_0)] - \alpha \sum_i \mathcal{D}_{\mathrm{KL}}\!\big( p^\theta(\cdot\mid x_{t_i}) \;\|\; p^{\mathrm{ref}}(\cdot\mid x_{t_i}) \big).$$0 is an old-policy score, $$\max_\theta \mathbb{E}_{x_0\sim p_0^\theta}[r(x_0)] - \alpha \sum_i \mathcal{D}_{\mathrm{KL}}\!\big( p^\theta(\cdot\mid x_{t_i}) \;\|\; p^{\mathrm{ref}}(\cdot\mid x_{t_i}) \big).$$1 is the primary timestep weight, and $$\max_\theta \mathbb{E}_{x_0\sim p_0^\theta}[r(x_0)] - \alpha \sum_i \mathcal{D}_{\mathrm{KL}}\!\big( p^\theta(\cdot\mid x_{t_i}) \;\|\; p^{\mathrm{ref}}(\cdot\mid x_{t_i}) \big).$$2 is a trust-region or old-policy anchor coefficient. The target guidance used in practice is typically

$$\max_\theta \mathbb{E}_{x_0\sim p_0^\theta}[r(x_0)] - \alpha \sum_i \mathcal{D}_{\mathrm{KL}}\!\big( p^\theta(\cdot\mid x_{t_i}) \;\|\; p^{\mathrm{ref}}(\cdot\mid x_{t_i}) \big).$$3

where $$\max_\theta \mathbb{E}_{x_0\sim p_0^\theta}[r(x_0)] - \alpha \sum_i \mathcal{D}_{\mathrm{KL}}\!\big( p^\theta(\cdot\mid x_{t_i}) \;\|\; p^{\mathrm{ref}}(\cdot\mid x_{t_i}) \big).$$4 approximates $$\max_\theta \mathbb{E}_{x_0\sim p_0^\theta}[r(x_0)] - \alpha \sum_i \mathcal{D}_{\mathrm{KL}}\!\big( p^\theta(\cdot\mid x_{t_i}) \;\|\; p^{\mathrm{ref}}(\cdot\mid x_{t_i}) \big).$$5.

Differentiating the loss yields the canonical gradient

$$\max_\theta \mathbb{E}_{x_0\sim p_0^\theta}[r(x_0)] - \alpha \sum_i \mathcal{D}_{\mathrm{KL}}\!\big( p^\theta(\cdot\mid x_{t_i}) \;\|\; p^{\mathrm{ref}}(\cdot\mid x_{t_i}) \big).$$6

with

$$\max_\theta \mathbb{E}_{x_0\sim p_0^\theta}[r(x_0)] - \alpha \sum_i \mathcal{D}_{\mathrm{KL}}\!\big( p^\theta(\cdot\mid x_{t_i}) \;\|\; p^{\mathrm{ref}}(\cdot\mid x_{t_i}) \big).$$7

This decomposition isolates three components. The term $$\max_\theta \mathbb{E}_{x_0\sim p_0^\theta}[r(x_0)] - \alpha \sum_i \mathcal{D}_{\mathrm{KL}}\!\big( p^\theta(\cdot\mid x_{t_i}) \;\|\; p^{\mathrm{ref}}(\cdot\mid x_{t_i}) \big).$$8 is the reward-seeking update. The term $$\max_\theta \mathbb{E}_{x_0\sim p_0^\theta}[r(x_0)] - \alpha \sum_i \mathcal{D}_{\mathrm{KL}}\!\big( p^\theta(\cdot\mid x_{t_i}) \;\|\; p^{\mathrm{ref}}(\cdot\mid x_{t_i}) \big).$$9 is the KL regularizer that keeps the fine-tuned model close to the pretrained reference. The term $x_t$0 is a trust-region stabilizer. Under RSM, apparently different algorithms differ less in their abstract justification than in how they instantiate these three ingredients (Lee et al., 19 Apr 2026).

3. Estimating value guidance

The main practical difficulty is estimating

$x_t$1

RSM organizes existing estimators into current-state, lookahead, first-order, and zeroth-order families.

For differentiable rewards, the current-state first-order estimator is

$x_t$2

where $x_t$3 is a Tweedie or posterior-mean estimate of the clean sample. This avoids rollout, but the bias can be substantial, especially at low SNR.

For lookahead depth $x_t$4 and branching width $x_t$5, the first-order lookahead estimator is

$x_t$6

and the zeroth-order lookahead estimator is

$x_t$7

The zeroth-order estimator is the relevant choice for black-box rewards. For full rollout $x_t$8, RSM shows that the zeroth-order estimator is unbiased for the soft value gradient: $x_t$9 This equality is obtained via Stein’s identity, which relates reward-weighted noise to a gradient of expected reward.

The framework also includes residual correction. Some first-order methods use

$dx_t = v_t(x_t)\,dt,$0

so that, under exact residual parameterization, current-state and one-step first-order estimators become unbiased. Empirically, however, the residual term $dx_t = v_t(x_t)\,dt,$1 is reported as negligible, and the paper consequently treats many such architectural add-ons as auxiliary rather than fundamental (Lee et al., 19 Apr 2026).

4. Optimization strength across timesteps, unified method families, and redesigned variants

RSM argues that estimator choice is only half of the problem. The other half is how much effective optimization mass reaches each timestep. This is summarized by the Normalized Influence Metric

$dx_t = v_t(x_t)\,dt,$2

The factor $dx_t = v_t(x_t)\,dt,$3 depends on parameterization; examples given in the paper are

$dx_t = v_t(x_t)\,dt,$4

and

$dx_t = v_t(x_t)\,dt,$5

The update can then be rewritten in terms of a common reward-gradient signal: $dx_t = v_t(x_t)\,dt,$6 Under this view, methods that appear philosophically distinct may differ mainly in $dx_t = v_t(x_t)\,dt,$7, branching budget, trust-region realization, or clipping behavior.

The unification covers first-order methods such as VGG-Flow, SQDF, and Residual $dx_t = v_t(x_t)\,dt,$8-DB, as well as zeroth-order methods such as KL-regularized REINFORCE, PPO/GRPO/PCPO-base, PCPO-reweight, Branch-GRPO, TempFlow-GRPO, and GRPO-Guard. For example, KL-regularized REINFORCE uses

$dx_t = v_t(x_t)\,dt,$9

with

$$dx_t = \Big(v_t(x_t)-\frac{\tilde\sigma_t^2}{2}\nabla \log p_t(x_t)\Big)\,dt + \tilde\sigma_t\,dw_t.$$0

while PPO / GRPO / PCPO-base reduce to the same RSM target but with an old-policy anchor

$$dx_t = \Big(v_t(x_t)-\frac{\tilde\sigma_t^2}{2}\nabla \log p_t(x_t)\Big)\,dt + \tilde\sigma_t\,dw_t.$$1

RSM also provides a bias-variance-compute account of these design choices. First-order estimators are low-variance but biased when they rely on local Tweedie guidance. Full-rollout zeroth-order estimators are unbiased but sample-hungry, with conditional variance scaling as

$$dx_t = \Big(v_t(x_t)-\frac{\tilde\sigma_t^2}{2}\nabla \log p_t(x_t)\Big)\,dt + \tilde\sigma_t\,dw_t.$$2

Reward centering is treated as a control variate that reduces variance without changing the target guidance. This suggests that branching depth, reward centering, and timestep allocation are not secondary implementation details; they are part of the effective optimization geometry.

The paper uses this analysis to simplify and redesign several methods. For zeroth-order flow, it replaces timestep-unfair clipping with

$$dx_t = \Big(v_t(x_t)-\frac{\tilde\sigma_t^2}{2}\nabla \log p_t(x_t)\Big)\,dt + \tilde\sigma_t\,dw_t.$$3

reallocates branching budget to intrinsically noisier steps, and deactivates hopelessly noisy steps. In the reported flow experiment, the redesigned zeroth-order method reaches GenEval $$dx_t = \Big(v_t(x_t)-\frac{\tilde\sigma_t^2}{2}\nabla \log p_t(x_t)\Big)\,dt + \tilde\sigma_t\,dw_t.$$4 with a $$dx_t = \Big(v_t(x_t)-\frac{\tilde\sigma_t^2}{2}\nabla \log p_t(x_t)\Big)\,dt + \tilde\sigma_t\,dw_t.$$5 wall-clock speedup over TempFlow-GRPO. For first-order methods, replacing local Tweedie gradients with terminal-image gradients substantially accelerates reward improvement in settings where semantic fidelity is important (Lee et al., 19 Apr 2026).

5. Related formulations and neighboring methods

RSM is the explicit name of the unifying continuous-time framework, but several adjacent methods instantiate narrower or alternative versions of reward-guided score or distribution correction.

Method	Relation to RSM	Key distinction
TCSM (Zhang et al., 23 Apr 2025)	Discrete analogue of reward-modulated score matching	Matches the concrete score of a reward-tilted target in clean discrete space
RewardSDS (Chachy et al., 12 Mar 2025)	Reward-weighted score-distillation estimator	Reweights Monte Carlo SDS/VSD terms rather than learning a reward-conditioned score field
ScoRe-Flow (Qiu et al., 13 Apr 2026)	Score-guided RL fine-tuning of flow-matching policies	Uses an analytically induced FM score during PPO, not reward-weighted score matching
DRL (Beltran-Velez et al., 17 Jun 2026)	Density-ratio reward extraction from data-model discrepancy	Uses discriminator logits as rewards in KL-regularized RL, not direct score correction
RSPO (Yu et al., 11 May 2026)	Relative reward score/log-ratio calibration	Matches a centered current-vs-reference ELBO score surrogate to a reward-implied target
RTDMD (Huang et al., 25 May 2026)	Reward-tilted distribution matching for few-step generators	Decomposes reverse KL to a reward-tilted teacher into DMD-style distribution matching plus reward maximization

The closest explicit discrete counterpart is TCSM. Its reward-modulated target distribution is

$$dx_t = \Big(v_t(x_t)-\frac{\tilde\sigma_t^2}{2}\nabla \log p_t(x_t)\Big)\,dt + \tilde\sigma_t\,dw_t.$$6

and the reward-modified concrete score satisfies

$$dx_t = \Big(v_t(x_t)-\frac{\tilde\sigma_t^2}{2}\nabla \log p_t(x_t)\Big)\,dt + \tilde\sigma_t\,dw_t.$$7

Because the partition function cancels in the ratio, reward fine-tuning becomes exact concrete-score matching on a reward-tilted target, together with an RLHF-like reverse-KL interpretation (Zhang et al., 23 Apr 2025).

Several neighboring methods are intentionally not literal instances of RSM. RewardSDS keeps the standard SDS or VSD score residual and changes only the weighting of sampled noise realizations; it is a reward-weighted Monte Carlo estimator rather than a new global score model (Chachy et al., 12 Mar 2025). ScoRe-Flow computes a score analytically from the velocity field of a flow-matching policy,

$$dx_t = \Big(v_t(x_t)-\frac{\tilde\sigma_t^2}{2}\nabla \log p_t(x_t)\Big)\,dt + \tilde\sigma_t\,dw_t.$$8

and uses that score to modulate drift in a controlled SDE optimized with PPO, but reward enters only through policy gradients, not through a reward-weighted score target (Qiu et al., 13 Apr 2026).

DRL is adjacent in a different way. It learns a discriminator reward

$$dx_t = \Big(v_t(x_t)-\frac{\tilde\sigma_t^2}{2}\nabla \log p_t(x_t)\Big)\,dt + \tilde\sigma_t\,dw_t.$$9

which estimates the log density ratio between data and base model, and then optimizes a KL-regularized RL objective on trajectories. The paper explicitly positions this as an RL-based correction rather than a reward score matching objective, even though the density-ratio interpretation suggests a latent score-correction view (Beltran-Velez et al., 17 Jun 2026). RTDMD likewise defines a reward-tilted teacher

$s_t \equiv s(x_t,t) := \nabla_{x_t}\log p_t(x_t).$0

but minimizes reverse KL to that teacher by decomposing it into DMD-style distribution matching plus reward maximization rather than by directly fitting $s_t \equiv s(x_t,t) := \nabla_{x_t}\log p_t(x_t).$1 (Huang et al., 25 May 2026). RSPO moves even further from classical diffusion score matching: it calibrates a centered current-vs-reference ELBO score surrogate to a reward-implied target relative log-ratio, which makes it better described as relative reward score matching than as ordinary score matching (Yu et al., 11 May 2026).

6. Limitations, caveats, and broader significance

RSM clarifies design structure, but it does not eliminate the core problems of alignment. The unifying paper explicitly notes several limitations: offline preference methods such as Diffusion-DPO are outside the current scope; reward hacking is not solved; first-order residual critics remain weak; trust-region design via clipping or $s_t \equiv s(x_t,t) := \nabla_{x_t}\log p_t(x_t).$2-style anchoring remains heuristic; and joint optimization of sampler design with the training objective remains open (Lee et al., 19 Apr 2026).

A broader caveat concerns the reward itself. Research on reward models shows that reward-like signals derived from language-model probabilities or reward heads can inherit value biases from pretraining. In particular, the paper on inherited value bias derives an implicit reward

$s_t \equiv s(x_t,t) := \nabla_{x_t}\log p_t(x_t).$3

and proposes a mixture-weighted log-ratio

$s_t \equiv s(x_t,t) := \nabla_{x_t}\log p_t(x_t).$4

showing that likelihood-derived reward signals are usable but not value-neutral (Christian et al., 28 Jan 2026). This suggests that any RSM pipeline that estimates reward or score corrections from model outputs inherits the calibration and bias properties of those reward surrogates.

A further implication is that reward-score ideas need not remain at the sample level. Expected Reward Prediction studies

$s_t \equiv s(x_t,t) := \nabla_{x_t}\log p_t(x_t).$5

and uses regression on prompt embeddings to predict model-level expected reward for routing. This is not score matching in the generative-density sense, but it shows that reward-based matching can be lifted from individual samples to prompt-model choices, extending the RSM intuition from generative fine-tuning to inference-time selection (Hasanaliyev et al., 3 Mar 2026).

Taken together, these developments establish RSM as both a specific framework and a broader design language. In the strict sense introduced for flow and diffusion models, RSM means matching the model score to the score of a reward-guided target, with algorithmic differences localized to guidance estimation and timestep weighting. In the broader 2025–2026 literature, the same logic reappears as concrete-score matching in discrete diffusion, reward-weighted score-distillation estimators, relative score/log-ratio calibration for diffusion LLMs, and reward-tilted distribution matching for few-step generators. The unifying insight is constant across these settings: reward-based fine-tuning can often be understood most cleanly by identifying the reward-tilted target distribution and then asking which score, score surrogate, or score-like quantity is actually being matched.