One-Step Denoising Reward Methods

• The paper presents CoCA, which redistributes the terminal reward across individual denoising steps using cosine similarity to address the temporal credit-assignment problem.
• It converts a sparse terminal reward into dense, step-level supervision via windowed smoothing and normalization, resulting in improved sample efficiency.
• Extensions like DenseGRPO and TP-GRPO apply one-step reward principles to flow-based models and elegant RL formulations, enhancing image generation quality and consistency.

One-Step Denoising Reward denotes a family of reward-assignment schemes for diffusion and flow-based generative models in which feedback is attached to the effect of an individual denoising transition, rather than being supplied only once at the end of the entire reverse trajectory. In reinforcement-learning formulations of denoising, the central motivation is the sparse-reward pathology of terminal-only supervision: the denoising chain is treated as a Markov decision process, yet the final image reward is propagated back to all intermediate actions without distinguishing their local contributions (Liao et al., 25 May 2025). In adjacent literature, the phrase is also used more loosely for reward-guided procedures applied to one-step generators or one-step denoisers, but the technically strict usage concerns temporal credit assignment across denoising steps (Deng et al., 28 Jan 2026).

1. Sparse terminal reward and the credit-assignment problem

RL fine-tuning methods for text-to-image diffusion models commonly model reverse denoising as a Markov decision process whose state at timestep $t$ is the current latent $x_t$, whose action is the denoising transition $x_t \to x_{t-1}$, and whose policy is the conditional reverse process $p_\theta(x_{t-1}\mid x_t,c)$ under prompt $c$ (Kordzanganeh et al., 2024). In this formulation, the canonical sparse reward assigns a single scalar only at the terminal step: $$R(s_t,a_t)= \begin{cases} r(x_0,c), & t=T-1\ 0, & t<T-1 \end{cases}$$ where $r(x_0,c)$ is the reward of the final generated image $x_0$ given prompt $c$ (Liao et al., 25 May 2025).

Under this setup, the REINFORCE-style gradient multiplies every denoising-step log-probability by the same terminal reward: $$\nabla_\theta\mathbb{E}_{\tau \sim \pi_\theta}\left[r(x_0,c)\right] = \mathbb{E}_{\tau \sim \pi_\theta} \left[ r(x_0,c)\sum_{t=1}^T\nabla_\theta\log p_\theta(x_{t-1}|x_t,c) \right].$$ The consequence is a temporal credit-assignment mismatch: all denoising steps are updated as if they had contributed equally to the final sample, although early steps often determine global structure while later steps refine details (Liao et al., 25 May 2025).

This problem is distinct from spatial feedback decomposition. PXPO, for example, replaces the single scalar terminal reward with a pixel-wise terminal heatmap $x_t$0, but the reward remains tied to the final image rather than to individual denoising timesteps. The method is therefore a richer terminal reward, not a temporal one-step denoising reward in the strict sense (Kordzanganeh et al., 2024).

2. Contribution-based step-level reward shaping

A concrete formulation of one-step denoising reward is Contribution-based Credit Assignment (CoCA), which redistributes the final trajectory reward across denoising steps according to each step’s estimated contribution to the final image (Liao et al., 25 May 2025). The core proxy is cosine similarity between an intermediate latent and the final latent: $x_t$1 Here, $x_t$2 is the intermediate latent after $x_t$3 reverse steps, and $x_t$4 is the final clean latent. The increment $x_t$5 is used as a proxy for how much that denoising step improved proximity to the final image (Liao et al., 25 May 2025).

Because per-step similarity is noisy, CoCA performs windowed smoothing. For window size $x_t$6,

$x_t$7

with $x_t$8. The appendix reports that window size $x_t$9 works best (Liao et al., 25 May 2025).

The final reward $x_t \to x_{t-1}$0 is then redistributed as dense step-level reward: $x_t \to x_{t-1}$1 Each timestep in a window receives reward in proportion to that window’s normalized contribution. The sparse terminal signal is thereby converted into dense denoising-time supervision without introducing an auxiliary neural network for step-level preference modeling (Liao et al., 25 May 2025).

The choice of cosine similarity in diffusion latent space is integral to the method. The stated reasons are that it avoids costly VAE decoding, that latents preserve structure and appearance better than semantic embeddings such as CLIP or DINO for this purpose, and that it measures progress in the actual denoising trajectory more faithfully. Reported ablations show cosine similarity outperforming $x_t \to x_{t-1}$2 distance and reward prediction from intermediate $x_t \to x_{t-1}$3 estimates (Liao et al., 25 May 2025).

3. Policy-gradient effect, invariance, and stabilization

Once the reward is redistributed, the policy-gradient weighting becomes timestep-dependent. After CoCA shaping, the gradient takes the form

$x_t \to x_{t-1}$4

The effective reward coefficient therefore varies across the denoising chain, so credit is aligned with estimated contribution rather than spread uniformly (Liao et al., 25 May 2025).

A central theoretical claim is that this redistribution preserves the optimal policy because it can be written as potential-based shaping: $x_t \to x_{t-1}$5 with $x_t \to x_{t-1}$6 and

$x_t \to x_{t-1}$7

Under this construction, dense step-level shaping improves learning efficiency without changing the intended optimum (Liao et al., 25 May 2025).

The same work introduces two-stage normalization for variance reduction. First, for a prompt $x_t \to x_{t-1}$8, trajectory rewards are normalized across sampled trajectories: $x_t \to x_{t-1}$9 Second, after redistribution, timestep rewards are normalized again: $p_\theta(x_{t-1}\mid x_t,c)$0 This preserves within-prompt ranking while stabilizing the shaped dense reward (Liao et al., 25 May 2025).

The reported training setup uses Stable Diffusion v1.5 with LoRA fine-tuning, $p_\theta(x_{t-1}\mid x_t,c)$1 denoising steps, window size $p_\theta(x_{t-1}\mid x_t,c)$2, guidance scale $p_\theta(x_{t-1}\mid x_t,c)$3, AdamW, and learning rate $p_\theta(x_{t-1}\mid x_t,c)$4. Evaluation is conducted on Aesthetic, PickScore, ImageReward, and HPSv2. The method is reported to achieve $p_\theta(x_{t-1}\mid x_t,c)$5 to $p_\theta(x_{t-1}\mid x_t,c)$6 times higher sample efficiency, better generalization across four human preference reward functions, improved generalization to unseen prompts, and no auxiliary step-level preference models (Liao et al., 25 May 2025).

4. Step-wise reward gains in flow-matching and GRPO-based alignment

The same underlying idea—replacing a trajectory-level reward with per-step denoising gains—has been extended to flow-matching alignment. DenseGRPO defines the dense reward of step $p_\theta(x_{t-1}\mid x_t,c)$7 as the reward gain between consecutive denoising states,

$p_\theta(x_{t-1}\mid x_t,c)$8

where each intermediate latent is mapped to a clean image via deterministic ODE denoising: $p_\theta(x_{t-1}\mid x_t,c)$9 Advantages are then normalized per timestep: $c$0 DenseGRPO couples this dense reward construction with a reward-aware timestep-specific stochasticity calibration in the SDE sampler, replacing a uniform noise schedule with $c$1 (Deng et al., 28 Jan 2026).

TP-GRPO defines an incremental one-step denoising reward by comparing two ODE-completed images that differ by exactly one denoising action: $c$2 To capture delayed effects, it introduces turning points—steps detected solely by sign changes in incremental rewards—and replaces the local increment at those steps with an aggregated long-term reward,

$c$3

The method reports consistent improvements over Flow-GRPO on compositional image generation, visual text rendering, and human preference alignment (Tong et al., 6 Feb 2026).

Method	Step reward definition	Distinctive feature
CoCA	$c$4	Cosine-similarity redistribution
DenseGRPO	$c$5	ODE-based latent reward gain
TP-GRPO	$c$6	Turning-point long-term reward

These methods share a common premise: the denoising trajectory should not be optimized with a uniform copy of the terminal reward. They differ in how they estimate the “pure” effect of a step. CoCA uses similarity progress toward the realized final latent; DenseGRPO uses reward gain between ODE-projected intermediate clean images; TP-GRPO augments local reward increments with explicit long-term credit assignment at sign-change turning points.

5. Step-level RL and exact denoising-time objectives

Step-level denoising reward has also been formalized as an RL objective rather than only as reward shaping. A step-level diffusion alignment formulation defines denoising as an MDP with deterministic dynamics and terminal reward, then optimizes a per-step objective based on the pretrained-policy advantage: $c$7 Here,

$c$8

and

$c$9

This reformulation yields a closed-form optimal policy,

$$R(s_t,a_t)= \begin{cases} r(x_0,c), & t=T-1\ 0, & t<T-1 \end{cases}$$0

which eliminates the circular dependence on the updated policy that affects marginal-level formulations (Zhang et al., 15 Apr 2026).

The same framework supports exact multi-objective denoising-time composition. If single-objective aligned reverse conditionals are $$R(s_t,a_t)= \begin{cases} r(x_0,c), & t=T-1\ 0, & t<T-1 \end{cases}$$1 and the reward is $$R(s_t,a_t)= \begin{cases} r(x_0,c), & t=T-1\ 0, & t<T-1 \end{cases}$$2, the optimal reverse conditional is

$$R(s_t,a_t)= \begin{cases} r(x_0,c), & t=T-1\ 0, & t<T-1 \end{cases}$$3

For Gaussian reverse kernels $$R(s_t,a_t)= \begin{cases} r(x_0,c), & t=T-1\ 0, & t<T-1 \end{cases}$$4, the fused distribution remains Gaussian: $$R(s_t,a_t)= \begin{cases} r(x_0,c), & t=T-1\ 0, & t<T-1 \end{cases}$$5 This exact denoising-time rule is presented as equivalent to the step-level RL fine-tuning objective and as introducing no approximation error (Zhang et al., 15 Apr 2026).

A plausible implication is that one-step denoising reward has evolved from a practical response to sparse rewards into a broader analytic framework in which denoising-time alignment can be stated directly at the level of single reverse transitions.

6. Terminological boundaries and adjacent uses of “one-step”

The expression “one-step” is not uniform across the literature. In strict denoising-reward work, it refers to reward attached to one denoising transition or to dense per-step redistribution across the trajectory. In several neighboring lines of work, however, “one-step” refers instead to generator architecture, inference-time optimization, or direct reconstruction.

ReNO is a reward-based inference method for one-step text-to-image models in which the generator is frozen and the initial Gaussian noise is optimized by gradient ascent on a weighted combination of ImageReward, PickScore, HPSv2, and CLIPScore. Its optimization variable is the seed noise, not the denoising-step reward, and the reported budget is $$R(s_t,a_t)= \begin{cases} r(x_0,c), & t=T-1\ 0, & t<T-1 \end{cases}$$6 gradient-ascent steps with roughly $$R(s_t,a_t)= \begin{cases} r(x_0,c), & t=T-1\ 0, & t<T-1 \end{cases}$$7–$$R(s_t,a_t)= \begin{cases} r(x_0,c), & t=T-1\ 0, & t<T-1 \end{cases}$$8 seconds per image depending on model and resolution (Eyring et al., 2024). DrPO is likewise a one-step preference-finetuning method for deterministic generators, but it uses target rewards only for ranking samples and constructs a detached feature-space dipole preference field plus reference drift; inference remains a single generator call (Jiang et al., 1 Jun 2026). DiffusionAD introduces a rapid one-step denoising paradigm for anomaly detection, achieving about $$R(s_t,a_t)= \begin{cases} r(x_0,c), & t=T-1\ 0, & t<T-1 \end{cases}$$9 times acceleration relative to iterative denoising in its ablation, but it is a reconstruction mechanism rather than a reward-assignment framework (Zhang et al., 2023).

A related but different response to sparse reward is selective optimization rather than dense reward design. AdaScope argues that RL should not optimize every denoising step: the early stage is described as structurally unstable and distant from the final reward signal, whereas in the later stage reward gains saturate and training tends to overfit local details. The method therefore adaptively chooses an interval $r(x_0,c)$0 and reports performance improvement of $r(x_0,c)$1 while cutting computational cost by $r(x_0,c)$2 (Yan et al., 15 May 2026).

These distinctions matter because the term can otherwise be misread in at least two ways. First, a one-step generator is not the same as a one-step denoising reward. Second, richer terminal feedback, such as PXPO’s pixel-wise heatmap, is not automatically a temporal step-level reward (Kordzanganeh et al., 2024).

7. Significance, misconceptions, and research trajectory

The main significance of one-step denoising reward lies in temporal credit assignment. Terminal-only reward treats the denoising chain as if all actions were equally responsible for the final image, whereas step-level reward attempts to isolate the marginal effect of individual reverse transitions. The literature converges on several recurring design principles: use intermediate states rather than only $r(x_0,c)$3, estimate step contribution through similarity or reward difference, stabilize those estimates through smoothing or normalization, and preserve the intended optimum either explicitly through potential-based shaping or implicitly through per-step trust-region structure (Liao et al., 25 May 2025, Zhang et al., 15 Apr 2026).

A common misconception is that dense reward shaping necessarily changes the task. CoCA explicitly claims otherwise by showing that its redistribution can be written as potential-based shaping with $r(x_0,c)$4, hence preserving the optimal policy (Liao et al., 25 May 2025). Another misconception is that all step-level schemes are equivalent. They are not. CoCA measures progress toward the realized final latent, DenseGRPO estimates reward gain from ODE-denoised intermediate clean images, and TP-GRPO supplements local gains with long-term aggregated rewards at sign-change turning points (Deng et al., 28 Jan 2026, Tong et al., 6 Feb 2026).

The broader research trajectory suggests two complementary directions. One direction increases the fidelity of per-step attribution, as in ODE-based reward gain estimation and turning-point modeling. The other direction reduces the need to optimize the entire trajectory, as in selective-scope methods or exact denoising-time composition. Together, these developments suggest that the original sparse terminal-reward formulation is increasingly viewed as an inadequate approximation of the denoising process rather than as the default RL interface for diffusion alignment.