Reward-Guided Diffusion Sampling

• Reward-guided diffusion sampling is a method that tilts the generative prior using an external reward to favor high-value outputs, integrating principles from RL and controlled sampling.
• It employs both training-time fine-tuning (including policy-gradient RL and reward-weighted maximum likelihood) and inference-time approaches (such as soft value-based decoding and gradient guidance) to align generated samples with desired outcomes.
• Empirical analyses highlight improved sample efficiency, enhanced reward alignment, and theoretical guarantees, while also noting computational complexities in cases with negative-definite rewards.

Reward-Guided Diffusion Sampling is a family of methods that steer the sample generation process of diffusion models toward regions in data space that maximize an external reward function. These frameworks augment standard diffusion-based generative sampling with reward-driven modifications at training and/or inference time, and have become essential for inducing controllable, preference-aligned, or goal-oriented generation in imaging, language modeling, scientific design, and multimodal reasoning. Rigorous analyses reveal both the power and the theoretical limits of reward-guidance in diffusion processes.

1. Foundations: Reward-Guided Sampling as Distribution Tilting

Reward-guided diffusion sampling fundamentally targets the problem of sampling from a tilted distribution of the form

$p^*(x) \propto p(x)\exp\left(r(x)\right)$

where $p(x)$ is the generative prior (typically modeled via a pretrained diffusion process) and $r(x)$ is a scalar reward. This distributional tilt can encode aesthetic preference, alignment to human feedback, task performance, or any auxiliary valuation measure available at sample generation time.

The central challenge is that, given only score-oracle access to $p(x)$ (as in pretrained diffusion models), and arbitrary (possibly non-differentiable) $r(x)$, generating samples from $p^*(x)$ is algorithmically and computationally complex. Theoretical work fully characterizes this complexity for linear and quadratic reward functions, showing tractability for positive/low-rank tilts and NP-hardness for negative-definite cases (Moitra et al., 18 Feb 2026). In practice, various heuristic, RL-inspired, and algorithmically principled methods have been developed for reward-guided sampling (Uehara et al., 16 Jan 2025, Jiao et al., 4 Dec 2025, Zhang et al., 2023).

2. Reward Guidance in Training: Fine-Tuning and Soft RL

Several frameworks incorporate reward signals into the model’s training objective:

• Policy-Gradient RL Fine-Tuning: Reward-based fine-tuning treats the reverse diffusion denoising chain as a Markov Decision Process (MDP), and applies policy-gradient RL (e.g., DDPO [Black et al.], DPOK [Fan et al.] frameworks), optimizing

$$\nabla_\theta \mathbb{E}_{c, x_0}\left[\sum_{t=1}^T \nabla_\theta\log p_\theta(x_{t-1}|x_t, c)r(x_0, c)\right]$$

where each denoising action is rewarded based on the final output (Chae et al., 19 Feb 2025).

• Maximum Entropy RL and Reward-Weighted Likelihood: PDE-based training objectives can be reframed as entropy-regularized RL, leading to

$$J(\theta) = \mathbb{E}_{X_T}\left[\sum_{t=1}^T \mathbb{E}_{X_{t-1}\sim p_\theta}[r(X_{t-1})] + \tau H(p_\theta)\right]$$

yielding sample-efficient, reward-augmented training (Zhang et al., 2023). The corresponding reward “payoff” distribution exponentially weights reverse diffusion steps by exp(r/τ).

• Fine-Tuning via Reward-Weighted Maximum Likelihood: In generative optimization, reward-weighted MLE is employed, assigning sample weights $w(x_0) = \exp(r(x_0)/\alpha)$ during likelihood maximization, and provably produces samples with higher reward metrics after fine-tuning (Keramati et al., 2 Aug 2025). This is critical in scientific engineering, as in 3D ship hull and 2D airfoil design optimization.
• Flow-Based and Proximal Diffusion RL: Advanced methods extend reward-driven fine-tuning to flow matching (ODE-based) or backward-discretized diffusion (proximal operators), combined with policy gradients (GRPO, G²RPO) that improve sample alignment by denser, granular credit attribution and groupwise reward normalization (Zhou et al., 2 Oct 2025, Fang et al., 24 Nov 2025).

3. Test-Time Reward Guidance: Inference-Time Algorithms

A large class of algorithms applies reward-guidance exclusively at sampling (no fine-tuning required), making use of black-box reward evaluation and, in some cases, offline value functions:

• Soft Value-Based Decoding (SVDD): For non-differentiable or costly reward models, SVDD defines the “soft value” at noisy state $x_t$ as

$p(x)$0

and samples with importance weighting. This approach subsumes beam search, SMC, and value-based resampling, and unifies continuous and discrete state spaces (Li et al., 2024, Uehara et al., 16 Jan 2025).

• Gradient-Based (Derivative) Guidance: When r is differentiable, the score function of the diffusion reverse process is augmented by the reward’s gradient or the gradient of the soft value:

$p(x)$1

This forms the theoretical basis for classifier and reward guidance SDEs (Uehara et al., 16 Jan 2025, Jiao et al., 4 Dec 2025). A fully unified framework proves that reward-guided SDEs improve the expected value of the reward (and in the special case of CFG, provably minimize the expected reciprocal classifier probability) (Jiao et al., 4 Dec 2025).

• Particle and Search Algorithms: Methods such as SMC (Sequential Monte Carlo), beam search, or MCTS maintain a population of candidate particles or trajectories, perform reward-weighted resampling or expansion, and select or combine high-performing outputs. This includes both practical approaches for image, motion, and molecule generation and theoretical analyses connecting these to soft-optimal denoising policies (Uehara et al., 16 Jan 2025, Chu et al., 3 Mar 2025, Miles et al., 26 Feb 2026).
• Zigzag and Local Exploration Loops: Techniques such as Ctrl-Z sampling introduce controlled backtracking (retreat and noise injection) to escape reward plateaus, alternating between forward optimization and stochastic restarts upon stagnation as measured by reward improvement (Mao et al., 25 Jun 2025).
• Lookahead Sample Reward Guidance: The LiDAR method computes the expected future reward for the current particle in closed-form using only marginal samples from a pretrained model, bypassing costly neural backpropagation, and then applies a closed-form gradient update in the reverse SDE (Kim et al., 3 Feb 2026).

4. Exploration Strategies for Sample Efficiency

Reward-fine-tuning via online generation often leads to slow convergence due to sample redundancy. Explicit exploration strategies have been shown to dramatically accelerate reward maximization:

• Dynamic Classifier-Free Guidance (CFG) Scheduling: DiffExp dynamically adjusts the CFG scale, using a low guidance scale (high diversity) in early denoising steps and high guidance scale (high fidelity) in later steps. This allows initial broad exploration of the sample space, uncovering reward-rich regions (Chae et al., 19 Feb 2025).
• Random Prompt/Conditional Embedding Reweighting: In text-to-image models, randomly amplifying the embedding of one prompt phrase per sample injects strong variability into which aspects of the prompt are foregrounded, accelerating the discovery of reward-productive completions (Chae et al., 19 Feb 2025).
• Latent-Space Reward-Guided Resampling: For high-dimensional latent diffusion (e.g., video), running parallel candidate denoising trajectories, scoring at multiple intermediate steps with a latent reward model, and resampling/pruning by reward-weighted probabilities enables efficient search and controllability with minimal redundant computation (Zhao et al., 15 Mar 2026).
• Multi-Granularity Evaluation in Flow Models: G²RPO applies stochastic exploration at isolated timesteps, evaluates the impact of the injected noise at multiple denoising granularities, and aggregates advantages, increasing robustness to step selection and providing more faithful reward-to-noise credit assignment (Zhou et al., 2 Oct 2025).

5. Practical Implementations and Empirical Gains

Empirical studies consistently establish substantial sample efficiency, alignment, and reward improvement using reward-guided sampling:

• Sample Efficiency: Combining dynamic CFG and random prompt weighting (DiffExp) reduces reward queries by 20–25% and increases the final human-aligned reward score by up to 18% (PickScore) and 5.6% (aesthetics) in text-to-image fine-tuning (Chae et al., 19 Feb 2025).
• Generalization: Methods with explicit exploration outperform baselines on unseen prompts, consistently raising validation metrics such as ImageReward by 0.1–0.3 points (Chae et al., 19 Feb 2025). For discrete data, split Gibbs (SGDD) and CSMC samplers outperform standard SMC or value-based decoding by 30–40% in high-reward tasks in DNA/molecule generation (Chu et al., 3 Mar 2025, Phunyaphibarn et al., 10 Feb 2026).
• Efficiency in Video and Complex Data: LatSearch reduces runtime cost by up to 79% compared to best-of-N or brute-force decoding, while delivering consistent multi-metric improvement in video generation (Zhao et al., 15 Mar 2026).
• Language and Reasoning: Reward-weighted and reward-guided stitching in masked diffusion LLMs notably increases global coherence, non-autoregressive decoding, and final task accuracy in math/coding by 23–30 pp (Gwak et al., 31 Aug 2025, Miles et al., 26 Feb 2026).

6. Theoretical Guarantees, Limitations, and Open Problems

• Provable Improvement: Unified SDE theory establishes that injecting the difference between reward-reweighted and original scores (as a guidance term) strictly increases the expected reward in the generated distribution for all positive guidance weights. In the case of classifier-free guidance, it minimizes the expected reciprocal classifier probability (Jiao et al., 4 Dec 2025).
• Computational Tractability: For rewards that are linear or positive-definite low-rank quadratic, efficient reduction to base samplers is possible. For general high-rank or negative-definite rewards, sampling becomes NP-hard, limiting applicability to more restricted objectives (Moitra et al., 18 Feb 2026).
• Trade-offs: Strong reward-exploitation (low α, large CFG, high resampling temperature) leads to rapid convergence but possible mode collapse, reduced sample diversity, or reward overfitting, necessitating careful tuning and hybrid exploration-exploitation schedules (Chae et al., 19 Feb 2025, Jiao et al., 4 Dec 2025).
• Non-Differentiability and Black-Box Rewards: Derivative-free approaches (SVDD, SMC, CSMC) enable optimizing highly non-smooth, computationally expensive, or black-box objectives, expanding applicability in scientific domains (Li et al., 2024, Phunyaphibarn et al., 10 Feb 2026).
• Scalability and Cost: Methods that avoid per-step neural backprop (LiDAR, SVDD, latent reward guidance) achieve near-vanilla efficiency; those that require fine-tuning or dense value estimation can be computationally intensive (Kim et al., 3 Feb 2026, Li et al., 2024, Keramati et al., 2 Aug 2025).

7. Extensions and Future Directions

The field is rapidly progressing, with several avenues for further research:

• Unifying discrete and continuous methods: Frameworks such as split Gibbs, CSMC, and SVDD operate in both settings, and continuous–discrete hybrids are under exploration (Chu et al., 3 Mar 2025, Li et al., 2024).
• Improved exploration schedules: Adaptive, task-conditioned exploration vs. exploitation during sampling remains under-studied.
• Joint optimization of sampler and reward model: Current methods presuppose a fixed reward; joint training and credit assignment may further leverage sample and reward information (Zhao et al., 15 Mar 2026).
• Search and step-level recombination: Stitching and resampling of stepwise candidates enables high accuracy and reduced latency in complex multi-step generative reasoning tasks (Miles et al., 26 Feb 2026).

Reward-guided diffusion sampling embodies a broad and theoretically grounded approach to aligning generative models with arbitrary downstream objectives, encompassing RL-like fine-tuning, value-based inference-time guidance, exploration-enhancing augmentations, and rigorous complexity bounds. Properly tuned and theoretically informed, these methods are now core tools for controllable, preference-aligned, and task-driven data generation across scientific and engineering disciplines.

Key references: (Chae et al., 19 Feb 2025, Jiao et al., 4 Dec 2025, Zhou et al., 2 Oct 2025, Moitra et al., 18 Feb 2026, Li et al., 2024, Uehara et al., 16 Jan 2025, Chu et al., 3 Mar 2025, Zhao et al., 15 Mar 2026, Gwak et al., 31 Aug 2025, Kim et al., 3 Feb 2026, Fang et al., 24 Nov 2025, Zhang et al., 2023, Phunyaphibarn et al., 10 Feb 2026).