Are we really tilting? The mechanics of reward guidance in flow and diffusion models

Abstract: Reward guidance algorithms steer a learned generative process toward the reward-tilted measure at inference time. While empirically powerful, these methods are prone to reward hacking: the guided model over-optimizes the reward at the cost of fidelity to the learned distribution. Prior work has attributed this to the complexity of neural reward functions or implicit biases in diffusion training, but its fundamental origins remain poorly understood. We show that reward hacking arises from an approximation made in most practical implementations of reward-guided diffusion -- finite-particle plug-in estimation of the Doob h-function -- even in the simplest non-trivial settings of Gaussian and Gaussian mixture targets with quadratic rewards. In closed form, we isolate two distinct failure modes of the plug-in estimator: it leads to reward hacking within each mode and it cannot select high-reward modes. We propose a closed-form reward damping schedule that corrects the within-mode bias with no additional compute, and clarify the role of best-of-n sampling in compensating for the mode selection failure. Experiments on Gaussian mixture targets, a 2D checkerboard, and FLUX.1 text-to-image generation confirm that our theoretical insights carry over to practical settings.

• The paper shows that plug-in estimation leads to within-mode reward hacking by causing overshoot and excessive covariance contraction in reward-guided flows.
• The paper demonstrates that standard guidance estimators fail at long-range mode selection in multi-modal distributions, limiting effective reward tilting.
• The paper presents mitigation strategies—reward damping and best-of-n sampling—that correct biases and improve sample fidelity in generative tasks.

The Mechanics of Reward Guidance Failures in Flow and Diffusion Models

Introduction

Reward guidance provides a direct mechanism to steer generative models—especially flows and diffusions—toward reward-tilted distributions at generation time. Despite their empirical success, these guidance methods consistently suffer from reward hacking: over-optimization of the reward at the cost of sample fidelity, often manifesting as collapsed diversity and unnatural outputs. This paper, "Are we really tilting? The mechanics of reward guidance in flow and diffusion models" (2606.02884), gives a formal analysis of these pathologies, identifying their architectural origins and offering theoretically motivated mechanisms for mitigation. The authors systematically decompose reward-guided inference bias into two fundamental sources: within-mode reward hacking driven by plug-in estimation, and a failure of practical guidance estimators to enable long-range mode selection in multi-modal distributions.

Figure 2: Reward damping, introduced as a guidance schedule, robustly mitigates reward hacking in text-to-image generation with FLUX.1.

Theoretical Analysis of Reward-Guided Flows

Reward-Guided Sampling and the Doob $h$-Transform

Reward guidance in generative models is canonically formalized as sampling from the reward-tilted distribution $$\tilde{\rho}_1(x) \propto \rho_1(x) e^{\lambda r(x)}$$, where $\rho_1$ is the base generative model, $r(x)$ is a scalar reward, and $\lambda$ is an inverse temperature. In the ideal, the Doob $h$-transform modifies the drift of the generative SDE or ODE to target $\tilde{\rho}_1$ exactly, with a guidance term involving $$h_t(x) = \mathbb{E}[e^{\lambda r(X_1)} \mid X_t = x]$$. However, $h_t(x)$ is analytically intractable except in toy problems, necessitating an empirical estimation.

Finite-Particle Plug-in Estimation and Bias

The standard practice is to replace $h_t(x)$ by a $$\tilde{\rho}_1(x) \propto \rho_1(x) e^{\lambda r(x)}$$0-sample plug-in estimator: $$\tilde{\rho}_1(x) \propto \rho_1(x) e^{\lambda r(x)}$$1 where $$\tilde{\rho}_1(x) \propto \rho_1(x) e^{\lambda r(x)}$$2. The core theoretical contributions of the paper are as follows:

1. Within-mode reward hacking: In even the lowest-entropy scenario of an isotropic Gaussian base and quadratic reward, plug-in guidance yields terminal samples whose mean "overshoots" relative to the true reward-tilted mean, and contracts the covariance excessively fast—faster than the analytic solution. This bias does not shrink polynomially with increasing $$\tilde{\rho}_1(x) \propto \rho_1(x) e^{\lambda r(x)}$$3; rather, at least $$\tilde{\rho}_1(x) \propto \rho_1(x) e^{\lambda r(x)}$$4 is required to counteract the bias in high dimension.
2. Failure of mode selection in multi-modal settings: In Gaussian mixtures, the plug-in estimator only captures reward gradients in the mode in which a sample was initialized; it cannot upweight or traverse to distant, high-reward modes if they are not visited by the base sampler. As a result, the practical reward-guided flow cannot change mode probabilities, in direct contradiction to the analytic reward tilt.

   Figure 4: Best-of-$$\tilde{\rho}_1(x) \propto \rho_1(x) e^{\lambda r(x)}$$5 sampling sharply increases correct mode probability, better matching the analytic reward-tilted measure compared to single-seed ($$\tilde{\rho}_1(x) \propto \rho_1(x) e^{\lambda r(x)}$$6) guidance.

This is the analytic explanation of observed reward overoptimization ("hacking") and inability of guidance to perform global reweighting.

Mitigation: Reward Damping and Best-of-n Selection

The authors propose two robust practical solutions, each targeting a distinct bias.

Reward Damping for Within-Mode Correction

A closed-form, time-dependent reward scaling schedule,

$$\tilde{\rho}_1(x) \propto \rho_1(x) e^{\lambda r(x)}$$7

where $$\tilde{\rho}_1(x) \propto \rho_1(x) e^{\lambda r(x)}$$8 is the conditional posterior variance, counters the plug-in bias. Empirically, even a simple estimate suffices, and this approach is computationally free compared to increasing the number of inner samples. In all examined regimes—including high-dimensional text-to-image generation—reward damping dramatically reduces collapse and mean overshoot.

Figure 1: Masked intensity reward. Naive guidance removes content to maximize local brightness; reward damping preserves semantic fidelity while achieving high reward.

Figure 7: Blueness reward (artist). Naive guidance results in mode collapse and semantic loss; reward damping preserves both reward optimization and semantic structure.

Figure 3: Blueness reward (fox). Reward damping selectively modifies targeted attributes (background/sweater) while preserving untargeted semantics (fur).

Best-of-n for Mode Selection

To resolve mode selection failure, best-of-$$\tilde{\rho}_1(x) \propto \rho_1(x) e^{\lambda r(x)}$$9 sampling is employed: one draws $\rho_1$0 independent guided samples and selects the one with highest reward. This surrogate for sampling from the reward-tilted measure works because, unlike a single guided trajectory, it allows exploration of multiple base model modes, so that rare but high-reward modes are discovered. The authors analytically prove that, for an $\rho_1$1-mode mixture with an indicator reward, best-of-$\rho_1$2 can exponentially improve mode selection probability with $\rho_1$3, while plug-in estimation alone plateaus at the base model's mixture weights.

Figure 5: Checkerboard guidance. Best-of-$\rho_1$4 and reward damping eliminate the local bias and enable correct allocation of probability mass to high-reward modes.

Figure 6: FLUX text-to-image, VLM reward: only best-of-$\rho_1$5 achieves substantial reward increase and matches the intended reweighting of complex scene attributes.

Experimental Results

Experiments span low-dimensional analytic models (Gaussians, GMMs, checkerboard data) and large-scale text-to-image synthesis with FLUX.1. Strong, consistent empirical support is presented for the theoretical claims:

• Reward damping recovers sample diversity and corrects overoptimization across a battery of synthetic and generative modeling tasks, more efficiently than increased particle estimation.
• Best-of-n enables correct mode discovery and attribution, providing a practical mechanism to achieve reward tilting in multi-modal settings.
• Plug-in guidance without these corrections systematically leads to reward hacking and failure in mode selection.

Implications and Future Directions

This work exposes architectural limitations in widely deployed, plug-in-based reward-guided generation and grounds mitigation strategies in closed-form analysis. Practically, these observations constrain the use of vanilla reward guidance in high-stakes applications such as molecular and protein design, where spurious optimization or missed mode coverage could have significant implications. The theoretically justified reward damping schedule is trivial to implement and should see immediate adoption as a baseline across domains. Furthermore, the necessity of best-of-n for mode allocation suggests that optimizing single forward passes of reward-guided generative models is inherently limited, and motivates further research in algorithms capable of explicit mode traversal or allocation.

Theoretically, the results open questions about guidance in non-Gaussian, non-quadratic, and structured reward settings, and suggest deep connections with the theory of stochastic control and reinforcement learning. Extensions to RL-fine-tuning regimes, and investigation into the optimality and alternatives for best-of-n selection, are promising lines of future work.

Conclusion

This paper provides a comprehensive theoretical and empirical decomposition of reward guidance pathologies in flow and diffusion models, identifying finite-particle guidance as the source of both within-mode overoptimization and the incapacity for long-range mode selection. The proposed reward damping and best-of-$\rho_1$6 selection—not only supported by analytical results, but also demonstrated in large-scale experiments—stand out as essential corrections to ensure that guided generative models truly approach their target reward-tilted measures.