NoiseTilt: Noise-Tilted Reverse Kernels for Diffusion Reward Alignment

Abstract: We introduce the Noise-Tilted Reverse Kernel (NTRK), a reward-guided diffusion sampler that injects reward gradients through the noise term, leaving the pretrained reverse kernel unchanged and requiring only a single sample per step. Reward-guided sampling at inference time has greatly expanded the versatility of pretrained diffusion models. Yet existing methods face a trade-off. Gradient-based guidance shifts the reverse mean, steering generation but pushing intermediate states outside the region that the model was trained on and degrading quality. Search-based methods preserve quality but gain no gradient signal. No prior method achieves both. NTRK resolves this by keeping the reverse mean fixed and biasing the noise term toward high reward. We introduce a whitening operator, the central mechanism behind NTRK, that makes the reward gradient safe to inject as noise without losing its guiding signal. Across various reward alignment tasks, NTRK outperforms recent state-of-the-art baselines without losing sample quality. Remarkably, on aesthetic generation, NTRK surpasses the reward of the best baseline at 500 NFEs using only 25 NFEs, a 20$\times$ reduction in compute.

• The paper introduces a noise-tilted reverse kernel for diffusion models that injects reward gradients via the noise term while preserving the pretrained reverse mean.
• It employs a high-dimensional whitening operator to convert structured reward gradients into statistically compatible Gaussian noise, ensuring in-distribution sample generation.
• The method achieves significant compute efficiency and improved trade-off between reward maximization and sample quality compared to traditional guidance approaches.

Noise-Tilted Reverse Kernels for Inference-Time Diffusion Reward Alignment

Introduction and Problem Formulation

This paper introduces the Noise-Tilted Reverse Kernel (NTRK), an inference-time reward alignment method for diffusion models that routes reward gradients through the noise term rather than modifying the reverse kernel mean. This resolves the chronic trade-off in reward-guided diffusion between gradient-based mean guidance—which is effective but leads to off-manifold sample drift and quality degradation—and search-based methods—which preserve sample quality but lack gradient signal and typically require expensive multi-sample selection per reverse step.

Standard reward alignment in diffusion aims to maximize a reward $r(\mathbf{x}_0)$ at terminal state under a regularized objective of the form

$$\max_q \;\mathbb{E}_{\mathbf{x}_0\sim q} \left[ r(\mathbf{x}_0) \right] - \beta D_\mathrm{KL}(q || p_0)$$

where $p_0$ is the base model distribution. The optimal reverse-time kernel then tilts the base Markov kernel by the reward’s continuation value. In existing inference-time guidance, two paradigms are dominant:

• Mean-shifted guidance (e.g., DPS): move the reverse kernel mean by a scaled reward gradient; this injects reward alignment but violates noise-compatibility, yielding out-of-distribution states and reduced visual fidelity.
• Search-based methods (e.g., SVDD, RBF): sample $K$ reverse kernel draws, select the one with highest predicted reward; this preserves the training noise manifold but is computationally prohibitive in high dimension.

NTRK is formulated to achieve both gradient-aligned guidance and strict preservation of the pretrained reverse mean with a single forward pass per reverse step.

Figure 1: Inference-time reward alignment results using NTRK across diverse reward-guided diffusion applications.

Noise-Tilted Reverse Kernel: Design and Mechanism

The core contribution is to keep the base reverse mean unchanged and inject the reward gradient through the noise variable:

$$\mathbf{x}_{t-1} = \boldsymbol{\mu}_\theta(\mathbf{x}_t, t) + \sigma_t \tilde{\boldsymbol{\epsilon}}_t$$

where

$$\tilde{\boldsymbol{\epsilon}}_t = \sqrt{\rho_t} \mathcal{W}(\nabla_{\mathbf{x}_t} r(\hat{\mathbf{x}}_{0|t})) + \sqrt{1-\rho_t} \boldsymbol{\epsilon}_t,\quad \boldsymbol{\epsilon}_t \sim \mathcal{N}(0, I)$$

and $\rho_t$ regulates the guidance strength. Here, $\mathcal{W}$ is a high-dimensional whitening operator projecting the structured reward gradient into the statistical regime of typical Gaussian noise, crucially preventing structured, non-Gaussian artifacts from entering the latent trajectory.

This construction allows a direct, geometry-preserving blend of reward-driven guidance and natural diffusion stochasticity per step, ensuring that all intermediate states remain in-distribution from the pretraining perspective of the diffusion kernel.

Whitening Operator Construction

Effective reward guidance through the noise term hinges on a robust operator $\mathcal{W}$ that converts an arbitrary gradient into a “typical” standard Gaussian noise sample. The paper’s whitening operator sequentially applies multidomain two-level order statistics (2OS) projections with confidence bounds, as well as constraints on tile-wise means and energies, and operates in both coordinate and orthogonal (Fourier, Hadamard) domains. The essential property is that standard Gaussian noise undergoes minimal modification (cosine similarity $>0.99999$), while structured, atypical gradients are projected onto the high-confidence typical set.

Figure 3: The two-level (2OS) confidence-interval hierarchy suppresses local and global structure and aligns statistics with the standard normal distribution.

Through detailed ablations, the operator is shown to be strictly superior to norm-only or blockwise-energy projections, simultaneously suppressing reward hacking and maintaining sample realism across large-scale image and video diffusion models.

Empirical Results and Ablation

Experiments are conducted on FLUX and Wan2.1 backbones, evaluating on (a) aesthetic image generation (Aesthetic Score), (b) text-aligned image generation (PickScore), (c) preference-aligned video generation (VideoReward), (d) quantity-aware generation, and (e) VLM-based rewards. The method is compared against state-of-the-art baselines (DPS, FreeDoM, SVDD, RBF, DAS, $$\max_q \;\mathbb{E}_{\mathbf{x}_0\sim q} \left[ r(\mathbf{x}_0) \right] - \beta D_\mathrm{KL}(q || p_0)$$0-Sampler) under matched compute (NFE) budgets—with quantitative and qualitative supervision from downstream task reward and out-of-distribution detectors.

NTRK achieves the following strong claims:

• In aesthetic generation, with only 25 NFEs, it exceeds the best baseline’s target reward at 500 NFEs—a 20× compute reduction.
• The trade-off between reward maximization and held-out distributional quality is improved; moderate $$\max_q \;\mathbb{E}_{\mathbf{x}_0\sim q} \left[ r(\mathbf{x}_0) \right] - \beta D_\mathrm{KL}(q || p_0)$$1 values yield maximal reward with no sacrifice of visual fidelity, and only extreme $$\max_q \;\mathbb{E}_{\mathbf{x}_0\sim q} \left[ r(\mathbf{x}_0) \right] - \beta D_\mathrm{KL}(q || p_0)$$2 induces quality dropoff.
• NTRK can be seamlessly composed atop fine-tuned (reward-aligned/backprop-) models, yielding further additive improvements on both target and auxiliary rewards.

Qualitative comparisons in both static and dynamic settings illustrate perceptible improvements in object placement, compositionality, and artifact suppression.

Figure 5: Qualitative comparison on image reward alignment for aesthetic image generation.

Figure 7: Qualitative comparison on preference-aligned video generation. NTRK achieves better text alignment and visual quality compared to previous methods.

Guidance Strength, Ablations, and Alternatives

Analysis of the guidance strength $$\max_q \;\mathbb{E}_{\mathbf{x}_0\sim q} \left[ r(\mathbf{x}_0) \right] - \beta D_\mathrm{KL}(q || p_0)$$3 demonstrates that even small $$\max_q \;\mathbb{E}_{\mathbf{x}_0\sim q} \left[ r(\mathbf{x}_0) \right] - \beta D_\mathrm{KL}(q || p_0)$$4 with a well-aligned whitened gradient can exert guidance as strong as very large, computationally infeasible hard-selection in search-based methods (equivalent effective $$\max_q \;\mathbb{E}_{\mathbf{x}_0\sim q} \left[ r(\mathbf{x}_0) \right] - \beta D_\mathrm{KL}(q || p_0)$$5). Ablations confirm that moderate $$\max_q \;\mathbb{E}_{\mathbf{x}_0\sim q} \left[ r(\mathbf{x}_0) \right] - \beta D_\mathrm{KL}(q || p_0)$$6 optimally balances reward maximization and non-collapse of sample diversity (see Figure 8).

Figure 4: Trade-off between reward alignment and held-out quality as $$\max_q \;\mathbb{E}_{\mathbf{x}_0\sim q} \left[ r(\mathbf{x}_0) \right] - \beta D_\mathrm{KL}(q || p_0)$$7 increases.

The full whitening pipeline’s necessity is established—the use of less principled norm projections or hard blockwise constraints were unable to achieve reward alignment without sample collapse or degradation.

Figure 6: Component-level ablation of $$\max_q \;\mathbb{E}_{\mathbf{x}_0\sim q} \left[ r(\mathbf{x}_0) \right] - \beta D_\mathrm{KL}(q || p_0)$$8: full components achieve maximum reward/quality trade-off.

Figure 8: Whitening operator ablation: only the full operator yields reward gains without harming sample quality.

Implications and Future Directions

Theoretical implications:

NTRK provides a single-draw, mean-preserving path for reward-driven steering in diffusion, resolving the fundamental incompatibility between gradient guidance and noise-compatibility. This positions reward guidance as a perturbation of the noise distribution, formally connected to conservative transport in stochastic control and applicable to a wider class of entropy-regularized objectives.

Practical impact:

NTRK’s main impact is to enable rapid, safe, reward-aligned generation with pretrained models—bypassing fine-tuning and achieving state-of-the-art compute/sample efficiency in alignment tasks. Its strict mean preservation ensures that none of the intermediate states traverse out-of-distribution regions (the source of reward hacking and sample collapse in prior art). The modularity of the whitening operator allows future adaptation to new reward models (e.g., compositional, VLM-based, RL-hybrid objectives) and custom noise priors.

Future work:

Directions for extension include automated selection/annealing of $$\max_q \;\mathbb{E}_{\mathbf{x}_0\sim q} \left[ r(\mathbf{x}_0) \right] - \beta D_\mathrm{KL}(q || p_0)$$9, theoretically optimal adaptive whitening under changing diffusion geometries, extension to multi-agent reward settings (for controllable content moderation), and application to reinforcement learning world models where reward-overfitting remains a central challenge. Furthermore, the notion of “typicality preserving” action in high-dimensional sampling is generally applicable in other generative paradigms (e.g., flows, autoregressive, rectified flows) and offers a useful design prior for safe and robust reward alignment.

Conclusion

Noise-Tilted Reverse Kernels represent a precise and expressive mechanism for inference-time reward alignment in diffusion generative models. By routing reward signal through a statistically compatible perturbation of the noise variable rather than the mean, NTRK achieves state-of-the-art reward maximization without out-of-distribution drift, severe computation, or degradation in sample quality. Its theoretical justification, practical effectiveness, and extensibility position it as a foundation for future advances in reward-aligned controllable generation.

---

Citation: "NoiseTilt: Noise-Tilted Reverse Kernels for Diffusion Reward Alignment" (2606.18066)