Colored Noise Diffusion Sampling

Abstract: Diffusion models achieve state-of-the-art image synthesis, with their generative trajectories fundamentally exhibiting a spectral bias, resolving low-frequency global structures early and high-frequency fine details later. Conventional stochastic differential equation (SDE) solvers fail to account for this dynamic, naively injecting uniform white noise throughout the entire process and misusing the finite energy budget. In this work, we establish a mathematical framework that reconsiders SDE inference as a targeted, frequency-decoupled energy transfer. Leveraging this framework, we introduce Colored Noise Sampling (CNS), a novel, training-free stochastic solver. Rather than injecting uniform white noise, CNS utilizes a dynamic, timestep- and frequency-dependent schedule that more efficiently allocates injected energy toward structurally unresolved frequency bands. By actively exploiting the model's inherent spectral bias, CNS systematically steers the generated distribution toward the true data manifold. Extensive experiments demonstrate that CNS significantly outperforms standard ODE and SDE baselines as a strictly plug-and-play, inference-time sampler substitution across diverse architectures (SiT, JiT, FLUX). Compared to standard sampling on ImageNet-256, CNS achieves substantial unguided FID reductions, improving from 8.26 to 6.27 on SiT-XL/2, 32.39 to 26.69 on JiT-B/16, and 11.88 to 8.31 on JiT-H/16, while yielding consistent relative FID improvements with Classifier-Free Guidance. Project page is available at https://hadardavidson.github.io/CNS/.

• The paper presents a new inference mechanism called Colored Noise Sampling (CNS) that redistributes stochastic energy based on frequency content.
• It reformulates SDE noise injection as a variance-preserving, targeted energy transfer, achieving significant FID improvements and compositional gains.
• Empirical results on ImageNet-256 and text-to-image tasks demonstrate CNS's robustness and enhanced sample fidelity without requiring retraining.

Colored Noise Diffusion Sampling: Frequency-Aware Inference in Generative Models

Spectral Bias and Inefficiency in Standard Diffusion Sampling

Diffusion models define current benchmarks in photorealistic image synthesis due to their ability to model the generative trajectory from noise to data with high fidelity. During inference, these models exhibit a pronounced spectral bias: low-frequency global image structures are resolved early, while high-frequency details emerge late. However, standard Stochastic Differential Equation (SDE) solvers inject uniform white noise across all frequencies and steps, ignoring this temporal evolution of the spectral content. Such frequency-agnostic noise allocation misuses the strictly finite stochastic energy budget provided by the diffusion schedule and leads to suboptimal reconstruction—particularly underrepresenting high-frequency features.

Prior work attempting to leverage spectral bias has focused either on modifying the training noise distribution or introducing ad-hoc inference-time adjustments such as frequency-domain operations or activation reweighting. However, these approaches fail to address the fundamental inefficiency in the sampling mechanism and often require costly retraining or are constrained by black-box solvers.

Mathematical Framework: Targeted Energy Transfer and Frequency Decoupling

The paper rigorously establishes that diffusion sampling trajectories are geometrically akin to non-orthogonal rotations in latent space, mapping noise directly into structured image features. Empirical analysis confirms that not only is the initial noise signal preserved and transferred during inference, but additional stochastic increments injected via SDE solvers are mapped into corresponding frequency bands in the final output.

Key theoretical contributions include:

• Reformulation of SDE noise injection as a strictly variance-preserving, targeted energy transfer across frequency bands, relaxing the classical Langevin requirement for uniformity without inducing out-of-distribution drift.
• Introduction of a progression index $y(f, t)$ quantifying the resolved structural content at band $f$ and timestep $t$, guiding optimal noise allocation.

Parseval's theorem and Fourier-domain projections are leveraged to track accurate energy transfer, and spectral gap analysis empowers identification of energy deficits across frequencies in generated distributions versus true data.

Colored Noise Sampling (CNS): State- and Frequency-Dependent Noise Routing

Building upon the spectral bias framework, Colored Noise Sampling (CNS) is proposed as a novel, training-free stochastic solver. CNS dynamically computes a frequency- and timestep-dependent scaling matrix $B_f(t)$ for noise injection, actively reallocating variance to bands with maximal unresolved structure (low $y(f, t)$). The global variance-conservation constraint is strictly enforced, ensuring that

$$B_f(t) = \frac{\sqrt{1 - y(f, t)}}{\text{RMS}_{f'}\sqrt{1 - y(f', t)}}$$

This schedule begins with uniform noise at initialization, progressively attenuating injection into resolved bands and routing stochastic energy into lagging high-frequency regions, increasing the fidelity and diversity of generated outputs. CNS is implemented as a plug-and-play sampler usable at inference-time with any compatible diffusion architecture and does not require retraining.

Numerical Results and Robustness

Extensive experiments validate CNS across multiple architectures and tasks:

• On ImageNet-256, CNS dramatically reduces Fréchet Inception Distance (FID) for SiT-XL/2 from 8.26 (SDE) to 6.27, JiT-B/16 from 32.39 (ODE) to 26.69, and JiT-H/16 from 11.88 (SDE) to 8.31, including consistent improvements in guided settings (CFG).
• CNS maintains robust performance across solver orders (1st- and 2nd-order weak), outperforming all tested ODE/SDE solvers.
• Superior performance is demonstrated in text-to-image pipelines (FLUX), improving both human-preference scores and compositional accuracy metrics (ImageReward, CLIPScore, GenEval).
• CNS retains orthogonal benefits when applied to models trained with alternative forward noise schedules (e.g., Blue Noise for Diffusion Models), confirming inference-time generalizability.
• Ablation studies reveal that global energy scaling, schedule perturbation, and alternative colored noise processes (multifractional Brownian motion) consistently yield reduced fidelity compared to CNS, highlighting the necessity of frequency- and state-dependent allocation with strict budget normalization.

Theoretical and Practical Implications

CNS offers a principled mechanism to steer generative distributions toward the true data manifold, mitigating spectral gaps and improving sample quality without any retraining. The frequency-aware noise routing mechanism directly optimizes structural alignment, harnessing network inductive biases rather than relying on numerical discretization or black-box post-processing.

The primary limitation remains that CNS is only compatible with SDE-based sampling, as deterministic ODE solvers lack inherent stochasticity. SDE-based inference also requires adequate step budgets to prevent discretization errors, which may restrict ultra-fast generation.

Future Directions

Several avenues emerge for further development:

• Extending frequency-dependent energy routing concepts into deterministic solvers, potentially accelerating low-step sampling with improved fidelity.
• Adapting CNS to video generation frameworks, leveraging both spatial and temporal frequency domains.
• Exploring dynamic spectral bias feedback loops using online $y(f, t)$ estimation for adaptive real-time sampling strategies.

Conclusion

Colored Noise Sampling constitutes a robust, theoretically justified, and empirically validated solution for improving generative fidelity in diffusion models. By explicitly leveraging model spectral bias and reallocating stochastic energy based on structural deficit, CNS offers substantial gains in image synthesis quality, maintains robust stability, and generalizes across architectures and training paradigms. The work provides new directions for principled frequency-aware inference-time optimization in generative modeling.