Show the Signal, Hide the Noise: Spectral Forcing for Pixel-Space Diffusion

Abstract: Pixel-space diffusion models are trained on full-bandwidth noisy images, yet the useful signal available to the denoiser is strongly frequency dependent. Under rectified-flow diffusion and natural-image power-law spectra, the per-band data-to-noise contour $k^{*}(t) = (1-t)^{-2/α}$ separates a signal-bearing low-frequency region from a noise-dominated high-frequency region at each time $t$. We show that this implicit coarse-to-fine structure is not merely descriptive: it induces a capacity-allocation problem. A standard pixel-space denoiser must discover the moving bandwidth boundary internally and can spend computation on frequency-time regions where the optimal prediction collapses to deterministic baselines rather than data-distribution modeling. To make this boundary explicit, we introduce Spectral Forcing, a parameter-free, time-conditional 2D-DCT low-pass operator applied to the noisy input before the patch embedder. Its cutoff expands monotonically with the diffusion time and becomes the identity at the data endpoint. Through controlled synthetic experiments, we identify the regime in which the operator is beneficial: coarse patch tokenization and data whose high-frequency content is predominantly noise rather than essential signal. On ImageNet-256 with JiT-700M/32, Spectral Forcing consistently improves both FID and Inception Score across different training epochs, demonstrating robust gains throughout training; at finer tokenization, the spectral forcing is still competitive. We further insert the unchanged operator into SenseNova-U1, a unified text-to-image model, where it improves DPG-Bench and GenEval, showing that the input-side spectral prior transfers beyond class-conditional generation. These results suggest a route to capacity-efficient pixel-space diffusion by showing the signal and hiding the noise.

• The paper introduces a parameter-free spectral forcing mechanism that explicitly filters out high-frequency noise in pixel-space diffusion models.
• It employs a time-conditional 2D-DCT low-pass filter to realign model capacity towards recoverable low-frequency signals, yielding improved FID and Inception Score metrics.
• The study establishes an analytical framework underpinning the efficiency and scalability of spectral forcing, with broad implications for generative modeling and transformer-based diffusion pipelines.

Spectral Forcing: Explicit Frequency-Bounded Input Priors for Pixel-Space Diffusion

Overview

This paper introduces "Spectral Forcing" (SF), a parameter-free, time-conditional 2D-DCT low-pass filter that is applied to noisy inputs in pixel-space diffusion models. The motivation is grounded in an empirical and analytical observation that, at each diffusion step, only a low-frequency portion of the input contains recoverable data signal, while the high-frequency components are dominated by noise. Standard pixel-space denoisers are thus forced to implicitly learn and adapt to this moving frequency boundary, wasting capacity on frequency-time regions where prediction collapses to trivial baselines. SF directly imposes this boundary at the input, freeing model capacity and improving generative performance, especially in regimes characterized by coarse patch tokenization and noise-dominated high-frequency bands.

Analytical Framework and Empirical Diagnostics

To analyze the distribution of recoverable signal versus noise across frequencies and diffusion times, the authors adopt the rectified-flow framework, where the model observes $z_t = t\,x + (1-t)\,\varepsilon$. For natural images with power spectrum $P(k) \propto k^{-\alpha}$, they define a closed-form per-band data-to-noise ratio (DNR): $\mathrm{DNR}(k, t) = k^{-\alpha}/(1-t)^2$. The critical frequency cut-off at each $t$ is then $k^*(t) = (1-t)^{-2/\alpha}$; below this contour, the signal is recoverable, above it noise prevails.

Empirical diagnostics on both synthetic and ImageNet data reveal that, at convergence, the denoiser only learns non-trivial structure in the wedge-shaped $(k,t)$ region below the DNR contour. Elsewhere, predictions converge to deterministic, trivial mappings such as linear rescaling or zeros, effectively wasting capacity.

Spectral Forcing Operator

SF imposes an explicit, deterministic low-pass mask in the DCT domain, whose cut-off radius $c(t)$ monotonically expands with $t$ (the denoising process), saturating at the endpoint for reconstruction of full-bandwidth data. Importantly, the SF operator is:

• Parameter-free and non-learned: No architectural or loss-side changes.
• Time-conditional: The cut-off follows a schedule (default: linear in $t$ or analytical tracking of the $k^*(t)$ boundary).
• Efficient: Computationally, introduces only $P(k) \propto k^{-\alpha}$0 overhead per denoising step at $P(k) \propto k^{-\alpha}$1 resolutions.

The schedule can be tailored to the empirical spectrum ($P(k) \propto k^{-\alpha}$2), but a simple linear ramp performs best on ImageNet at standard resolutions; analytical schedules recover at higher resolutions.

Main Results

The efficacy of SF is conditioned on two factors:

• Coarse patch tokenization: Where the patch grid already bandlimits input, highlighting the model's vulnerability to wasted capacity at high frequency.
• Data with power-law spectrum: Natural images with rapidly decaying high-frequency energy, wherein most high-frequency bands are noise.

In this regime (e.g., JiT-700M/32 trained on ImageNet-256 with 64 tokens), SF achieves:

• FID improvement: $P(k) \propto k^{-\alpha}$3 (from 24.19 to 20.68) at 60 epochs, robust across epoch budgets (still $P(k) \propto k^{-\alpha}$4 at 120 epochs).
• Inception Score improvement: $P(k) \propto k^{-\alpha}$5 (from 83.28 to 93.96 at 60 epochs).
• Consistent gains throughout training: Improvements are seen at every training epoch, not merely at early stages.

Comparative studies show that alternative frequency-domain or blurring operators (constant DCT low-pass, spatial Gaussian blur, Focal Frequency Loss, blurring diffusion, DCTDiff) either harm performance or are neutral, uniquely positioning SF as an effective spectral input constraint. SF also transfers to unified text--image models (SenseNova-U1), improving both DPG-Bench and GenEval at early checkpoints, indicating robustness to task and conditioning type.

Regime Dependencies and Limitations

Ablative studies clarify the precise regime of applicability:

• Coarse patchification with noise-dominated high frequencies: SF significant improvement.
• Finer patch or edge-heavy data: SF is neutral or slightly harmful, as high-frequency content is essential to the signal.
• Resolution scaling: At higher image resolution, SF benefits persist even as token count grows, provided the token grid remains aggressive relative to recoverable frequency content.

Hyperparameters (transition sharpness, cutoff bounds) are robust, and the method does not require per-task retuning. Computation cost is negligible for practical batch sizes.

Theoretical and Practical Implications

Theoretically, SF operationalizes the empirical finding that diffusion models' frequency-time dynamics are dominated by a moving recoverable signal boundary. Input masking explicitizes this, aligning model capacity with "data regions" rather than having the network learn to waste computation in "noise regions."

Practically, this opens avenues for:

• Optimizing pixel-space diffusion architectures for larger scales and lower compute budgets.
• Building spectral priors into vision-LLMs, especially those operating natively on images with aggressive tokenization.
• Informing design of transformer-based diffusion and generative pipelines, possibly revising the standard practice of operating in latent space for efficiency.

Future developments may explore adaptive spectral masking schedules and integrating spectral priors with scalable joint vision-language modeling.

Conclusion

Spectral Forcing constitutes an explicit, efficient mechanism for aligning model capacity with information-bearing regions in pixel-space diffusion, leading to significant performance gains in the relevant regime. This work formalizes the frequency-time allocation problem, introduces a robust and generalizable solution, and provides a practical method ready for integration into modern diffusion pipelines, with strong implications for both theory and practice in large-scale generative modeling.