Spectral Progressive Diffusion for Efficient Image and Video Generation

Abstract: Diffusion models have been shown to implicitly generate visual content autoregressively in the frequency domain, where low-frequency components are generated earlier in the denoising process while high-frequency details emerge only in later timesteps. This structure offers a natural opportunity for efficient generation, as high-resolution computation on noise-dominated frequencies is largely redundant. We propose Spectral Progressive Diffusion, a general framework that progressively grows resolution along the denoising trajectory of pretrained diffusion models. To this end, we develop a spectral noise expansion mechanism and derive an optimal resolution schedule from the model's power spectrum. Our framework supports training-free acceleration and a novel fine-tuning recipe that further improves efficiency and quality. We demonstrate significant speedups on state-of-the-art pretrained image and video generation models while preserving visual quality.

• The paper introduces SPD, a method that leverages spectral autoregression for progressive resolution growth, reducing redundant computation in early diffusion stages.
• It employs spectral noise expansion and closed-form analysis to optimally schedule transitions, achieving up to 7× speedup in image generation and 2.5× in video.
• SPD integrates with pretrained diffusion models without changes, offering both training-free operation and fine-tuning options to preserve high generation quality.

Spectral Progressive Diffusion: Spectral-Autoregressive Resolution Growth for Efficient Visual Generation

Introduction and Motivation

With the increasing computational cost of high-resolution image and video generation via diffusion models—especially with the widespread adoption of Diffusion Transformers (DiTs)—efficient sampling is now a central concern. The quadratic scaling of self-attention with token count exacerbates the resource bottleneck as visual fidelity demands increase. Although prior works in frequency-domain generative modeling and progressive resolution pipelines have targeted this efficiency problem, most either mandate full retraining, modifying architectures, or rely on brittle heuristics unsuitable for state-of-the-art pretrained models.

This paper introduces Spectral Progressive Diffusion (SPD), a general, training-free and fine-tunable framework for efficient image and video generation via progressive resolution growth governed by spectral-domain autoregression in diffusion models (2605.18736). SPD directly leverages the frequency-wise emergence of signal in diffusion trajectories: early denoising steps are low-frequency dominated and can therefore be computed at a reduced spatial resolution, only upsampling when high-frequency content emerges from noise. This approach not only reduces the number of tokens processed through expensive DiT modules, but also applies directly to pretrained models in both training-free and fine-tuned modes, without architectural modifications.

Spectral-Autoregressive Properties of Diffusion Models

The theoretical foundation for SPD is the spectral autoregression property observed in diffusion models: as the denoising process unfolds, low-frequency components emerge first, with high-frequency structure delayed until later steps. This follows the empirical power-law decay of the power spectra of natural images and their latent representations—low frequencies have substantially higher variance, while high frequencies become signal-dominated only at late denoising steps.

Figure 1: Latent power spectra in image and video models show rapid frequency decay (a) and autoregressive emergence of frequency bands (b) along diffusion steps.

As a result, computation spent on high-resolution token processing at early steps is largely redundant. SPD formalizes progressive resolution growth by using spectral domain analysis to guide optimal scheduling of transitions from low- to high-resolution computation.

Spectral Progressive Diffusion: Methodology

Progressive Resolution Schedule and Spectral Noise Expansion

At the core of SPD is a progressive growth of resolution during denoising, matched to the signal-to-noise profile calculated per-frequency. For $S$ stages indexed by increasing resolution scales $s_1 < \cdots < s_S$, transitions occur at analytically derived timesteps $t_1 > \cdots > t_{S-1}$ determined by the frequency activation time—that is, the timestep at which each frequency band becomes signal-dominated above a prescribed error threshold $\delta$.

At each transition, a spectral noise expansion mechanism injects Gaussian noise into newly available high-frequency bands, while preserving the state of low-frequency bands partially denoised so far. Transition state rescaling and timestep alignment ensure faithful ODE integration resumes in accordance with the original diffusion schedule.

Figure 2: SPD grows resolution along the denoising trajectory, using spectral noise expansion to inject high-frequency noise while keeping denoised low-frequency content.

SPD can employ any orthonormal spectral basis, with experiments demonstrating that Discrete Cosine Transform (DCT) and Discrete Wavelet Transform (DWT) are effective; Fast Fourier Transform (FFT) is less favorable due to oversmoothing artifacts.

Optimal Resolution Scheduling

Transition times are not hand-tuned but are derived from closed-form analysis of the spectral domain flow-matching ODE for a given error threshold $\delta$. The activation time $t_\omega$ at which frequency $\omega$ should be revealed satisfies:

$$t_\omega = \frac{1}{1 + \sqrt{\delta / (P_\omega(1 + P_\omega - \delta))}}$$

where $P_\omega$ is the empirical power spectrum. The transition from $s_i$ to $s_1 < \cdots < s_S$0 scales occurs at the minimum $s_1 < \cdots < s_S$1 over the frequencies representable at scale $s_1 < \cdots < s_S$2, thus robustly maximizing both information utilization and computational savings.

Training-Free and Fine-Tuned Operation

SPD applies training-free to pretrained models by simply substituting the standard forward process with the spectrally scheduled, progressive-resolution variant, requiring no model changes or retraining. For further performance gains, SPD includes a spectral-transformation-based fine-tuning recipe, especially useful for bridging distributional mismatches arising when the pretrained model was not exposed to intermediate resolutions. This LoRA-based fine-tuning adapts the generative backbone to directly follow the spectrally progressive schedule, boosting both quality and efficiency.

Empirical Evaluation

Quantitative Results

SPD is evaluated on state-of-the-art generative backbones for (i) latent-space image synthesis (FLUX.1-dev, Z-Image), (ii) pixel-space generation (PixelGen), and (iii) latent-space video (WAN-2.1). Across tasks, SPD offers substantial acceleration—up to 7× speedup in image generation and 2.5× speedup in video—while maintaining generation quality as measured by ImageReward, CLIP-IQA, NIQE, and text-image alignment metrics. In all settings, SPD either matches or surpasses the quality/speed tradeoff of baseline spatial and temporal accelerators such as RALU and Bottleneck.

Figure 3: SPD achieves higher fidelity and speed versus the state-of-the-art spatial acceleration method RALU in latent-space image generation.

Fine-tuning with SPD further closes the gap to the full-resolution, full-step baselines, with consistent improvements in both generation quality and throughput.

Qualitative Analysis

SPD preserves semantic fidelity and visual quality as compared to baseline approaches, notably avoiding artifacts such as ghosting or oversmoothing when using DCT or DWT bases. Qualitative ablations demonstrate that increasing $s_1 < \cdots < s_S$3 amplifies speedups at the expense of edge fidelity, while higher $s_1 < \cdots < s_S$4 (number of resolution stages) provides marginal extra gains past $s_1 < \cdots < s_S$5.

Figure 4: (a) Pareto tradeoff between speed and image quality as $s_1 < \cdots < s_S$6 or $s_1 < \cdots < s_S$7 increases; (b) SPD-based editing achieves higher prompt alignment and geometric consistency than SDEdit-style spatial editing.

Frequency-Based Image Editing

SPD’s spectral formalism naturally extends to prompt-based conditional editing. By manipulating specific spectral bands at aligned diffusion steps, SPD can modify texture or artistic style with superior structural consistency compared to spatial-domain SDEdit methods, which often induce geometric drift or fail to transfer high-frequency detail.

Theoretical and Practical Implications

Theoretical contributions include the formalization of frequency-wise activation criteria, linking spectral power to computational scheduling via information-theoretic optimality, and providing an analytically robust foundation for progressive resolution growth. The robustness to hyperparameter choices—defining all transitions from a single $s_1 < \cdots < s_S$8—is in stark contrast to heuristic progressive resolution pipelines.

Practically, SPD provides a truly general, plug-and-play acceleration strategy compatible with any pretrained DiT or flow-matching model, opening avenues for efficient, scalable deployment of high-fidelity image/video generation in resource-constrained settings.

The fine-tuning paradigm, using spectral noise expansion as an instructional regimen, suggests the potential for co-training generative models directly on spectrally progressive schedules, possibly improving robustness and further reducing the inference–training gap.

Future Directions

While focusing on spatial spectra, natural extensions include temporal-frequency progressive sampling—potentially yielding further acceleration for high-resolution, long-sequence video models—and application to multi-modal or 3D generative pipelines. Integrating SPD as an inductive bias during model pretraining may yield even larger efficiency dividends, and investigating its synergy with other acceleration technologies (token merging, sparse attention, feature caching) remains an open direction.

Conclusion

Spectral Progressive Diffusion presents a principled, theoretically-grounded, and highly effective approach to resolving the efficiency bottleneck in high-resolution generative modeling. By aligning computational effort with the emergent information in the spectral domain along the denoising trajectory, SPD enables substantial inference acceleration for both images and video while preserving or enhancing generation quality. The framework’s compatibility with current and future pretrained models, alongside its generality and theoretical rigor, mark it as a significant advance in scalable visual generative modeling (2605.18736).