Diffusion Probabilistic Model Made Slim (2211.17106v1)

Abstract: Despite the recent visually-pleasing results achieved, the massive computational cost has been a long-standing flaw for diffusion probabilistic models (DPMs), which, in turn, greatly limits their applications on resource-limited platforms. Prior methods towards efficient DPM, however, have largely focused on accelerating the testing yet overlooked their huge complexity and sizes. In this paper, we make a dedicated attempt to lighten DPM while striving to preserve its favourable performance. We start by training a small-sized latent diffusion model (LDM) from scratch, but observe a significant fidelity drop in the synthetic images. Through a thorough assessment, we find that DPM is intrinsically biased against high-frequency generation, and learns to recover different frequency components at different time-steps. These properties make compact networks unable to represent frequency dynamics with accurate high-frequency estimation. Towards this end, we introduce a customized design for slim DPM, which we term as Spectral Diffusion (SD), for light-weight image synthesis. SD incorporates wavelet gating in its architecture to enable frequency dynamic feature extraction at every reverse steps, and conducts spectrum-aware distillation to promote high-frequency recovery by inverse weighting the objective based on spectrum magni tudes. Experimental results demonstrate that, SD achieves 8-18x computational complexity reduction as compared to the latent diffusion models on a series of conditional and unconditional image generation tasks while retaining competitive image fidelity.

• The paper introduces Spectral Diffusion (SD), a method that leverages frequency dynamics for lightweight, high-fidelity image synthesis.
• Wavelet gating dynamically adjusts frequency responses during diffusion, enabling progressive refinement of high-frequency image details.
• Frequency-aware distillation reduces computational complexity by 8–18 times while achieving competitive FID scores across multiple datasets.

Diffusion Probabilistic Model Made Slim

The paper "Diffusion Probabilistic Model Made Slim" provides innovative methodologies to reduce the computational burden of diffusion probabilistic models (DPMs) while maintaining image fidelity. This subject is crucial as DPMs, known for producing high-quality generative outputs, are computationally intensive, limiting their applicability on resource-constrained platforms.

Key Contributions

1. Spectral Diffusion (SD) Design: The paper introduces a novel approach, termed Spectral Diffusion (SD), for constructing lightweight diffusion models for image synthesis. The SD model incorporates frequency dynamics into both the architecture and training objectives, enhancing the ability of compact models to recover high-frequency details crucial for image quality.
2. Wavelet Gating Mechanism: SD utilizes wavelet gating to dynamically adapt the model's frequency response during different stages of the diffusion process. This allows the model to process low-frequency components initially, progressively incorporating high-frequency details as the model nears completion of the reverse diffusion process.
3. Frequency-Aware Distillation: Addressing DPMs' intrinsic frequency bias, the paper presents a distillation approach that emphasizes high-frequency components. By modulating the distillation loss according to spectral magnitudes, the compact student model can mimic the output of larger networks in a more balanced manner across different frequency bands.

Experimental Validation

The proposed SD model is empirically validated across multiple datasets, including FFHQ, CelebA-HQ, and LSUN. The experimental results are compelling, with SD achieving an 8 to 18 times reduction in computational complexity compared to standard latent diffusion models, while retaining competitive image fidelity. For instance, SD attains nearly equivalent FID scores on FFHQ with a significantly reduced model size and computational requirement.

Implications and Future Directions

• Practical Applications: The findings augment the practical applicability of DPMs, particularly in scenarios where computational resources are limited, such as on mobile or embedded devices.
• Theoretical Insights: The paper provides theoretical insights into the frequency evolution phenomena within DPMs, backed by the observation that DPMs inherently manage different frequency components over their denoising steps.
• Potential Extensions: Future research may focus on leveraging these spectral insights to optimize other neural architectures within the domain of generative models, potentially integrating these methodologies into frameworks like GANs and VAEs for further efficiency gains.

In conclusion, the paper makes substantial strides in addressing the computational challenges associated with state-of-the-art diffusion models. Through meticulous architectural innovations and targeted training enhancements, it positions the spectral-aware slim DPMs as a feasible alternative for high-fidelity generative tasks across various application domains.