DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecture (2409.03550v2)

Abstract: Diffusion models (DMs) have demonstrated exceptional generative capabilities across various domains, including image, video, and so on. A key factor contributing to their effectiveness is the high quantity and quality of data used during training. However, mainstream DMs now consume increasingly large amounts of data. For example, training a Stable Diffusion model requires billions of image-text pairs. This enormous data requirement poses significant challenges for training large DMs due to high data acquisition costs and storage expenses. To alleviate this data burden, we propose a novel scenario: using existing DMs as data sources to train new DMs with any architecture. We refer to this scenario as Data-Free Knowledge Distillation for Diffusion Models (DKDM), where the generative ability of DMs is transferred to new ones in a data-free manner. To tackle this challenge, we make two main contributions. First, we introduce a DKDM objective that enables the training of new DMs via distillation, without requiring access to the data. Second, we develop a dynamic iterative distillation method that efficiently extracts time-domain knowledge from existing DMs, enabling direct retrieval of training data without the need for a prolonged generative process. To the best of our knowledge, we are the first to explore this scenario. Experimental results demonstrate that our data-free approach not only achieves competitive generative performance but also, in some instances, outperforms models trained with the entire dataset.

• The paper introduces DKDM as a novel data-free framework to distill diffusion models by generating synthetic denoising data.
• It proposes a dynamic iterative distillation objective that mirrors traditional diffusion model training while reducing computational resources.
• Experiments show that DKDM-accelerated models achieve up to twice the generation speed with maintained high-quality outputs across multiple datasets.

Overview of "DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecture"

The paper "DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecture" introduces a novel paradigm for accelerating Diffusion Models (DMs) through a data-free approach. This method, termed Data-Free Knowledge Distillation for Diffusion Models (DKDM), is designed to transfer the generative capabilities of large pretrained DMs to more efficient architectures without the need for source data. The authors address the challenges of slow inference speeds and high computational demands, which limit the practical deployment of DMs.

Key Contributions

1. Data-Free Knowledge Distillation (DKDM): The paper introduces DKDM as a new framework for distilling DMs without accessing the source data. This approach leverages the generative ability of pretrained models to create synthetic denoising data, thereby optimizing faster DM architectures.
2. Novel DKDM Objective: The authors propose a DKDM objective that aligns closely with the traditional DM optimization goals while eliminating dependencies on source data. This objective effectively distills knowledge from the teacher to the student model.
3. Dynamic Iterative Distillation: To handle the bottleneck of generating synthetic data, the authors introduce a dynamic iterative distillation method. This approach allows for efficient collection and use of denoising data, significantly reducing computation and storage requirements compared to naïve methods.
4. Versatility Across Architectures: DKDM is architecture-agnostic, allowing the distillation of knowledge from large models into student models of any architectural design. This flexibility enables broad applicability across various model configurations.

Experimental Results

The experiments demonstrate that DMs distilled using DKDM can achieve up to twice the generation speed compared to baseline models while maintaining comparable performance in terms of generative quality. Specifically, the DKDM-accelerated models show substantial improvements in metrics such as Inception Score (IS) and Fréchet Inception Distance (FID) across multiple datasets, including CIFAR10, CelebA, and ImageNet.

DKDM is shown to complement existing acceleration techniques like denoising step reduction and quantization. Notably, it facilitates the efficient training of DMs without requiring access to large, proprietary datasets, thereby addressing logistical and privacy concerns prevalent in current KD methods.

Implications and Future Directions

The DKDM paradigm presents a significant advancement in the scalability and deployment of diffusion models, particularly for consumer applications where processing power and computational efficiency are critical. By removing the dependency on source datasets, DKDM allows for greater flexibility and reduces barriers to experimentation and deployment in varied environments.

Future research could explore integrating DKDM with advanced techniques like Neural Architecture Search (NAS) to further optimize DMs in terms of speed and performance. Furthermore, addressing the underlying distribution gap between the synthetic data generated by the teacher model and the original data could potentially enhance the efficacy of the distilled models.

Conclusion

The paper presents a substantial contribution to the field of machine learning and model optimization, offering a practical solution to the challenges faced by diffusion models. By enabling data-free distillation, DKDM not only improves the inference speed of DMs but also bypasses data privacy issues, paving the way for more robust and versatile applications of generative models. This research sets a foundation for future endeavors in accelerating and deploying complex generative models efficiently and securely.