Improving Sample Quality of Diffusion Models Using Self-Attention Guidance (2210.00939v6)

Abstract: Denoising diffusion models (DDMs) have attracted attention for their exceptional generation quality and diversity. This success is largely attributed to the use of class- or text-conditional diffusion guidance methods, such as classifier and classifier-free guidance. In this paper, we present a more comprehensive perspective that goes beyond the traditional guidance methods. From this generalized perspective, we introduce novel condition- and training-free strategies to enhance the quality of generated images. As a simple solution, blur guidance improves the suitability of intermediate samples for their fine-scale information and structures, enabling diffusion models to generate higher quality samples with a moderate guidance scale. Improving upon this, Self-Attention Guidance (SAG) uses the intermediate self-attention maps of diffusion models to enhance their stability and efficacy. Specifically, SAG adversarially blurs only the regions that diffusion models attend to at each iteration and guides them accordingly. Our experimental results show that our SAG improves the performance of various diffusion models, including ADM, IDDPM, Stable Diffusion, and DiT. Moreover, combining SAG with conventional guidance methods leads to further improvement.

• The paper introduces Self-Attention Guidance (SAG), a method that uses internal self-attention maps to enhance sample quality in denoising diffusion models.
• The methodology refines intermediate representations by selectively blurring salient regions, leading to improved metrics such as FID and Inception Scores.
• SAG's approach is versatile for both conditional and unconditional settings, simplifying training by eliminating the need for external labels or captions.

Improving Sample Quality of Diffusion Models Using Self-Attention Guidance

This paper addresses the challenge of improving the sample quality of denoising diffusion models (DDMs), which have gained significant attention for their high image generation quality and diversity. While diffusion models have successfully leveraged class- or text-conditional guidance methods like classifier and classifier-free guidance to enhance image generation, these approaches depend on additional external conditions, such as class labels or captions, adding complexity to the model training and limiting their applicability to conditional settings. The paper proposes an alternative approach that utilizes internal model information to enhance sample quality, broadening the applicability by removing the dependency on external conditions.

The authors introduce Self-Attention Guidance (SAG), a novel method aimed at refining the sample quality of diffusion models without additional training or external guidance cues. SAG leverages the self-attention maps of diffusion models to improve the stability and quality of image synthesis. In contrast to traditional guidance mechanisms, SAG requires no external conditions, making it versatile for both conditional and unconditional models.

Key Insights and Methodology

1. Generalized Diffusion Guidance:
   • The paper proposes a generalized framework for diffusion guidance that encompasses both conditional and unconditional settings. This framework relies on internal information within intermediate diffusion process samples rather than external labels, opening up guidance applications for scenarios lacking labeled data.
2. Blur Guidance:
   • As a precursor to SAG, blur guidance is introduced, which utilizes Gaussian blur to progressively eliminate fine-scale details of intermediate samples. Although this method demonstrates sample quality improvements with a moderate guidance scale, it introduces structural ambiguities at larger scales, leading to noisy results—a limitation SAG aims to overcome.
3. Self-Attention Guidance (SAG):
   • SAG operates by selectively blurring self-attended regions identified through self-attention maps, focusing on the salient information necessary for high-quality generation. By targeting these specific areas during the diffusion process, SAG enhances fidelity and reduces artifacts more effectively than blur guidance alone.
   • It optimizes the generation process, especially in large-scale models, by integrating the residual information iteratively across generation steps, leading to improved artifact reduction and enhanced detail elaboration in the outputs.

Experimental Outcomes

• Scope and Evaluation:
  • SAG was evaluated across a broad spectrum of models, including ADM, IDDPM, Stable Diffusion, and DiT. The results highlight a consistent enhancement in sample quality and detail, confirmable through metrics like FID, IS, and precision scores.
  • The method maintains orthogonality with existing guidance strategies, enabling possible fusion with traditional methods to further boost performance.
• Quantitative Improvements:
  • Using SAG on various models resulted in noticeable reductions in FID scores and increases in Inception Scores across multiple datasets, indicating significant gains in both quality and diversity of generated images.
  • Additionally, user studies indicated a preference for qualitatively improved SAG-enhanced samples over baseline outputs.

Future Directions and Implications

SAG represents a valuable advancement in the field of generative modeling, particularly diffusion models. By creating an approach that is free from external conditions and additional training requirements, this method simplifies the implementation while improving generative performance. It could inspire further exploration into leveraging internal model representations, such as self-attention, to refine generative processes across different types of generative adversarial networks (GANs) and other probabilistic models. Future work might focus on optimizing computational efficiency or exploring the broader integration of SAG with diverse generative architectures and tasks beyond image synthesis.