UniFL: Improve Latent Diffusion Model via Unified Feedback Learning

Abstract: Latent diffusion models (LDM) have revolutionized text-to-image generation, leading to the proliferation of various advanced models and diverse downstream applications. However, despite these significant advancements, current diffusion models still suffer from several limitations, including inferior visual quality, inadequate aesthetic appeal, and inefficient inference, without a comprehensive solution in sight. To address these challenges, we present UniFL, a unified framework that leverages feedback learning to enhance diffusion models comprehensively. UniFL stands out as a universal, effective, and generalizable solution applicable to various diffusion models, such as SD1.5 and SDXL. Notably, UniFL consists of three key components: perceptual feedback learning, which enhances visual quality; decoupled feedback learning, which improves aesthetic appeal; and adversarial feedback learning, which accelerates inference. In-depth experiments and extensive user studies validate the superior performance of our method in enhancing generation quality and inference acceleration. For instance, UniFL surpasses ImageReward by 17% user preference in terms of generation quality and outperforms LCM and SDXL Turbo by 57% and 20% general preference with 4-step inference.

• The paper introduces UniFL, a framework that unifies perceptual, decoupled, and adversarial feedback learning to significantly improve visual fidelity and inference speed.
• It employs VGG-based style optimization, instance segmentation for structure, and active prompt selection to holistically enhance image generation quality.
• Experimental results reveal a 17% increase in user preference and up to 57% faster inference compared to leading diffusion model approaches.

UniFL: A Unified Framework for Enhancing Stable Diffusion Models

Introduction

The rapid evolution of diffusion models, particularly in the domain of Text-to-Image (T2I) generation, has heralded significant advancements, setting new benchmarks in image generation quality, innovation in application domains, and a leap in T2I, T2V (Text-to-Video) generations. Despite the propensity for progress, existing solutions grapple with challenges like inferior visual generation quality, aesthetic misalignments, and suboptimal inference speeds. Addressing these lacunae, this paper introduces UniFL (Unified Feedback Learning), a universal framework designed to enhance diffusion models' performance across dimensions of visual and aesthetic quality and inference efficiency. UniFL's novel integration of perceptual feedback learning, decoupled feedback learning, and adversarial feedback learning delineates a path toward achieving unprecedented enhancements in visual and aesthetic generation quality alongside inference acceleration.

Methodology

UniFL's approach is delineated into three key components: Perceptual Feedback Learning (PeFL), Decoupled Feedback Learning, and Adversarial Feedback Learning, each targeting distinct facets of the generation process to holistically improve diffusion models.

Perceptual Feedback Learning (PeFL)

PeFL is crafted to ameliorate visual generation quality through the integration of existing perceptual models to provide feedback. This enables UniFL to elevate generation details with higher fidelity across diverse visual aspects, such as style, structure, and layout. For instance, leveraging VGG networks for style optimization and instance segmentation models for structure, PeFL imparts precision in feedback signals to enhance generated visual content.

Decoupled Feedback Learning

Addressing the subjective nature of aesthetics, UniFL employs decoupled feedback learning. This method decomposes the overarching concept of aesthetics into tangible components like color, atmosphere, and texture. It further introduces an active prompt selection strategy, ensuring that the feedback process is efficient and equipped to refine aesthetic preferences in generated images.

Adversarial Feedback Learning

To mitigate the inefficiencies inherent in the iterative denoising process of diffusion models, UniFL adopts an adversarial feedback learning strategy. By training the reward and diffusion models in an adversarial manner, this component significantly accelerates the inference process, enabling swift generation without compromising quality.

Results and Implications

UniFL's efficacy is empirically validated through extensive experiments and user studies. The framework demonstrates superior performance over existing methods, including a 17% increase in user preference over ImageReward concerning generation quality and outperforming notable acceleration approaches like LCM and SDXL Turbo by 57% and 20%, respectively. These improvements underscore UniFL's potential to serve as a cornerstone in driving future research and applications in image generation, fostering greater efficiency and quality in output.

Future Directions

The paper speculates on several avenues for future exploration, such as expanding the repertoire of visual perception models within PeFL, pushing the boundaries of acceleration to extreme scenarios, and simplifying the two-stage optimization process into a cohesive single-stage approach. These prospects highlight an exciting trajectory for further advancements in AI-driven generative models, particularly within the field of stable diffusion models.

Conclusion

In conclusion, UniFL offers a pioneering and comprehensive solution addressing the intertwined challenges of visual quality, aesthetic alignment, and inference efficiency in stable diffusion models. By amalgamating perceptual insights, decoupled aesthetics optimization, and adversarial feedback mechanisms, UniFL not only sets a new benchmark in the enhancement of diffusion models but also paves the way for multifaceted improvements in generative AI applications.