PIXART-δ: Fast and Controllable Image Generation with Latent Consistency Models (2401.05252v1)

Abstract: This technical report introduces PIXART-{\delta}, a text-to-image synthesis framework that integrates the Latent Consistency Model (LCM) and ControlNet into the advanced PIXART-{\alpha} model. PIXART-{\alpha} is recognized for its ability to generate high-quality images of 1024px resolution through a remarkably efficient training process. The integration of LCM in PIXART-{\delta} significantly accelerates the inference speed, enabling the production of high-quality images in just 2-4 steps. Notably, PIXART-{\delta} achieves a breakthrough 0.5 seconds for generating 1024x1024 pixel images, marking a 7x improvement over the PIXART-{\alpha}. Additionally, PIXART-{\delta} is designed to be efficiently trainable on 32GB V100 GPUs within a single day. With its 8-bit inference capability (von Platen et al., 2023), PIXART-{\delta} can synthesize 1024px images within 8GB GPU memory constraints, greatly enhancing its usability and accessibility. Furthermore, incorporating a ControlNet-like module enables fine-grained control over text-to-image diffusion models. We introduce a novel ControlNet-Transformer architecture, specifically tailored for Transformers, achieving explicit controllability alongside high-quality image generation. As a state-of-the-art, open-source image generation model, PIXART-{\delta} offers a promising alternative to the Stable Diffusion family of models, contributing significantly to text-to-image synthesis.

Overview of PixArt-: Fast and Controllable Image Generation with Latent Consistency Models

This paper introduces PixArt-$$, an advanced text-to-image synthesis framework that integrates a Latent Consistency Model (LCM) and a novel ControlNet-Transformer architecture into the existing PixArt-$$ model. It primarily aims to enhance both the speed and control of image generation tasks in high-resolution contexts. PixArt-$$is particularly notable for producing 1024px images in a mere 0.5 seconds, representing a marked improvement over previous iterations such as the PixArt-$$. Moreover, it offers an efficient training process that can run on 32GB V100 GPUs within a day, demonstrating both computational efficiency and rapid learning convergence.

Technical Contributions

The integration of the LCM into PixArt-$$provides a significant acceleration in the inference speed by approaching the reverse diffusion process as solving an augmented probability flow ODE, allowing for image generation in merely 2 to 4 steps. This framework permits effective sampling while maintaining the image quality from pre-trained latent diffusion models (LDMs). The model incorporates LCM-LoRA, enhancing user experience by supporting fine-tuning with limited computational resources.</p> <p>In addition to the speed enhancements, the paper addresses the challenge of controlling the output of generated images, especially when generated through Transformer models. Traditional ControlNet architectures had difficulties when adapted directly to Transformers. In response, the authors developed the ControlNet-Transformer architecture, effectively customizing the control capabilities to provide precise control and quality in high-resolution image generation.</p> <h3 class='paper-heading'>Experimental Outcomes</h3> <p>The empirical results show that PixArt-$$performs with significantly improved inference speeds and maintains high image generation quality. For hardware such as A100 GPUs, PixArt-$$achieves image generation in approximately 0.5 seconds—a sevenfold increase over previous methods. Further, with 8-bit inference, PixArt-$$ demonstrates the ability to synthesize high-resolution images even within the constraints of 8GB GPU memory.

In terms of ControlNet integration, the authors conducted detailed ablation studies that demonstrate the improved controllability and image quality using the ControlNet-Transformer architecture in comparison to ControlNet adaptations that mimic UNet architectures. Their results indicate a substantial improvement in controllability, especially when handling complex image details and compositions.

Theoretical and Practical Implications

The advancements presented in PixArt-$$have notable implications for both theoretical research and practical applications in <a href="https://www.emergentmind.com/topics/responsible-artificial-intelligence-ai" title="" rel="nofollow" data-turbo="false" class="assistant-link">AI</a>. The reduced inference time and memory requirements potentially broaden the accessibility and applicability of image synthesis tasks across different hardware settings, including consumer-grade GPUs. The ability to produce high-quality, controllable images rapidly has promising applications in creative industries and real-time systems where latency is a critical concern.</p> <p>Theoretically, the novel integration of LCM and ControlNet in Transformer architectures could inspire further research into bridging generative capabilities and control mechanisms, especially within transformer-based frameworks. The hybridization of models signified through PixArt-$$ may lead to future AI systems that combine efficiency with enhanced generative capabilities, underpinning practical deployments where both speed and precision are required.

Future Directions

The research opened avenues for future exploration into the optimization of ControlNet architectures, particularly in their application to diverse diffusion models beyond the scope of Transformers. Further refinement of the LCM methodologies and their application in other types of generative tasks could potentially enhance the efficiency and control of similar AI solutions. The high adaptability and potential for real-time application make PixArt-$ a pivotal step towards highly efficient and controlled generative models, likely impacting AI's role in immersive media, design, and user-centered applications.