ElasticDiffusion: Training-free Arbitrary Size Image Generation through Global-Local Content Separation (2311.18822v2)

Abstract: Diffusion models have revolutionized image generation in recent years, yet they are still limited to a few sizes and aspect ratios. We propose ElasticDiffusion, a novel training-free decoding method that enables pretrained text-to-image diffusion models to generate images with various sizes. ElasticDiffusion attempts to decouple the generation trajectory of a pretrained model into local and global signals. The local signal controls low-level pixel information and can be estimated on local patches, while the global signal is used to maintain overall structural consistency and is estimated with a reference image. We test our method on CelebA-HQ (faces) and LAION-COCO (objects/indoor/outdoor scenes). Our experiments and qualitative results show superior image coherence quality across aspect ratios compared to MultiDiffusion and the standard decoding strategy of Stable Diffusion. Project page: https://elasticdiffusion.github.io/

Analyzing Diffusion Models for Image Generation Across Variable Resolutions and Aspect Ratios

This paper presents a significant exploration into the capabilities of diffusion models in generating images across varied resolutions and aspect ratios, utilizing a novel training-free decoding method named . The paper explores the limitations of traditional diffusion models, which are often constrained to specific image sizes, and proposes an innovative solution that allows for diverse size generation while maintaining coherent image quality without additional training overhead.

Methodological Overview

The authors propose a method to extend the capabilities of pretrained diffusion models, specifically focusing on the decoupling of generation trajectories into local and global signals. This distinction enables the generation of images at arbitrary resolutions by controlling local pixel-level details and maintaining overall structural consistency through global signals. The approach is grounded in the application of classifier-free guidance, splitting it into local and global content mechanisms which govern pixel precision and overall image composition, respectively.

The methodology involves several key components:

1. Unconditional Score Calculation: Local patches are used to estimate pixel-level details with surrounding context information, enhancing efficiency and boundary transitions.
2. Class Direction Score Estimation: Downsampling and upsampling procedures are employed to maintain global coherence across varied image resolutions, drawing on reference latents at the model's pretrained dimensions.
3. Iterative Resampling: This technique is used to refine and upscale the resolution of the class direction score, improving image detail and reducing oversmoothing.
4. Reduced-Resolution Guidance (RRG): A strategy to align generated images with a low-resolution reference, minimizing artifacts and maintaining coherence.

Empirical Results

The paper provides extensive empirical evidence through qualitative and quantitative evaluations on datasets such as CelebA-HQ and LAION-COCO. The proposed method significantly improves image coherence and quality across standard and extreme aspect ratios compared to benchmark models, including Stable Diffusion and SDXL, without additional computational demands.

Quantitatively, the method achieves competitive FID and CLIP-scores even at resolutions that surpass the native training sizes of the baseline models. These metrics are crucial indicators of the model's ability to generate realistic images that align with textual cues.

Practical and Theoretical Implications

Practically, the work allows for flexible applications of generative models in scenarios that require diverse image sizes and aspect ratios, such as in digital media, wearable tech, and automotive displays. Theoretically, it presents a new understanding of the guidance mechanisms within diffusion models. The separation of global and local content generation opens avenues for further research into style transfer, selective content manipulation, and enhancement of diffusion models' capabilities.

Future developments could explore perfecting the separation of signals and exploring how this impacts the generation fidelity beyond the explored 2X resolution range. Additionally, examining the interplay between global structure and local detail could lead to advancements in other generative model domains, potentially influencing more efficient training and decoding paradigms.

In conclusion, this approach effectively expands the functional range of diffusion models in image synthesis, paving the way for broader applications and setting the stage for future innovations in adaptive and flexible image generation techniques.