FreeScale: Unleashing the Resolution of Diffusion Models via Tuning-Free Scale Fusion (2412.09626v1)

Abstract: Visual diffusion models achieve remarkable progress, yet they are typically trained at limited resolutions due to the lack of high-resolution data and constrained computation resources, hampering their ability to generate high-fidelity images or videos at higher resolutions. Recent efforts have explored tuning-free strategies to exhibit the untapped potential higher-resolution visual generation of pre-trained models. However, these methods are still prone to producing low-quality visual content with repetitive patterns. The key obstacle lies in the inevitable increase in high-frequency information when the model generates visual content exceeding its training resolution, leading to undesirable repetitive patterns deriving from the accumulated errors. To tackle this challenge, we propose FreeScale, a tuning-free inference paradigm to enable higher-resolution visual generation via scale fusion. Specifically, FreeScale processes information from different receptive scales and then fuses it by extracting desired frequency components. Extensive experiments validate the superiority of our paradigm in extending the capabilities of higher-resolution visual generation for both image and video models. Notably, compared with the previous best-performing method, FreeScale unlocks the generation of 8k-resolution images for the first time.

• The paper introduces a tuning-free framework that fuses multi-scale information to boost high-resolution image and video generation.
• It employs tailored self-cascade upscaling and restrained dilated convolution to preserve global coherence and local details.
• Empirical results show superior FID and KID scores at resolutions up to 8192×8192 using a single GPU.

FreeScale: Tuning-Free Scale Fusion for High-Resolution Generation in Diffusion Models

The paper "FreeScale: Unleashing the Resolution of Diffusion Models via Tuning-Free Scale Fusion" presents a methodological innovation in extending the capabilities of pre-trained diffusion models for higher-resolution image and video generation. The significant impediments in scaling visual diffusion models to higher resolutions primarily arise from limitations in high-resolution data and substantial computation resources. FreeScale introduces a novel tuning-free inference paradigm that integrates scale fusion to overcome these challenges.

Methodological Innovations

FreeScale is built on three core components that address the common limitations in existing approaches, such as local repetitiveness and quality degradation:

1. Tailored Self-Cascade Upscaling: This process progressively upscales generated content, beginning with lower resolutions, ensuring that the structural integrity of the visual content is maintained while limiting repetitive patterns. The self-cascade approach achieves a balance between maintaining global coherence and adding local details.
2. Restrained Dilated Convolution: By employing dilated convolution layers selectively in the self-attention mechanism rather than more computationally intensive up-convolution blocks, FreeScale effectively increases the receptive field without introducing excessive noise or degradation.
3. Scale Fusion: The most pivotal aspect of the FreeScale methodology, this component fuses information from different scale levels, harmonizing high-frequency local details with low-frequency global semantics, thus eliminating repetitive artifacts and ensuring detailed and coherent high-resolution outputs.

Empirical Evaluation and Results

In extensive quantitative and qualitative evaluations, FreeScale outperformed existing tuning-free strategies such as ScaleCrafter and DemoFusion, particularly in maintaining the visual fidelity and consistency of upscaled outputs. Experimental results demonstrate that, in terms of Frechet Image Distance (FID) and Kernel Image Distance (KID), FreeScale achieves superior scores at resolutions up to 8192 × 8192—an unprecedented feat using a single GPU. Furthermore, empirical experiments in text-to-image and text-to-video generation tasks verify FreeScale's efficiency, showing significant improvements in visual quality and inference time.

Implications and Future Directions

The introduction of FreeScale's tuning-free paradigm implies significant theoretical and practical advancements. Theoretically, it sets a precedent in neural network design by illustrating how scale fusion can effectively bypass the computational and data scarcity bottlenecks currently impeding high-resolution synthesis in visual diffusion models. The tuning-free nature reduces dependency on extensive retraining or additional tuning processes, making it a versatile tool for various generative tasks.

Looking ahead, future research could explore the integration of these methodologies with transformer-based LDMs, further enhancing fidelity and enabling more robust generalization to diverse generation tasks. Additionally, advancements in multi-GPU optimization might offer opportunities to extend these techniques for even higher resolutions beyond current hardware constraints.

Finally, the FreeScale framework, while robust, heavily relies on the capabilities of the underlying base models. Thus, improvements in base model architectures will naturally augment FreeScale's performance, solidifying its stance as a pivotal tool in the ongoing evolution of high-resolution generative modeling.