PaGoDA: Progressive Growing of a One-Step Generator from a Low-Resolution Diffusion Teacher (2405.14822v2)

Abstract: The diffusion model performs remarkable in generating high-dimensional content but is computationally intensive, especially during training. We propose Progressive Growing of Diffusion Autoencoder (PaGoDA), a novel pipeline that reduces the training costs through three stages: training diffusion on downsampled data, distilling the pretrained diffusion, and progressive super-resolution. With the proposed pipeline, PaGoDA achieves a $64\times$ reduced cost in training its diffusion model on 8x downsampled data; while at the inference, with the single-step, it performs state-of-the-art on ImageNet across all resolutions from 64x64 to 512x512, and text-to-image. PaGoDA's pipeline can be applied directly in the latent space, adding compression alongside the pre-trained autoencoder in Latent Diffusion Models (e.g., Stable Diffusion). The code is available at https://github.com/sony/pagoda.

• The paper presents a novel one-step generator that progressively grows image resolution, achieving state-of-the-art FID scores and 2x faster performance.
• It employs a pre-trained diffusion model as a frozen encoder with a decoder that incrementally upsamples images from 64x64 to 512x512 pixels.
• The approach enables scalable, cost-effective training for high-resolution image synthesis with practical applications in inpainting and controllable generation.

Progressive Growing of Diffusion Autoencoder (PaGoDA)

Introduction

When it comes to generating high-resolution images, diffusion models (DMs) are quite powerful but inherently slow because they generate images via an incremental denoising process. This process is akin to solving a complex differential equation. To speed things up, researchers have worked on distilling these models into faster generators. However, these distilled models have typically been constrained by the resolution limits of their original DMs. Enter PaGoDA. This method leverages a technique for progressively growing the resolution of these generators, allowing for scalable, high-quality image generation without the heavy computational burden of training new high-resolution DMs.

Key Components

Pre-Trained Encoder

PaGoDA makes use of a pre-trained diffusion model that acts as a frozen encoder. This model only works at a base resolution, say $64 \times 64$ pixels, and converts high-resolution input images to a structured latent space by solving a specific kind of differential equation known as the PF-ODE.

Progressively Growing Decoder

The magic happens with the decoder. Instead of generating high-resolution images in one go, PaGoDA grows the resolution of the decoder incrementally. What this means is that the decoder starts generating images at a lower resolution, gradually adding new layers to increase the resolution step-by-step. This approach drastically reduces training costs and makes upsampling to higher resolutions more efficient.

Results

Speed and Quality

In experiments, PaGoDA was used to upsample images from a base resolution of $64 \times 64$ pixels to $512 \times 512$ pixels. Impressively, it achieved this in half the time as compared to existing single-step distillation methods like Stable Diffusion. Moreover, PaGoDA set new state-of-the-art scores in Fréchet Inception Distance (FID) across all resolutions—from $64 \times 64$ to $512 \times 512$ pixels—when tested on the ImageNet dataset.

Here’s a quick summary of resulting performance:

• Speed: 2x faster inference compared to previous methods.
• Quality: State-of-the-art FID scores for image generation.

Numerical Results

To give you a clearer picture, here are some numerical highlights:

• PaGoDA achieved a FID of 1.21 in generating $64 \times 64$ pixels, surpassing older models like StyleGAN-XL (1.51).
• For $512 \times 512$ resolution images, PaGoDA managed a FID of 1.80, again outperforming other models in its class.

Practical Implications

Faster and Scalable Training

The progressive growth strategy means that as new, higher resolutions are required, one doesn't need to re-train the teacher-student models from scratch. This makes the entire training pipeline much more efficient and cost-effective.

Versatile Applications

PaGoDA isn't just fast; it's versatile. It has shown remarkable efficacy in tasks like solving inverse problems and enabling controllable image generation. For instance, the method allows you to accurately fill in missing parts of images (inpainting) or even alter image classes while keeping the core structure intact.

Speculations for Future

As AI continues to evolve, the implications of methods like PaGoDA are significant. Progressive growing mechanisms could become more mainstream, especially in fields requiring real-time, high-resolution image generation like gaming, film, and medical imaging. Additionally, the application of these methods in solving complex inverse problems can revolutionize areas such as remote sensing and robotic vision, making models more efficient and accurate.

Conclusion

PaGoDA is a significant advancement in the field of generative models. By cleverly utilizing pre-trained DMs for encoding and progressively growing the resolution of the decoder, it achieves high-quality, high-resolution image generation efficiently and quickly. These advancements suggest new avenues for further research and applications, making scalable and fast image generation more accessible than ever.