StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2202.00273v2)

Abstract: Computer graphics has experienced a recent surge of data-centric approaches for photorealistic and controllable content creation. StyleGAN in particular sets new standards for generative modeling regarding image quality and controllability. However, StyleGAN's performance severely degrades on large unstructured datasets such as ImageNet. StyleGAN was designed for controllability; hence, prior works suspect its restrictive design to be unsuitable for diverse datasets. In contrast, we find the main limiting factor to be the current training strategy. Following the recently introduced Projected GAN paradigm, we leverage powerful neural network priors and a progressive growing strategy to successfully train the latest StyleGAN3 generator on ImageNet. Our final model, StyleGAN-XL, sets a new state-of-the-art on large-scale image synthesis and is the first to generate images at a resolution of $1024^2$ at such a dataset scale. We demonstrate that this model can invert and edit images beyond the narrow domain of portraits or specific object classes.

Overview of "StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets"

The paper "StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets" presents an enhancement of the StyleGAN architecture, referred to as StyleGAN-XL, which aims to address the challenges of scaling to large and diverse datasets such as ImageNet. The researchers propose various modifications to the StyleGAN structure and training methodologies, leveraging the strengths of projected generative adversarial networks (GAN) and progressive growing techniques. This approach is demonstrated to achieve impressive results, setting a new benchmark for large-scale image synthesis at high resolutions.

Key Contributions and Techniques

1. Progressive Growing for Stable Training: The authors reintroduce progressive growing, an approach initially discarded in previous StyleGAN versions due to contributing to aliasing and texture sticking. By utilizing carefully designed anti-aliasing measures, the authors manage to alleviate these issues, enabling quicker and more stable training across various resolutions.
2. Projected GAN Framework: StyleGAN-XL adapts the Projected GAN paradigm, which involves projecting images into feature spaces defined by well-established neural networks before feeding them into discriminators. This shift results in increased training stability, efficiency, and effectiveness over traditional GAN setups, particularly when paired with conditional generation.
3. Inclusion of Classifier Guidance: Inspired by diffusion models, classifier guidance is incorporated into the GAN training process. This aspect enhances class-conditional generation performance by integrating gradients from a pre-trained classifier, effectively guiding the generator towards higher fidelity image synthesis.
4. Efficient Architecture and Training Strategy: The architecture incorporates features like translational equivariance from StyleGAN3, combined with a progressively growing framework and pretrained class embeddings. The model also optimizes layer configurations and introduces a smaller latent space for improved learning efficiency and better alignment with the intrinsic dimensions of datasets.
5. Enhanced Inversion and Editing Capabilities: StyleGAN-XL improves upon image inversion and editing tasks by implementing techniques like Pivotal Tuning Inversion (PTI), which allows more accurate and semantically meaningful inversions. The model supports style mixing and extrapolation, further enhancing its utility for image manipulation applications.

Numerical Results and Evaluation

StyleGAN-XL showcases remarkable performance across various metrics, including Frechet Inception Distance (FID) and Inception Score (IS), surpassing competitive GAN and diffusion models across resolutions, up to $1024^2$ pixels. It notably reduces the variance of sample quality compared to existing state-of-the-art models like BigGAN and ADM, and delivers significant speed improvements regarding training and inference time.

Theoretical and Practical Implications

The successful enhancement of StyleGAN to large and diverse datasets provides invaluable insights for future GAN-based generative modeling. By demonstrating superior image quality and model flexibility, this work reinforces the practical utility of GANs in diverse applications, such as high-fidelity content creation, data augmentation, and scientific simulations.

Future Directions

The research opens several avenues for continued exploration, such as:

• Optimizing the trade-offs between computational overhead and model performance for broader accessibility.
• Developing advanced editing techniques leveraging the StyleGAN-XL architecture.
• Applying the methodology to even larger datasets that are currently beyond the capabilities of existing models due to their scale and diversity.

In conclusion, StyleGAN-XL represents a significant stride in generative modeling, presenting a robust approach to achieve state-of-the-art performance on large-scale, diverse datasets, while paving the way for future research and practical innovations in the field of image synthesis.