Training Generative Adversarial Networks with Limited Data (2006.06676v2)

Abstract: Training generative adversarial networks (GAN) using too little data typically leads to discriminator overfitting, causing training to diverge. We propose an adaptive discriminator augmentation mechanism that significantly stabilizes training in limited data regimes. The approach does not require changes to loss functions or network architectures, and is applicable both when training from scratch and when fine-tuning an existing GAN on another dataset. We demonstrate, on several datasets, that good results are now possible using only a few thousand training images, often matching StyleGAN2 results with an order of magnitude fewer images. We expect this to open up new application domains for GANs. We also find that the widely used CIFAR-10 is, in fact, a limited data benchmark, and improve the record FID from 5.59 to 2.42.

Training Generative Adversarial Networks with Limited Data

The paper "Training Generative Adversarial Networks with Limited Data" by Tero Karras et al. addresses the challenge of training Generative Adversarial Networks (GANs) effectively when limited data is available. GANs, introduced by Goodfellow et al. (2014), have shown remarkable capabilities in generating high-quality images by learning from vast datasets. However, obtaining large datasets that meet specific application requirements, including constraints on subject type, image quality, geographical location, and privacy, remains a significant challenge. This paper proposes an adaptive discriminator augmentation (ADA) technique to train GANs with limited data while avoiding discriminator overfitting, thus ensuring stable training.

Methodology

The authors propose an ADA mechanism that augments the images shown to the discriminator, preventing discriminator overfitting without modifying the loss functions or network architectures. The augmentations are diverse, including geometric transformations, color adjustments, image-space filtering, additive noise, and cutout. Importantly, these augmentations are applied to ensure they do not leak into the generated images, maintaining the quality and integrity of the generated data.

The augmentation strength is controlled adaptively based on the degree of overfitting observed during training. Two heuristics are proposed to measure overfitting: one using a separate validation set and another without requiring one. The second heuristic leverages the discriminator outputs on real and generated images, adjusting the augmentation strength dynamically to maintain training stability.

Key Results

The paper demonstrates the effectiveness of ADA on several datasets, including FFHQ, LSUN Cat, CIFAR-10, and newly introduced datasets like MetFaces. The strong numerical results highlight the impact of ADA:

• FFHQ and LSUN Cat: ADA improves the Fréchet Inception Distance (FID) significantly compared to baseline models, especially in limited data scenarios (e.g., FFHQ-2k, FFHQ-10k). For instance, with FFHQ-2k, ADA reduces FID from 78.8 to 16.49.
• CIFAR-10: ADA achieves a new state-of-the-art FID of 2.42, down from the previous best of 5.59.
• MetFaces: ADA yields high-quality results even with only 1336 training images, surpassing previous methods in both FID and Kernel Inception Distance (KID).

The authors also observe that the CIFAR-10 benchmark is effectively a limited data scenario, with ADA offering substantial improvements over state-of-the-art methods.

Implications

The practical implications of this research are profound:

1. Broader Applicability of GANs: ADA enables high-quality GAN training with significantly less data, opening up new applications in fields where data acquisition is expensive or constrained, such as medical imaging, historical document digitization, and art preservation.
2. Efficiency: The technique provides a cost-effective way to generate quality results without the need for extensive data collection and annotation, reducing both time and resource investments.
3. Enhanced Model Robustness: By preventing discriminator overfitting, ADA ensures more stable and reliable GAN training, which is crucial for deployment in sensitive applications.

Theoretical Insights

From a theoretical perspective, ADA tackles the fundamental problem of discriminator overfitting in limited data regimes by maintaining the integrity of the augmentation process:

• Non-leaking Augmentations: The paper provides a thorough analysis of conditions under which augmentations do not leak into the generated images. The proposed augmentations are designed to be invertible, ensuring that the feedback provided to the generator remains informative despite the stochastic augmentations applied.
• Adaptive Control: The adaptive approach to controlling augmentation strength is innovative, dynamically adjusting to the training needs rather than relying on fixed parameters that may be suboptimal as the training progresses.

Future Developments

The paper paves the way for further research in applying GANs to limited data scenarios. Potential future developments include:

• Exploring Diverse Augmentations: Investigating other augmentation techniques that can be integrated into the ADA framework without causing leaks, such as semantic-level transformations.
• Transfer Learning Enhancements: Combining ADA with advanced transfer learning techniques for even more efficient training on small datasets.
• Extending to Other Models: Adapting ADA for use with other generative models, such as Variational Autoencoders (VAEs) and Flow-based models, to assess its generalizability.

In conclusion, the ADA technique proposed by Karras et al. is a significant contribution to the field of generative modeling, allowing high-quality GAN training with limited data. The adaptive mechanism ensures efficient and stable training, making generative models more accessible and applicable across various domains.