Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspective

Abstract: Training generative adversarial networks (GANs) with limited real image data generally results in deteriorated performance and collapsed models. To conquer this challenge, we are inspired by the latest observation, that one can discover independently trainable and highly sparse subnetworks (a.k.a., lottery tickets) from GANs. Treating this as an inductive prior, we suggest a brand-new angle towards data-efficient GAN training: by first identifying the lottery ticket from the original GAN using the small training set of real images; and then focusing on training that sparse subnetwork by re-using the same set. We find our coordinated framework to offer orthogonal gains to existing real image data augmentation methods, and we additionally present a new feature-level augmentation that can be applied together with them. Comprehensive experiments endorse the effectiveness of our proposed framework, across various GAN architectures (SNGAN, BigGAN, and StyleGAN-V2) and diverse datasets (CIFAR-10, CIFAR-100, Tiny-ImageNet, ImageNet, and multiple few-shot generation datasets). Codes are available at: https://github.com/VITA-Group/Ultra-Data-Efficient-GAN-Training.

• The paper adapts the Lottery Ticket Hypothesis to GANs, demonstrating that sparse subnetworks can be efficiently trained with limited real image data.
• It employs Iterative Magnitude Pruning alongside feature-level adversarial augmentation (AdvAug) to stabilize GAN training in data-scarce environments.
• Experimental results across architectures like SNGAN, BigGAN, and StyleGAN-V2 show significant improvements in FID and IS metrics without relying on extensive datasets.

An Expert Analysis of "Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspective"

The paper by Chen et al. proposes a novel approach to enhance the data-efficiency of Generative Adversarial Networks (GANs) training using the concept of "lottery tickets". The fundamental idea explored is how one can leverage the identification of sparse subnetworks—lottery tickets—within GANs to train these models more efficiently, especially when only limited real image data is available. This research is motivated by the observation that training GANs with insufficient data typically leads to overfitting and performance degradation.

Foundational Insights and Methodology

A significant contribution is the adaptation of the Lottery Ticket Hypothesis (LTH) to the paradigm of GANs in data-scarce environments. The LTH posits that within an overparameterized network, one can find sparse subnetworks that can be trained to achieve performance comparable to that of the original network. In this context, Chen et al. extend the LTH to the generative domain, which traditionally involves more complex min-max optimizations than standard tasks like classification.

The methodology involves two sequential steps: identifying sparse winning ticket subnetworks and subsequently optimizing these subnetworks, potentially augmented by techniques such as data-level and feature-level augmentations. The authors employ Iterative Magnitude Pruning (IMP) to achieve network sparsification, identifying subnetworks that require less data to perform effectively. The introduction of a robust feature-level adversarial augmentation (AdvAug) further enriches this method, providing a mechanism to stabilize training dynamics under the small data regime.

Experimental Framework and Results

The authors conduct extensive experiments across various GAN architectures, including SNGAN, BigGAN, and StyleGAN-V2, utilizing datasets such as CIFAR-10, CIFAR-100, Tiny-ImageNet, ImageNet, and a collection of few-shot datasets. The results consistently demonstrate that their proposed framework, augmented with AdvAug and data-level augmentations like DiffAug, achieves superior performance in data-scarce conditions. Notably, BigGAN tickets with high sparsity levels show substantial improvements in FID and IS metrics when trained with a fraction of the original data.

Moreover, the experiments also cover few-shot generation tasks, comparing their approach against existing transfer learning methods that require pre-training on related datasets. Despite the absence of such pre-training, the proposed method achieves competitive performance, underscoring its effectiveness in extremely limited data contexts.

Implications and Future Research Directions

This paper's contributions lie in both the theoretical extension of LTH to GANs and the practical enhancement of GAN training strategies for limited data scenarios. The implications are significant for fields where data is scarce or expensive to gather, such as medical imaging or rare species identification. By reducing the dependency on large datasets, the approach enhances GANs' applicability across diverse domains.

However, several future research directions emerge from this study. One promising avenue is the joint pursuit of data efficiency alongside computational efficiency, potentially by integrating the lottery ticket framework with other model compression techniques. Another path is exploring the structural characteristics of lottery tickets that enable them to maintain high performance in data-limited settings. Understanding these properties could inform the design of new architectures tailored for specific data constraints.

In summary, the paper by Chen et al. offers a substantial advancement in GAN training through sparsity-driven strategies and represents a meaningful step toward the broader goal of enhancing the data efficacy of neural networks.