Dist-GAN: An Improved GAN using Distance Constraints (1803.08887v3)

Abstract: We introduce effective training algorithms for Generative Adversarial Networks (GAN) to alleviate mode collapse and gradient vanishing. In our system, we constrain the generator by an Autoencoder (AE). We propose a formulation to consider the reconstructed samples from AE as "real" samples for the discriminator. This couples the convergence of the AE with that of the discriminator, effectively slowing down the convergence of discriminator and reducing gradient vanishing. Importantly, we propose two novel distance constraints to improve the generator. First, we propose a latent-data distance constraint to enforce compatibility between the latent sample distances and the corresponding data sample distances. We use this constraint to explicitly prevent the generator from mode collapse. Second, we propose a discriminator-score distance constraint to align the distribution of the generated samples with that of the real samples through the discriminator score. We use this constraint to guide the generator to synthesize samples that resemble the real ones. Our proposed GAN using these distance constraints, namely Dist-GAN, can achieve better results than state-of-the-art methods across benchmark datasets: synthetic, MNIST, MNIST-1K, CelebA, CIFAR-10 and STL-10 datasets. Our code is published here (https://github.com/tntrung/gan) for research.

Overview of Dist-GAN: An Improved GAN using Distance Constraints

The paper introduces a novel method to enhance the training of Generative Adversarial Networks (GANs) using a framework termed Dist-GAN. This approach addresses notable challenges in GAN training, specifically mode collapse and gradient vanishing, which are pivotal issues impacting the effectiveness of GANs. The authors present a detailed architecture that integrates autoencoders (AEs) and imposes distance constraints to maintain the stability and performance of the network.

The key innovation of Dist-GAN lies in the application of distance constraints incorporated into the GAN framework. These constraints play a critical role in aligning the distributions of latent variables and generated outputs, addressing mode collapse by enforcing diversity in the generated samples. The main components include the latent-data distance constraint and the discriminator-score distance constraint. The former ensures compatibility between distances in the latent space and their corresponding data space, preventing the generator from producing overly similar samples. The latter aligns the distribution of generated samples with real samples using the discriminator's scores as a guide, effectively reducing mode collapse instances.

The research also explores the interconnected training mechanism between the autoencoder and the GAN. The autoencoder's role stabilizes the training process and refrains the discriminator from fast convergence, thereby addressing gradient vanishing. The authors propose treating reconstructed samples from the autoencoder as 'real' in the discriminator's training, coupling the convergence of the encoder-decoder pair with the discriminator's learning process. This innovative coupling inherently slows the discriminator's convergence rate, providing the generator with more informative gradients for improved training.

From an empirical perspective, the authors benchmarked Dist-GAN against several state-of-the-art GAN variants, including DCGAN and WGANGP, across diverse datasets such as MNIST, CelebA, CIFAR-10, and STL-10. The results consistently indicate that Dist-GAN substantially reduces mode collapse and achieves competitive, if not superior, performance as indicated by better FID (Frechet Inception Distance) scores. Importantly, on challenging datasets like MNIST-1K, the proposed method outperforms existing solutions in both covered modes and mode balance, signifying its robustness in diverse generative scenarios.

Theoretical implications of this paper suggest a paradigm shift in GAN training methodologies, specifically the role of autoencoders and systematic distance constraints in enforcing diversity and stability. By integrating these mechanisms, Dist-GAN advances the understanding of GAN optimization and prompts new avenues for research, particularly in refining adversarial training processes and exploring new applications in computer vision tasks.

Looking forward, it is plausible that enhancements and refinements in the design of such distance constraints, as well as a deeper understanding of the interplay between different components of GANs, will continue to propel the capabilities of generative models. This paper provides a pathway not only for future research in GAN architectures but also for broader AI applications where generative models are fundamental.