The relativistic discriminator: a key element missing from standard GAN (1807.00734v3)

Abstract: In standard generative adversarial network (SGAN), the discriminator estimates the probability that the input data is real. The generator is trained to increase the probability that fake data is real. We argue that it should also simultaneously decrease the probability that real data is real because 1) this would account for a priori knowledge that half of the data in the mini-batch is fake, 2) this would be observed with divergence minimization, and 3) in optimal settings, SGAN would be equivalent to integral probability metric (IPM) GANs. We show that this property can be induced by using a relativistic discriminator which estimate the probability that the given real data is more realistic than a randomly sampled fake data. We also present a variant in which the discriminator estimate the probability that the given real data is more realistic than fake data, on average. We generalize both approaches to non-standard GAN loss functions and we refer to them respectively as Relativistic GANs (RGANs) and Relativistic average GANs (RaGANs). We show that IPM-based GANs are a subset of RGANs which use the identity function. Empirically, we observe that 1) RGANs and RaGANs are significantly more stable and generate higher quality data samples than their non-relativistic counterparts, 2) Standard RaGAN with gradient penalty generate data of better quality than WGAN-GP while only requiring a single discriminator update per generator update (reducing the time taken for reaching the state-of-the-art by 400%), and 3) RaGANs are able to generate plausible high resolutions images (256x256) from a very small sample (N=2011), while GAN and LSGAN cannot; these images are of significantly better quality than the ones generated by WGAN-GP and SGAN with spectral normalization.

• The paper introduces a relativistic discriminator that improves GAN training stability by comparing real and fake data simultaneously.
• The methodology extends to RGAN and RaGAN variants, demonstrating lower FID scores and consistent high-resolution image generation.
• Empirical results show enhanced performance and efficiency, highlighting the potential for advanced GAN stabilization techniques.

The Relativistic Discriminator: A Key Element Missing from Standard GANs

The paper "The relativistic discriminator: a key element missing from standard GAN" by Alexia Jolicoeur-Martineau introduces a novel approach to improving the stability and performance of Generative Adversarial Networks (GANs) through the introduction of a relativistic discriminator. This essay provides an in-depth overview of the concepts, methodologies, and empirical results presented in the paper.

Introduction to GANs and Their Challenges

Generative Adversarial Networks (GANs) consist of two neural networks: the generator $G$ and the discriminator $D$, which engage in a two-player minimax game. The generator aims to produce realistic fake data that can fool the discriminator, while the discriminator tries to distinguish between real and fake data. The standard GAN (SGAN) setting, in which the discriminator estimates the probability of the input data being real, has shown significant promise but suffers from stability issues during training.

Key Insight: Relativistic Discriminator

SGAN's major shortcoming, as argued in the paper, is its inability to consider the relative realism of real and fake data simultaneously. The proposed solution is the introduction of a "relativistic discriminator," which rather than estimating the probability of an individual data point being real, estimates the probability that a real data point is more realistic than a fake data point. This approach leverages the inherent balance between real and fake data in each training batch.

Methodology

The paper formalizes the relativistic discriminator using two main variants:

1. Relativistic GAN (RGAN): The discriminator is defined such that it evaluates the probability that real data is more realistic than randomly sampled fake data.
2. Relativistic Average GAN (RaGAN): The discriminator assesses whether a real data point is more realistic than the average realism of fake data points in the batch.

These formulations are shown to extend seamlessly to different types of GANs, including those utilizing non-standard loss functions. Furthermore, RGANs provide an equivalency to Integral Probability Metrics (IPM)-based GANs, useful for understanding their stability compared to SGAN.

Empirical Results

Empirical studies demonstrate that the relativistic discriminator significantly enhances the performance and stability of GANs across various datasets and settings. Key findings include:

1. CIFAR-10 Dataset:
   • Relativistic GANs (RaGANs) maintain more consistent and lower Fréchet Inception Distance (FID) compared to their non-relativistic counterparts, indicating higher quality generated samples.
   • RaSGAN with gradient penalty achieves notably superior data quality with one discriminator update per generator update, significantly reducing computational costs.
2. High-Resolution Images:
   • For challenging datasets like the CAT dataset, relativistic GANs produce better and more stable outputs. Specifically, RaSGANs are capable of generating plausible high-resolution images (256x256) while non-relativistic GANs like SGAN and LSGAN struggle or fail to converge.

Theoretical Implications

The introduction of the relativistic discriminator has multiple theoretical implications:

• Divergence Minimization: Aligns the training dynamics of GANs closer to divergence minimization techniques, enhancing stability.
• Gradient Dynamics: By ensuring that the generator indirectly influences the probability of real data points being real, relativistic discriminators inherently incorporate real data into the learning dynamics, preventing the discriminator from focusing exclusively on fake data.

Future Directions

The paper suggests several future directions to explore:

• A deeper mathematical analysis of the implications of relativism in GANs.
• Extensive empirical validation across diverse datasets and configurations to identify the most effective relativistic GAN variant.
• Combining relativistic discriminators with other stabilization techniques like spectral normalization and gradient penalty to push the boundaries of state-of-the-art GAN performance.

Conclusion

The introduction of the relativistic discriminator addresses a fundamental limitation in standard GANs, offering a robust and theoretically sound improvement in generating high-quality data. By considering the relative realism of real and fake data simultaneously, this approach stabilizes training dynamics and improves the quality of generated samples, marking a significant contribution to the field of generative modeling.