On the regularization of Wasserstein GANs (1709.08894v2)

Abstract: Since their invention, generative adversarial networks (GANs) have become a popular approach for learning to model a distribution of real (unlabeled) data. Convergence problems during training are overcome by Wasserstein GANs which minimize the distance between the model and the empirical distribution in terms of a different metric, but thereby introduce a Lipschitz constraint into the optimization problem. A simple way to enforce the Lipschitz constraint on the class of functions, which can be modeled by the neural network, is weight clipping. It was proposed that training can be improved by instead augmenting the loss by a regularization term that penalizes the deviation of the gradient of the critic (as a function of the network's input) from one. We present theoretical arguments why using a weaker regularization term enforcing the Lipschitz constraint is preferable. These arguments are supported by experimental results on toy data sets.

• The paper proposes a one-sided gradient penalty that relaxes the Lipschitz constraint, improving training stability for WGANs.
• It critically examines the limitations of the existing gradient penalty technique and supports its claims with empirical evidence on various datasets.
• The study bridges optimal transport theory with GAN training, suggesting practical improvements for generative modeling tasks.

The Regularization of Wasserstein GANs

The paper "On the regularization of Wasserstein GANs" addresses an important aspect of generative adversarial networks (GANs), specifically focusing on the regularization necessary for stabilizing the training process of Wasserstein GANs (WGANs). WGANs have gained prominence due to their ability to mitigate some of the convergence issues inherent in traditional GANs. This paper provides a critical examination of the regularization mechanisms used in WGANs and proposes an alternative that the authors argue leads to more stable training and better performance.

Key Contributions

The authors present several key contributions:

1. Review of Current Regularization Techniques:
   • The paper reviews the regularization technique proposed in the Improved WGAN by augmenting the loss function with a gradient penalty (GP). This penalty enforces the Lipschitz constraint necessary for the theoretical foundation of WGANs.
   • The authors critically analyze the assumptions of differentiability and sampling from a joint distribution underlying the GP regularization strategy, suggesting these assumptions might not hold in practice and could impair training.
2. Proposal of a New Regularization Term:
   • The paper introduces a less restrictive regularization term, termed the one-sided gradient penalty (LP). This term penalizes instances where the gradient norm exceeds one, which the authors argue is a more relaxed and empirically effective constraint for enforcing Lipschitz continuity.
   • The authors support their theoretical proposal with empirical evidence, showing improved performance across various datasets when using this weaker penalty.
3. Discussion on Lipschitz Constraint and Optimal Transport:
   • The authors explore the mathematical basis of WGANs rooted in optimal transport theory and the Kantorovich-Rubinstein duality. They emphasize that the 1-Lipschitz constraint, critical for the WGAN objective, could be more judiciously enforced through their proposed penalty.
4. Empirical Results:
   • The empirical section provides compelling evidence, showing that the proposed LP penalty leads to more stable and less sensitive training compared to the GP term, across synthetic datasets like 8Gaussians and Swiss Roll, and more complex datasets like CIFAR-10.

Implications

The implications of this research are twofold. Practically, the introduction of a less restrictive regularization term offers a viable alternative for improving the stability and performance of WGANs in generative modeling tasks. For models like Cramer GANs, which incorporate the gradient penalty, the LP penalty could offer potential improvements. Theoretically, this work bridges aspects of optimal transport theory with GAN training, pushing forward the understanding of how mathematical constraints can be effectively applied in machine learning models.

Future Directions

The authors’ proposed changes to the regularization of WGANs prompt several directions for future research, such as:

• Extending the LP regularization technique to other GAN frameworks and verifying its effectiveness across different domains.
• Further exploring the implications of weaker regularization terms in the context of high-dimensional and complex data distributions.
• Investigating the interplay between different formulations of Lipschitz constraints leveraging other metrics within the optimal transport theory.

In conclusion, the paper offers a rigorous analysis and proposed improvement in the regularization of Wasserstein GANs by suggesting a one-sided penalty approaches. This not only enhances the WGAN framework but also contributes to broader discussions on stabilizing GAN training methodologies. The implications of this work are relevant to both practitioners and theorists interested in the continuing evolution of generative adversarial networks.