Regularization via Mass Transportation (1710.10016v3)

Abstract: The goal of regression and classification methods in supervised learning is to minimize the empirical risk, that is, the expectation of some loss function quantifying the prediction error under the empirical distribution. When facing scarce training data, overfitting is typically mitigated by adding regularization terms to the objective that penalize hypothesis complexity. In this paper we introduce new regularization techniques using ideas from distributionally robust optimization, and we give new probabilistic interpretations to existing techniques. Specifically, we propose to minimize the worst-case expected loss, where the worst case is taken over the ball of all (continuous or discrete) distributions that have a bounded transportation distance from the (discrete) empirical distribution. By choosing the radius of this ball judiciously, we can guarantee that the worst-case expected loss provides an upper confidence bound on the loss on test data, thus offering new generalization bounds. We prove that the resulting regularized learning problems are tractable and can be tractably kernelized for many popular loss functions. We validate our theoretical out-of-sample guarantees through simulated and empirical experiments.

• The paper presents a novel DRO regularization approach that minimizes worst-case expected loss over a Wasserstein ball of probability distributions.
• The paper demystifies classical regularization methods such as Tikhonov and Lasso by grounding them in optimal transport and robust probabilistic interpretations.
• The paper establishes tractability results and new generalization bounds while enabling stress testing via worst-case distribution construction.

Regularization via Mass Transportation: An Overview

The paper entitled "Regularization via Mass Transportation" presents an innovative approach to regularization in machine learning, focusing on the use of distributionally robust optimization (DRO) techniques grounded in optimal transport theory. The authors introduce a novel framework that leverages the Wasserstein distance to tackle the challenges of overfitting in scenarios with limited data. This approach provides a new lens through which established regularization methodologies can be understood, while simultaneously expanding the toolkit available for regression and classification tasks.

The central idea hinges on the concept of minimizing the worst-case expected loss over a set of probability distributions defined by a bounded transportation distance from the empirical distribution. This deviation from the conventional approach of adding explicit regularization terms to the hypothesis complexity marks a significant theoretical departure. The set of distributions considered, referred to as a Wasserstein ball, allows the model to account for distributional uncertainty in a principled manner.

Key Contributions

1. Tractability and Kernelization: The authors offer proof of the tractability of the proposed distributionally robust learning problems when using common loss functions and linear hypothesis spaces. They extend these findings to nonlinear spaces via kernel methods, which is particularly beneficial for support vector machines and other kernelized learning paradigms.
2. Probabilistic Interpretation of Regularization: Through this proposed framework, traditional regularization schemes such as Tikhonov and Lasso are shown to emerge as special cases, thus demystifying these methods by providing them with a robust probabilistic foundation based on the geometry of Wasserstein balls.
3. Generalization Bounds: The paper provides novel generalization bounds that do not depend on the complexity of the hypothesis class, thereby opening new avenues for theoretical analysis in spaces with potentially infinite VC-dimensions.
4. Robust and Distributionally Robust Equivalence: The authors demonstrate that their distributionally robust models coincide with classical robust optimization approaches under certain conditions in both regression and classification, thus bridging a gap between robust optimization and regularization.
5. Error and Risk Estimation: The methodology also extends to provide confidence intervals for prediction errors and classification risks, offering practical tools for model evaluation in uncertain environments.
6. Worst-Case Distribution Construction: The paper concludes with methods to compute worst-case distributions, enabling practitioners to perform stress testing and scenario analysis effectively.

Implications and Future Directions

This paper is a rich resource for researchers and practitioners in the field of machine learning and optimization. By framing regularization as a distributionally robust optimization problem, it not only provides a new perspective on how regularization works but also ameliorates some limitations of classical methods, prominently addressing issues related to overfitting and distributional shifts.

The practical implications of this research are significant. The tractability results pave the way for more efficient computation in large-scale applications, and the generalization bounds offer valuable theoretical guarantees that are essential for deploying models in real-world settings with high stakes.

Looking forward, this framework invites exploration into more complex hypothesis spaces, including deep neural networks, where interplay between the model's expressiveness and distributional robustness could yield impactful insights. Additionally, the scalability implications of the Wasserstein-based regularization could lead to advancements in streaming data and online learning contexts, where traditional methods struggle due to dynamic data distributions.

Overall, the paper "Regularization via Mass Transportation" contributes substantially to both the theoretical foundations and practical methodologies in robust machine learning, laying the groundwork for future innovations in this domain.