Smooth and Sparse Optimal Transport (1710.06276v2)

Abstract: Entropic regularization is quickly emerging as a new standard in optimal transport (OT). It enables to cast the OT computation as a differentiable and unconstrained convex optimization problem, which can be efficiently solved using the Sinkhorn algorithm. However, entropy keeps the transportation plan strictly positive and therefore completely dense, unlike unregularized OT. This lack of sparsity can be problematic in applications where the transportation plan itself is of interest. In this paper, we explore regularizing the primal and dual OT formulations with a strongly convex term, which corresponds to relaxing the dual and primal constraints with smooth approximations. We show how to incorporate squared $2$-norm and group lasso regularizations within that framework, leading to sparse and group-sparse transportation plans. On the theoretical side, we bound the approximation error introduced by regularizing the primal and dual formulations. Our results suggest that, for the regularized primal, the approximation error can often be smaller with squared $2$-norm than with entropic regularization. We showcase our proposed framework on the task of color transfer.

• The paper proposes novel regularization methods for optimal transport that achieve computational smoothness while ensuring sparse transportation plans, addressing the dense output issue of entropic regularization.
• These methods integrate strongly convex terms into primal and dual formulations, yielding smooth relaxed/semi-dual and relaxed/semi-relaxed primal problems solvable with gradient-based optimization.
• Theoretical analysis suggests squared 2-norm regularization yields smaller approximation errors than entropic regularization, empirically validated through color transfer experiments showing efficient, sparse, and visually appealing results.

Smooth and Sparse Optimal Transport: An Analytical Overview

In the domain of optimal transport (OT), entropic regularization has gained significant traction due to its ability to recast OT computations into differentiable, unconstrained convex optimization problems. This transformation facilitates the use of efficient algorithms, such as Sinkhorn iteration, which dramatically reduce computational complexity. However, entropic regularization inherently maintains a dense transportation plan, which can be undesirable in applications where sparsity is crucial. This paper addresses this limitation by proposing alternative regularization schemes that promote sparsity while benefiting from the smoothness required for efficient computation.

Problem Setting and Motivation

Optimal transport distances serve as a versatile tool for comparing probability distributions, finding applications in various machine learning tasks. Typically, OT computations are expensive, as they involve solving linear programs, with complexity often scaling super-cubically with the size of the data sets. Entropic regularization offers a solution by converting this problem into a smoother format, yet it fails to deliver sparse transportation plans. Sparsity is often desirable for interpretability and simplicity in applications like color transfer, domain adaptation, and ecological inference.

Contributions and Methodology

The authors focus on integrating strongly convex regularization terms into both the primal and dual formulations of optimal transport. Specifically, they explore smooth approximations of the primal and dual constraints through squared $2$-norm and group-lasso regularizations. These approaches yield the following theoretical contributions:

1. Smooth Relaxed Dual Formulation: By regularizing the primal problem with a strongly convex term, the paper derives a smooth version of the dual formulation. This approach allows for sparse solutions and maintains the convex nature of the optimization problem.
2. Smoothed Semi-Dual Formulation: Extending the dual approach, the authors eliminate one variable, resulting in a semi-dual that's both smooth and conducive to sparsity. The leverage here is in the smoothed max operator.
3. Relaxed and Semi-Relaxed Primal Formulations: The dual exploration extends to relaxing one or both of the primal marginal constraints. This technique further supports sparse transportation plans by associating squared Euclidean distance approximations.
4. Algorithmic Development: The paper details how these formulations can be effectively solved using gradient-based optimization methods, illustrating the feasibility of these approaches with closed-form solutions for specific regularizers.

Theoretical Insights and Empirical Evaluation

The theoretical section bounds the approximation error introduced by the regularization, comparing squared $2$-norm with entropic regularization. The analysis suggests that squared $2$-norm regularization often incurs smaller approximation errors, thus potentially offering a tighter estimate for optimal transport distances.

Empirical validation focuses on the task of color transfer. The methods were shown to yield improvements over unregularized OT, providing visually appealing results with high sparsity. Notably, the squared $2$-norm regularization delivered sparse plans efficiently, contrasting with the dense outputs of entropic regularization.

Future Directions

The paper points to several potential avenues for future investigation:

• Further empirical paper on large-scale problems could unveil practical scalability issues and optimization algorithms beyond L-BFGS.
• The regularization frameworks proposed may be extended and applied to more complex OT-based problems, such as multi-modal fusion.
• The integration of these techniques in broader machine learning pipelines, exploring the trade-off between precision in approximation and computational efficiency.

Conclusion

"Smooth and Sparse Optimal Transport" offers an invaluable contribution by bridging smooth computational efficiency with the interpretability and utility of sparse solutions. This work presents a balanced approach to optimal transport, enhancing its applicability across various domains in artificial intelligence and machine learning. By prioritizing sparsity without sacrificing the benefits of regularization, the paper establishes a framework that paves the way for more interpretable and computationally feasible applications in data-driven fields.