Near-linear time approximation algorithms for optimal transport via Sinkhorn iteration (1705.09634v2)

Abstract: Computing optimal transport distances such as the earth mover's distance is a fundamental problem in machine learning, statistics, and computer vision. Despite the recent introduction of several algorithms with good empirical performance, it is unknown whether general optimal transport distances can be approximated in near-linear time. This paper demonstrates that this ambitious goal is in fact achieved by Cuturi's Sinkhorn Distances. This result relies on a new analysis of Sinkhorn iteration, which also directly suggests a new greedy coordinate descent algorithm, Greenkhorn, with the same theoretical guarantees. Numerical simulations illustrate that Greenkhorn significantly outperforms the classical Sinkhorn algorithm in practice.

• The paper presents a near-linear time approximation method for optimal transport by refining Sinkhorn iteration techniques.
• It introduces the Greenkhorn algorithm, which selects the most significant updates to enhance computational performance.
• Empirical evaluations on synthetic and real datasets demonstrate improved speed and accuracy, broadening the scope of OT applications.

Near-Linear Time Approximation Algorithms for Optimal Transport via Sinkhorn Iteration

The paper presented by Altschuler, Weed, and Rigollet investigates the computational challenges of determining optimal transport (OT) distances, crucial in fields such as machine learning, statistics, and computer vision. Specifically, the paper focuses on approximating these distances in near-linear time through the application of Sinkhorn iteration techniques.

Research Context and Contributions

Optimal transport distances provide a powerful framework to capture the geometrical aspects of data distributions, but their computational intensity has historically limited their application to large-scale problems. Prior advancements, such as Cuturi's introduction of Sinkhorn Distances, demonstrated improved computation efficiencies but lacked rigorous time complexity guarantees. This research addresses this gap by providing theoretical backing for near-linear time approximations of OT distances using Sinkhorn iterations.

The paper proposes a novel perspective on analyzing Sinkhorn iteration, establishing that, with specific parameter tuning, the resulting algorithms operate within near-linear time bounds. It also introduces a new coordinate descent algorithm, termed $Greenkhorn$, which enhances the practical performance of classic Sinkhorn iterations while maintaining identical theoretical guarantees.

Methodological Insights

The core innovation centers around the entropic regularization of the OT problem. The algorithm leverages a refined analysis of the Sinkhorn algorithm, demonstrating that it can achieve approximation accuracy in $O((')^{-2} \log (s/\ell))$ iterations, where $s$ and $\ell$ denote the sum and minimum of matrix entries, respectively.

Additionally, the research introduces $Greenkhorn$, an adaptation that selects updates via a greedy approach, optimizing only the most significant row or column at each step, thereby improving computational efficiency in practice. This distinct difference highlights its superior performance over the classic Sinkhorn method, particularly in processing sparse data typical in practical scenarios.

Experimental Framework

The empirical evaluation comprises experiments using both synthetic and real datasets. The results consistently show $Greenkhorn$ as outperforming Sinkhorn in reducing computation time while maintaining or enhancing approximation quality. Particularly noteworthy is the algorithm's efficacy in sparse representation scenarios, aligning with real-world data characteristics.

Theoretical and Practical Implications

The theoretical contributions confirm that OT distances can be computed in near-linear time even for general cost matrices, broadening the potential for applying OT-based methods to more extensive data sets. The empirical success hints at future algorithmic developments that could further streamline computations in high-dimensional and dynamic data environments.

The paper paves the way for future research to refine the parameter settings for specific applications, further enhancing algorithmic efficiency. Additionally, exploration into hybrid methods that combine elements of neural computation and OT distances could offer new perspectives in both the AI and statistics communities.

Conclusion

This work marks a significant milestone in the computational management of optimal transport distances, providing both theoretical and empirical evidence for near-linear time performance. By extending the utility of OT distances to broader and more challenging applications, it sets the foundations for ongoing research and practical implementations across diverse domains requiring sophisticated data analysis tools.