Entropy-Regularized Wasserstein Distance

• Entropy-Regularized Wasserstein Distance is a smooth, convex family of cost functions that interpolates between traditional Wasserstein metrics and information divergences like KL.
• It integrates an entropic term into the optimal transport problem, enabling scalable Sinkhorn iterations and improved statistical properties for high-dimensional data.
• Its information-geometric framework facilitates closed-form solutions for Gaussian measures and underpins robust applications in machine learning, shape optimization, and statistical inference.

The entropy-regularized Wasserstein distance is a one-parameter family of smooth, convex cost functions that interpolate between the classical Wasserstein distance from optimal transport theory and various information-theoretic divergences such as Kullback–Leibler (KL) divergence. This regularization introduces an entropic term—weighted by a positive parameter—into the optimal transport objective, leading to computational benefits, improved statistical properties, and a rich information-geometric structure. Entropy regularization has enabled scalable algorithms for OT problems, provided dimension-free sample complexity, facilitated the construction of barycenters, and opened avenues in machine learning, shape optimization, and statistical inference.

1. Formulation and Information-Geometric Structure

The entropy-regularized transportation problem augments the classical Kantorovich optimal transport objective between discrete (or continuous) probability distributions $p, q \in S_{n-1}$:

$$\min_{P \in \mathcal{U}(p, q)} \langle M, P \rangle - \lambda H(P),$$

with

$H(P) = -\sum_{i,j} P_{ij}\log P_{ij},$

where $M$ is a ground cost matrix and $\lambda > 0$ is the entropic regularization parameter (Amari et al., 2017, Bigot et al., 2017, Tong et al., 2020).

The optimal plan $P^*_\lambda$ is given in Gibbs (exponential family) form as: $$P^*_{\lambda, ij} \propto \exp \left( -\frac{m_{ij}}{\lambda} + \frac{1+\lambda}{\lambda} (\alpha_i + \beta_j) \right),$$ with Lagrange multipliers $(\alpha, \beta)$ enforcing the marginal constraints. This structure reveals that the family $\{P^*_\lambda\}$ forms an exponential family, interconnecting the geometry of optimal transport (Wasserstein) with that of KL divergence (information geometry) (Amari et al., 2017).

2. Divergence Families and the Cuturi/Sinkhorn Cost

The “Cuturi function,” or entropy-relaxed OT cost, is defined as: $$C_\lambda(p, q) = \frac{1}{1+\lambda}\langle M, P^*_\lambda\rangle - \frac{\lambda}{1+\lambda} H(P^*_\lambda),$$ which is convex in $(p, q)$. For $\lambda \to 0$, $C_\lambda$ converges to the classical Wasserstein cost; for $\lambda > 0$, $C_\lambda(p, p)$ is strictly positive due to entropic penalization (Amari et al., 2017, Bigot et al., 2017).

To address the bias $C_\lambda(p, p) \neq 0$, a “Sinkhorn divergence” (or debiased entropy-regularized Wasserstein distance) is introduced: $$S_\lambda(p, q) = C_\lambda(p, q) - \frac{1}{2}C_\lambda(p, p) - \frac{1}{2}C_\lambda(q, q),$$ yielding a genuine divergence metric which is zero if and only if $p = q$ (Bigot et al., 2017, Tong et al., 2020).

Moreover, the information-geometric structure allows for the construction of a family of $\lambda$-divergences: $$D_\lambda[p : q] = \gamma_\lambda \,\mathrm{KL}\left( P^*_\lambda(p, p) \parallel P^*_\lambda(p, q) \right)$$ with an appropriate scaling $\gamma_\lambda$. This yields a Bregman-type divergence that interpolates between KL and Wasserstein.

3. Computational and Statistical Advantages

The strict convexity and smoothness of the entropy-regularized problem admit a unique optimal plan that depends smoothly on the marginals, which can be computed efficiently using the Sinkhorn–Knopp matrix scaling algorithm. This iterative scaling procedure alternates between enforcing marginal constraints and has rapid convergence relative to standard OT solvers (Amari et al., 2017, Bigot et al., 2017, Motamed, 2020).

Key computational features include:

• Efficiency: Sinkhorn iterations reduce the computational complexity of OT from $O(n^3)$ to nearly $O(n^2)$ per iteration, and with appropriate low-rank or multi-scale methods, to as low as $O(n\log^3 n)$ (Motamed, 2020).
• Scalability: Domain decomposition (Bonafini et al., 2020), hierarchical approaches, and parallelization over subdomains render computations tractable for large-scale or high-dimensional data, e.g., images or point clouds.
• Statistical Regularity: The regularized distance is infinitely differentiable on the interior of the simplex and enables finite-sample central limit theorems and dimension-free convergence rates (Luise et al., 2018, Bigot et al., 2017, Mallery et al., 13 Jan 2025).

4. Analytical Properties and Closed-Form Solutions

For certain classes of distributions, especially Gaussian and $q$-normal measures, the entropy-regularized Wasserstein distance admits closed-form expressions. For $\mathcal{N}(\mu_1, \Sigma_1)$ and $\mathcal{N}(\mu_2, \Sigma_2)$:

$$C_\lambda(P, Q) = \|\mu_1 - \mu_2\|^2 + \mathrm{tr}(\Sigma_1 + \Sigma_2 - 2(\Sigma_1^{1/2} \Sigma_2 \Sigma_1^{1/2} + \lambda^2I)^{1/2}) + \text{entropic terms},$$

with the optimal coupling's cross-covariance modified by entropy (Tong et al., 2020, Mallasto et al., 2020, Quang, 2020, Acciaio et al., 25 Dec 2024). In infinite-dimensional settings, the entropic regularization ensures Fréchet differentiability (contrasted with the non-smoothness of unregularized Wasserstein), giving existence, uniqueness, and analyticity for barycenter equations in the Hilbert space (Quang, 2020).

5. Interpolation Between Wasserstein and KL Divergence

As $\lambda\to0$, the entropy-regularized Wasserstein distance and its interpolating divergence converge to their classical (unregularized) OT analogues. As $\lambda\to\infty$, the cost becomes dominated by entropy, and the limiting divergence approaches the KL divergence:

$\lambda$	Limiting behavior
$\lambda \to 0$	Wasserstein metric structure (geometry of OT)
$\lambda \to \infty$	KL divergence (information geometry)

This interpolation underlies the continuous family of divergences and the Bregman-type structure (Amari et al., 2017, Tong et al., 2020).

6. Applications and Algorithmic Methods

Entropy-regularized Wasserstein distances and their Sinkhorn divergences have found broad application. Notable domains and methods include:

• Probability Distribution Analysis: Dimension-free discrepancy measures, statistical testing, and bootstrap-based confidence estimation (Bigot et al., 2017, Tong et al., 2020, Mallasto et al., 2020).
• Machine Learning: Generative modeling (e.g., Sinkhorn GANs (Reshetova et al., 2021)), robust estimation (via median-of-means (Staerman et al., 2020)), clustering, and learning with structured losses (Luise et al., 2018, Mallery et al., 13 Jan 2025).
• Barycenter Computation: The entropy-regularized barycenter is characterized by a fixed-point equation involving entropic displacement maps, supporting efficient Wasserstein gradient descent and convex quadratic programming for the analysis problem (Quang, 2020, Mallery et al., 13 Jan 2025).
• Robust Optimization: Distributionally robust shape and topology optimization (Dapogny et al., 2022) and quantization tasks (Lakshmanan et al., 2023) benefit from the smoothing regularization and convexity of the entropic term.
• Distributional Uncertainty and Stochastic Control: Causal entropy-regularized Wasserstein distances for time series/filtered processes with closed-form solutions for Gaussians (Acciaio et al., 25 Dec 2024).
• Cross-lingual Information Retrieval: Regularized Wasserstein methods for aligning multilingual document embeddings, leveraging term-weighted couplings and OOV handling (Balikas et al., 2018).

7. Theoretical Implications, Limitations, and Extensions

The entropy-regularized Wasserstein distance framework draws a rigorous connection between metric and information geometries via its exponential family structure. It admits sharp statistical analysis: parametric rates, bootstrap-valid tests, and differentiability that holds even in infinite-dimensional settings (Quang, 2020, Zhang et al., 2022).

However, certain limitations and subtleties remain:

• Bias in the Regularized Cost: The entropy-regularized cost is not “zero” for coinciding distributions; Sinkhorn divergence corrects this bias (Bigot et al., 2017, Tong et al., 2020).
• Numerical Artifacts: Excessively strong regularization (large $\lambda$) leads to overly “blurry” transport plans; choices of $\lambda$ must balance bias and computational tractability (Luise et al., 2018, Lakshmanan et al., 2023).
• Extension Beyond Simplex: For non-discrete or infinite-dimensional distributions, explicit solutions often require Gaussianity; non-Gaussian or nonparametric settings may only admit computational or variational characterization (Quang, 2020, Tong et al., 2020).
• Generalized Regularizations: Alternative entropy forms (e.g., Tsallis) and adapted measures (e.g., for time series) provide further avenues for analysis and application, with closed-form results in select cases (Tong et al., 2020, Acciaio et al., 25 Dec 2024).

---

In summary, the entropy-regularized Wasserstein distance provides a unified, computationally tractable framework that bridges metric OT geometry with information geometry, enables scalable algorithmics, supports robust inference and statistical learning, and connects to a wide spectrum of modern data analysis and optimization problems (Amari et al., 2017, Bigot et al., 2017, Luise et al., 2018, Quang, 2020, Tong et al., 2020, Porretta, 2022, Dapogny et al., 2022, Lakshmanan et al., 2023, Acciaio et al., 25 Dec 2024, Mallery et al., 13 Jan 2025).