---
title: 'Sinkhorn Distances: Entropic-Regularized OT'
url: https://www.emergentmind.com/topics/sinkhorn-distances
type: topic
---

# Sinkhorn Distances: Entropic-Regularized OT

Sinkhorn distances (also known as entropic-regularized optimal transport distances or regularized Wasserstein metrics) are a family of geometric divergences between probability measures that arise from adding an entropic penalty to the classical optimal transport (OT) problem and are computed efficiently by matrix scaling algorithms (notably the Sinkhorn–Knopp algorithm). They interpolate between the true Wasserstein metric and kernel-based Maximum Mean Discrepancies (MMDs), offering computational tractability and differentiability at scale. Beyond their foundational role in computational optimal transport, Sinkhorn distances are used in diverse areas such as generative modeling, imitation learning, geometric statistics on manifolds, scalable high-dimensional data analysis, and efficient streaming evaluation of OT. They are also the basis for numerous algorithmic and theoretical advances in large-scale statistical inference, machine learning, and computational geometry.

## 1. Definition, Formulation, and Theoretical Foundation

Let $(X,d)$ be a compact metric space or a bounded subset of $\mathbb{R}^d$ equipped with a continuous ground cost $c: X \times X \to \mathbb{R}_+$. For probability measures $\mu, \nu \in \mathcal{P}(X)$, the entropy-regularized OT (Sinkhorn) cost with regularization parameter $\varepsilon > 0$ is
\[
\mathrm{OT}_\varepsilon(\mu, \nu) = \inf_{\pi \in \Pi(\mu, \nu)} \left\{ \iint c(x, y) d\pi(x, y) + \varepsilon\, \mathrm{KL}(\pi \;\|\; \mu \otimes \nu)\right\}
\]
where $\Pi(\mu, \nu)$ is the set of couplings of $\mu$ and $\nu$, and $\mathrm{KL}$ is the Kullback–Leibler divergence. The dual form reads
\[
\mathrm{OT}_\varepsilon(\mu, \nu) = \sup_{f, g \in C(X)} \left\{ \langle f, \mu \rangle + \langle g, \nu \rangle - \varepsilon \langle \mu \otimes \nu, \exp\left(\frac{f \oplus g - c}{\varepsilon}\right) - 1 \rangle \right\}.
\]
Since $\mathrm{OT}_\varepsilon(\mu, \mu) > 0$ for $\varepsilon > 0$, an unbiased Sinkhorn divergence is constructed as
\[
S_\varepsilon(\mu, \nu) = \mathrm{OT}_\varepsilon(\mu, \nu) - \frac{1}{2} \mathrm{OT}_\varepsilon(\mu, \mu) - \frac{1}{2} \mathrm{OT}_\varepsilon(\nu, \nu).
\]
This debiased divergence is symmetric, non-negative, $S_\varepsilon(\mu, \mu) = 0$, and, under mild universal kernel hypotheses, $S_\varepsilon(\mu, \nu) = 0$ if and only if $\mu = \nu$ [1810.08278]. As $\varepsilon \to 0$, $S_\varepsilon \to \mathrm{OT}_0$, i.e., the true Wasserstein distance.

Key properties include separate convexity, smoothness, metrization of weak convergence, and connections between bias/variance and $\varepsilon$ [1810.08278, 2405.04987].

## 2. Algorithmic Framework and Sinkhorn–Knopp Iteration

For discrete measures $\mu = \sum_{i=1}^n a_i \delta_{x_i}$, $\nu = \sum_{j=1}^m b_j \delta_{y_j}$, and a cost matrix $C_{ij} = c(x_i, y_j)$, the entropic-regularized OT becomes a strictly convex program,
\[
\min_{P \in U(a, b)} \langle P, C \rangle + \varepsilon \sum_{i,j} P_{ij} (\log P_{ij} - 1)
\]
where $U(a, b) = \{ P \ge 0 : P \mathbf{1}_m = a, P^\top \mathbf{1}_n = b \}$. The solution $P^*$ has the factorization $P^* = \mathrm{diag}(u) K \mathrm{diag}(v)$ with $K_{ij} = \exp(-C_{ij} / \varepsilon)$, and $(u, v)$ found by iterating the Sinkhorn–Knopp scheme:
\[
u^{(k+1)} = a / (K v^{(k)}), \qquad v^{(k+1)} = b / (K^\top u^{(k+1)})
\]
until marginal errors are below a threshold [1306.0895, 1705.09634]. Each update costs $O(nm)$, converges geometrically under mild conditions, and is stable for moderate $\varepsilon$. Large-scale GPU acceleration is enabled via batched two-sided mat-vec products [1810.08278].

For square cost matrices, greedy coordinate variants such as Greenkhorn [1705.09634] and Newton-type accelerations [2401.12253] further optimize performance.

## 3. Statistical, Geometric, and Computational Properties

Sinkhorn divergences interpolate between Wasserstein geometry ($\varepsilon \to 0$) and kernel MMD ($\varepsilon \to \infty$) [1810.08278]. As $\varepsilon$ increases, the estimator becomes more regularized (biased) but exhibits improved sample complexity (from $O(n^{-1/d})$ for unregularized OT to $O(n^{-1/2})$ in the large-$\varepsilon$ regime) [1810.08278, 2006.08172]. For moderate $\varepsilon$, $S_\varepsilon$ preserves OT-type geometry but is computationally tractable and differentiable at scale.

In the large-$n$ regime, low-rank kernel compression methods (Nyström [1812.05189], hierarchical [2004.12511]) and graph-based geodesic approximations [2211.00805] achieve $O(n \log n)$ or near-linear complexity while maintaining controlled approximation error.

Theoretical results include non-asymptotic error and complexity bounds, bias-variance trade-offs for $S_\varepsilon$ versus plug-in estimators, and bounds on the regularization-induced error as a function of $\varepsilon$ and data smoothness [2006.08172].

Notably, the Hessian of $S_\varepsilon$ with respect to the measure defines a Riemannian metric on the probability simplex, inducing a geodesic distance metrizing the weak-* topology and connecting to the structure of RKHS [2405.04987].

## 4. Extensions: Streaming, Stochastic, and Online Sinkhorn

Sinkhorn divergences enable extensions to streaming and online learning contexts where classic batch OT is infeasible. The online Sinkhorn algorithm [2003.01415] maintains kernel mixture representations of dual potentials, updating them incrementally as new samples arrive, yielding nearly consistent estimation of regularized OT from streams. Further, compressed online Sinkhorn [2310.05019] introduces moment-preserving compression to stabilize memory and computation, achieving convergence rates matching the best nonparametric online schemes and leveraging efficient test function families (Fourier features, Gaussian quadrature).

These algorithms allow OT-based distances to be used as loss functions in continuous deep generative modeling, domain adaptation, and other settings where large or dynamically growing datasets prevent materializing the full cost matrix or kernel.

## 5. Applications and Generalizations

Sinkhorn divergences underpin numerous machine learning and statistical applications:

- **Generative modeling**: Sinkhorn Autoencoders (SAE) [1810.01118] and WAE-OT models leverage $S_\varepsilon$ as match metrics in latent spaces, yielding geometry-aware, differentiable training that generalizes to non-Euclidean supports and priors.
- **Imitation learning**: SIL (Sinkhorn imitation learning) [2008.09167] replaces adversarial divergence with an adversarially learned Sinkhorn distance between occupancy measures in an RL setting, providing superior discriminative power and stable training compared to GAN-based approaches.
- **Representation learning**: Sinkhorn divergences regularize unsupervised learning of structured audio representations, enhancing additivity and interpretability [2007.02780].
- **Mixture modeling**: Chain-Rule OT and its Sinkhorn regularization extend to divergences between statistical mixtures, including Renyi and KL divergences on GMMs, with guarantees on smoothness and upper bounds [1812.08113].
- **Statistics on manifolds**: Geodesic Sinkhorn utilizes graph Laplacian-based heat kernels to match empirically-defined distributions on non-Euclidean supports, capturing geometries inaccessible to classical ($\ell_2$-based) Sinkhorn [2211.00805].
- **Stochastic processes**: Nested Sinkhorn divergences efficiently compute entropy-regularized (multistage) OT for stochastic processes, offering sharply reduced computational costs compared to full nested LPs [2102.05413].

## 6. Limitations, Riemannian Geometry, and Open Problems

Despite positive definiteness and separate convexity, Sinkhorn divergences are not jointly convex in $(\mu, \nu)$ and their square root fails the triangle inequality, so $S_\varepsilon^{1/2}$ is not a metric [2405.04987, 1810.08278]. The Riemannian structure defined by the Hessian of $S_\varepsilon$ yields a geodesic distance metrizing the weak-* topology; however, geometric properties such as geodesic uniqueness or compatibility with classical OT geodesics can diverge in the large-$\varepsilon$ regime. The debiasing terms are necessary to avoid entropic fixed points that can conflict with true metric properties.

Ongoing challenges include extending sharp complexity bounds to high-dimensional and manifold scenarios (especially for extremely small $\varepsilon$), further improving approximation schemes for streaming and online contexts, and elucidating the geometry and topology of Sinkhorn-induced metric structures [2405.04987, 2211.00805]. Asymptotic statistical efficiency, bias correction (e.g., via Richardson extrapolation [2006.08172]), and compatibility with scalable GPU infrastructure remain active research directions.

## 7. Comparative Table: Key Properties and Regimes

| Property                   | Classical OT         | Sinkhorn ($\varepsilon>0$) | MMD ($\varepsilon \to \infty$) |
|----------------------------|---------------------|----------------------------|------------------------------|
| Positive definite          | Yes                 | Yes                        | Yes                          |
| (Joint) convexity          | Yes                 | No                         | Yes                          |
| Metric (triangle ineq.)    | Yes                 | Not for $S_\varepsilon^{1/2}$   | Yes                          |
| Geometry                   | Wasserstein         | Interpolates OT/MMD        | Kernel                       |
| Stochastic computation     | Infeasible          | Online/streaming feasible  | SGD/minibatch                |
| GPU/parallelization        | Hard (LP)           | Easy (mat-vec/scaling)     | Easiest                      |
| Applicability              | Small/moderate $n$  | Large scale                | Massive scale                |

This delineates the tractable, geometry-aware, and scalable regime occupied by Sinkhorn distances, central to modern computational optimal transport [1306.0895, 1810.08278, 2006.08172].

Source: https://www.emergentmind.com/topics/sinkhorn-distances