---
title: Sinkhorn-Approximated Losses
url: https://www.emergentmind.com/topics/sinkhorn-approximated-losses-230d4a58-5ddb-4d44-ae1c-08291c88fde8
type: topic
---

# Sinkhorn-Approximated Losses

Sinkhorn-Approximated Losses

The Sinkhorn-approximated loss is a computationally tractable, entropic regularization-based surrogate for the classical optimal transport (OT) cost, constructed to compare probability measures by interpolating between OT geometry and maximum mean discrepancy (MMD). The key mechanism involves introducing an entropic smoothing to the transport problem, yielding strictly convex objectives and enabling scalable, differentiable implementations. A central further correction is the removal of entropic "self-bias" via a debiasing scheme, leading to the so-called Sinkhorn divergence. This class of losses has rapidly become a standard tool in statistical machine learning, generative modeling, computational optimal transport, and geometric data analysis.

## 1. Formal Construction and Theoretical Properties

Let $X$ be a compact metric space, $c(x, y) \geq 0$ a continuous cost, and $\mu$, $\nu$ probability measures on $X$. The entropic regularized OT cost is defined by:
\[
W_\varepsilon(\mu, \nu) = \min_{\pi \in \Pi(\mu, \nu)} \iint c(x, y) d\pi(x, y) + \varepsilon\, \mathrm{KL}(\pi \| \mu \otimes \nu),
\]
where $\mathrm{KL}(\pi\,\|\mu\otimes\nu)$ is the Kullback–Leibler divergence and $\Pi(\mu, \nu)$ denotes the set of couplings with prescribed marginals. The Sinkhorn divergence removes the regularization bias:
\[
S_\varepsilon(\mu, \nu) = W_\varepsilon(\mu, \nu) - \frac{1}{2} W_\varepsilon(\mu, \mu) - \frac{1}{2} W_\varepsilon(\nu, \nu).
\]

Key properties under mild regularity (e.g., $c$ Lipschitz, $k_c(x,y) = \exp(-c(x,y)/\varepsilon)$ universal) [1810.08278, 2511.14278]:
- $S_\varepsilon \geq 0$ with $S_\varepsilon(\mu, \nu) = 0$ iff $\mu = \nu$.
- Symmetry: $S_\varepsilon(\mu, \nu) = S_\varepsilon(\nu, \mu)$.
- Convexity in each argument.
- Metric-like behavior: $S_\varepsilon$ metrizes weak convergence but does not satisfy the triangle inequality for $\sqrt{S_\varepsilon}$ [2405.04987].

For $\varepsilon \to 0$, $S_\varepsilon$ recovers the squared 2-Wasserstein distance $W_2^2$. For $\varepsilon \to \infty$, $S_\varepsilon$ approximates a kernel MMD with kernel $k(x, y) = \exp(-c(x, y)/\varepsilon)$ [1810.08278, 2511.14278].

## 2. Computational Schemes and Implementation

The dual formulation for the entropic OT cost is [1706.00292]:
\[
W_\varepsilon(\mu, \nu) = \max_{u, v} \int u\,d\mu + \int v\,d\nu - \varepsilon \iint \exp\left(\frac{u(x) + v(y) - c(x, y)}{\varepsilon}\right) d\mu(x) d\nu(y)
\]
with optimal dual potentials $(u, v)$, leading to the Gibbs coupling $\pi^*$. In the discrete setting with supports $\{x_i\}$, $\{y_j\}$ and cost matrix $C_{ij} = c(x_i, y_j)$:
- Define $K = \exp(-C/\varepsilon)$, and iterate Sinkhorn updates: $a \leftarrow \mathbf{1}_n \oslash (K b), ~ b \leftarrow \mathbf{1}_m \oslash (K^\mathsf{T} a)$ until convergence.
- The practical computational cost is $O(n^2 L)$ for $L$ Sinkhorn steps [2511.14278].

Significant advances include:
- Positive-feature representations, reducing iteration cost from $O(n^2)$ to $O(nr)$ where $r \ll n$ [2006.07057].
- Hierarchical low-rank schemes for grid data, with cost $O(n \log^3 n)$ [2004.12511].
- Efficient GPU/automatic differentiation through log-sum-exp/backpropagation for scalable training [1706.00292, 1810.08278].
- Coreset construction and Nyström/MMD reductions, yielding $O(\log n)$ size coresets for Gaussian-type kernels [2504.20194].

## 3. Statistical, Regularity, and Convergence Analyses

Sinkhorn-approximated losses interpolate the curse-of-dimensionality of OT with the parametric sample complexity of MMD [1706.00292, 2006.08172]. For measures in $\mathbb{R}^d$:
- Unregularized OT sample complexity: $O(n^{-1/d})$.
- MMD regime: $O(n^{-1/2})$.
- Sinkhorn: intermediate, controlled by $\varepsilon$; for large $\varepsilon$ the variance is improved.

Bias analysis [2006.08172]:
\[
| S_\varepsilon(\mu, \nu) - W_2^2(\mu, \nu) | = O(\varepsilon^2)
\]
with the debiasing crucial for removing $\varepsilon$-dependent bias evident in $W_\varepsilon(\mu, \nu)$ alone. Statistical and computational complexity trade-offs allow for orders-of-magnitude speedups over unbiased OT when moderate accuracy suffices.

Hadamard differentiability and higher-order expansions are established [2207.08683, 2504.20194]:
- First- and second-order differentiability for potentials as functions of marginal measures.
- Central limit theorems for empirical Sinkhorn divergences and bootstrap consistency.
- Explicit limit distributions under null and alternative hypotheses for independence testing.
- Local quadratic expansion of $S_\varepsilon$ links it to kernel MMD, with Hessian operator identified.

Ergodic and stochastic optimization perspectives, including stochastic approximation (Robbins–Monro) algorithms, show almost sure convergence and asymptotic normality for recursive Sinkhorn estimators in discrete and semi-discrete settings [1812.09150].

## 4. Generalizations and Extensions

There are several prominent generalizations of Sinkhorn-approximated losses:
- **Unbalanced Sinkhorn Divergences:** Marginal relaxations via $\varphi$-divergences replace hard mass constraints, enabling statistical robustness to outliers, total-mass discrepancies, and subsampling [1910.12958]. The generalized primal/dual and Sinkhorn iteration—alternating pointwise “prox” and softmin operators—yield convergence guarantees.
- **Nested Sinkhorn Divergence:** The multi-stage (tree-structured) setting, extending to filtrations, allows construction of entropy-regularized nested distances for stochastic processes, with efficient recursive implementation [2102.05413].
- **Barycenters and Gradient Flows:** Sinkhorn divergences define strictly convex barycenter problems with provable convergence via Frank-Wolfe algorithms [1905.13194] and underpin gradient flows analogous to those for Wasserstein distances, but with fundamentally different transport-diffusion interplay [2511.14278, 2401.14069].

## 5. Geometry, Metric Structure, and Connections to RKHS

Recent advances characterize the geometry of Sinkhorn divergences:
- The metric tensor induced by the Hessian $\nabla^2 S_\varepsilon(\mu, \mu)$ gives a Riemannian structure on the space of probability measures, related to the reproducing kernel Hilbert space (RKHS) defined by $k_c(x, y)$ [2405.04987].
- The intrinsic distance defined by geodesics in this metric is equivalent to the RKHS norm in appropriate coordinates.
- Sinkhorn divergences convexify only in each argument and lack joint convexity; $\sqrt{S_\varepsilon}$ fails the triangle inequality, necessitating second-order structure for rigorous geometry.
- The tangent space at $\mu$ matches the dual of the RKHS modulo constants.

## 6. Applications in Learning, Inference, and Optimization

Sinkhorn-approximated losses enable:
- Large-scale training and evaluation of generative models (GANs, VAEs), yielding effective bias-variance tradeoffs in practice [1706.00292, 2011.04162].
- Feature alignment in representation learning and domain adaptation by aligning empirical feature distributions across source domains [2305.15196].
- Fast and scalable diffeomorphic registration [2206.13948].
- Efficient score-based generative modeling via Sinkhorn-induced gradient flows and neural ODEs [2401.14069].
- Drifting generative models in which the Sinkhorn divergence confers robust identifiability and stabilization across kernel scales [2603.12366].
- Coreset selection and data compression for large datasets, with theoretical guarantees tied to kernel eigenvalue decay [2504.20194].
- Bayesian inference, sampling, and barycenter computation, where the S-JKO scheme enables “teleportation” across nonconvex barriers due to the intrinsic geometry of $S_\varepsilon$ [2511.14278].

## 7. Open Questions and Limitations

Despite their versatility, Sinkhorn-approximated losses have notable limitations and research frontiers:
- Joint convexity and metricity: $S_\varepsilon$ is not jointly convex or a metric; its geodesic path distance must be defined via second-order expansion.
- Choice of $\varepsilon$ remains problem-dependent, involving bias-variance and computational trade-offs.
- Approximation quality under singular, heavy-tailed, or highly discrete distributions requires further study.
- For very small $\varepsilon$, computational cost increases and numerical stability deteriorates.
- The extension to non-Euclidean costs, graphs, and manifold domains is ongoing.
- Scaling to extreme data sizes remains active, with new positive-feature and hierarchical approaches partly mitigating quadratic complexity [2006.07057, 2004.12511].

Sinkhorn-approximated losses thus provide a principled, flexible, and computationally tractable bridge between optimal transport theory and kernel-based machine learning, with rapidly expanding theoretical and practical scope.

Source: https://www.emergentmind.com/topics/sinkhorn-approximated-losses-230d4a58-5ddb-4d44-ae1c-08291c88fde8