Sinkhorn-Approximated Losses

Updated 30 March 2026

Sinkhorn-Approximated Losses are entropic regularization-based surrogates for optimal transport cost, balancing OT geometry with kernel methods like MMD.
They utilize iterative Sinkhorn updates and debiasing techniques to yield convex, scalable, and differentiable implementations that recover classical metrics in limits.
These losses have broad applications in generative modeling, domain adaptation, and Bayesian inference, improving computational efficiency and statistical robustness.

The Sinkhorn-approximated loss is a computationally tractable, entropic regularization-based surrogate for the classical optimal transport (OT) cost, constructed to compare probability measures by interpolating between OT geometry and maximum mean discrepancy (MMD). The key mechanism involves introducing an entropic smoothing to the transport problem, yielding strictly convex objectives and enabling scalable, differentiable implementations. A central further correction is the removal of entropic "self-bias" via a debiasing scheme, leading to the so-called Sinkhorn divergence. This class of losses has rapidly become a standard tool in statistical machine learning, generative modeling, computational optimal transport, and geometric data analysis.

1. Formal Construction and Theoretical Properties

Let $X$ be a compact metric space, $c(x, y) \geq 0$ a continuous cost, and $\mu$ , $\nu$ probability measures on $X$ . The entropic regularized OT cost is defined by: $W_\varepsilon(\mu, \nu) = \min_{\pi \in \Pi(\mu, \nu)} \iint c(x, y) d\pi(x, y) + \varepsilon\, \mathrm{KL}(\pi \| \mu \otimes \nu),$ where $\mathrm{KL}(\pi\,\|\mu\otimes\nu)$ is the Kullback–Leibler divergence and $\Pi(\mu, \nu)$ denotes the set of couplings with prescribed marginals. The Sinkhorn divergence removes the regularization bias: $S_\varepsilon(\mu, \nu) = W_\varepsilon(\mu, \nu) - \frac{1}{2} W_\varepsilon(\mu, \mu) - \frac{1}{2} W_\varepsilon(\nu, \nu).$

Key properties under mild regularity (e.g., $c$ Lipschitz, $k_c(x,y) = \exp(-c(x,y)/\varepsilon)$ universal) (Feydy et al., 2018, Hardion et al., 18 Nov 2025):

$S_\varepsilon \geq 0$ with $S_\varepsilon(\mu, \nu) = 0$ iff $\mu = \nu$ .
Symmetry: $S_\varepsilon(\mu, \nu) = S_\varepsilon(\nu, \mu)$ .
Convexity in each argument.
Metric-like behavior: $S_\varepsilon$ metrizes weak convergence but does not satisfy the triangle inequality for $\sqrt{S_\varepsilon}$ (Lavenant et al., 2024).

For $\varepsilon \to 0$ , $S_\varepsilon$ recovers the squared 2-Wasserstein distance $W_2^2$ . For $\varepsilon \to \infty$ , $S_\varepsilon$ approximates a kernel MMD with kernel $k(x, y) = \exp(-c(x, y)/\varepsilon)$ (Feydy et al., 2018, Hardion et al., 18 Nov 2025).

2. Computational Schemes and Implementation

The dual formulation for the entropic OT cost is (Genevay et al., 2017): $W_\varepsilon(\mu, \nu) = \max_{u, v} \int u\,d\mu + \int v\,d\nu - \varepsilon \iint \exp\left(\frac{u(x) + v(y) - c(x, y)}{\varepsilon}\right) d\mu(x) d\nu(y)$ with optimal dual potentials $(u, v)$ , leading to the Gibbs coupling $\pi^*$ . In the discrete setting with supports $\{x_i\}$ , $\{y_j\}$ and cost matrix $C_{ij} = c(x_i, y_j)$ :

Define $K = \exp(-C/\varepsilon)$ , and iterate Sinkhorn updates: $a \leftarrow \mathbf{1}_n \oslash (K b), ~ b \leftarrow \mathbf{1}_m \oslash (K^\mathsf{T} a)$ until convergence.
The practical computational cost is $O(n^2 L)$ for $L$ Sinkhorn steps (Hardion et al., 18 Nov 2025).

Significant advances include:

Positive-feature representations, reducing iteration cost from $O(n^2)$ to $O(nr)$ where $r \ll n$ (Scetbon et al., 2020).
Hierarchical low-rank schemes for grid data, with cost $O(n \log^3 n)$ (Motamed, 2020).
Efficient GPU/automatic differentiation through log-sum-exp/backpropagation for scalable training (Genevay et al., 2017, Feydy et al., 2018).
Coreset construction and Nyström/MMD reductions, yielding $O(\log n)$ size coresets for Gaussian-type kernels (Kokot et al., 28 Apr 2025).

3. Statistical, Regularity, and Convergence Analyses

Sinkhorn-approximated losses interpolate the curse-of-dimensionality of OT with the parametric sample complexity of MMD (Genevay et al., 2017, Chizat et al., 2020). For measures in $\mathbb{R}^d$ :

Unregularized OT sample complexity: $O(n^{-1/d})$ .
MMD regime: $O(n^{-1/2})$ .
Sinkhorn: intermediate, controlled by $\varepsilon$ ; for large $\varepsilon$ the variance is improved.

Bias analysis (Chizat et al., 2020): $| S_\varepsilon(\mu, \nu) - W_2^2(\mu, \nu) | = O(\varepsilon^2)$ with the debiasing crucial for removing $\varepsilon$ -dependent bias evident in $W_\varepsilon(\mu, \nu)$ alone. Statistical and computational complexity trade-offs allow for orders-of-magnitude speedups over unbiased OT when moderate accuracy suffices.

Hadamard differentiability and higher-order expansions are established (Goldfeld et al., 2022, Kokot et al., 28 Apr 2025):

First- and second-order differentiability for potentials as functions of marginal measures.
Central limit theorems for empirical Sinkhorn divergences and bootstrap consistency.
Explicit limit distributions under null and alternative hypotheses for independence testing.
Local quadratic expansion of $S_\varepsilon$ links it to kernel MMD, with Hessian operator identified.

Ergodic and stochastic optimization perspectives, including stochastic approximation (Robbins–Monro) algorithms, show almost sure convergence and asymptotic normality for recursive Sinkhorn estimators in discrete and semi-discrete settings (Bercu et al., 2018).

4. Generalizations and Extensions

There are several prominent generalizations of Sinkhorn-approximated losses:

Unbalanced Sinkhorn Divergences: Marginal relaxations via $\varphi$ -divergences replace hard mass constraints, enabling statistical robustness to outliers, total-mass discrepancies, and subsampling (Séjourné et al., 2019). The generalized primal/dual and Sinkhorn iteration—alternating pointwise “prox” and softmin operators—yield convergence guarantees.
Nested Sinkhorn Divergence: The multi-stage (tree-structured) setting, extending to filtrations, allows construction of entropy-regularized nested distances for stochastic processes, with efficient recursive implementation (Pichler et al., 2021).
Barycenters and Gradient Flows: Sinkhorn divergences define strictly convex barycenter problems with provable convergence via Frank-Wolfe algorithms (Luise et al., 2019) and underpin gradient flows analogous to those for Wasserstein distances, but with fundamentally different transport-diffusion interplay (Hardion et al., 18 Nov 2025, Zhu et al., 2024).

5. Geometry, Metric Structure, and Connections to RKHS

Recent advances characterize the geometry of Sinkhorn divergences:

The metric tensor induced by the Hessian $\nabla^2 S_\varepsilon(\mu, \mu)$ gives a Riemannian structure on the space of probability measures, related to the reproducing kernel Hilbert space (RKHS) defined by $k_c(x, y)$ (Lavenant et al., 2024).
The intrinsic distance defined by geodesics in this metric is equivalent to the RKHS norm in appropriate coordinates.
Sinkhorn divergences convexify only in each argument and lack joint convexity; $\sqrt{S_\varepsilon}$ fails the triangle inequality, necessitating second-order structure for rigorous geometry.
The tangent space at $\mu$ matches the dual of the RKHS modulo constants.

6. Applications in Learning, Inference, and Optimization

Sinkhorn-approximated losses enable:

Large-scale training and evaluation of generative models (GANs, VAEs), yielding effective bias-variance tradeoffs in practice (Genevay et al., 2017, Shen et al., 2020).
Feature alignment in representation learning and domain adaptation by aligning empirical feature distributions across source domains (Lee et al., 2023).
Fast and scalable diffeomorphic registration (Lara et al., 2022).
Efficient score-based generative modeling via Sinkhorn-induced gradient flows and neural ODEs (Zhu et al., 2024).
Drifting generative models in which the Sinkhorn divergence confers robust identifiability and stabilization across kernel scales (He et al., 12 Mar 2026).
Coreset selection and data compression for large datasets, with theoretical guarantees tied to kernel eigenvalue decay (Kokot et al., 28 Apr 2025).
Bayesian inference, sampling, and barycenter computation, where the S-JKO scheme enables “teleportation” across nonconvex barriers due to the intrinsic geometry of $S_\varepsilon$ (Hardion et al., 18 Nov 2025).

7. Open Questions and Limitations

Despite their versatility, Sinkhorn-approximated losses have notable limitations and research frontiers:

Joint convexity and metricity: $S_\varepsilon$ is not jointly convex or a metric; its geodesic path distance must be defined via second-order expansion.
Choice of $\varepsilon$ remains problem-dependent, involving bias-variance and computational trade-offs.
Approximation quality under singular, heavy-tailed, or highly discrete distributions requires further study.
For very small $\varepsilon$ , computational cost increases and numerical stability deteriorates.
The extension to non-Euclidean costs, graphs, and manifold domains is ongoing.
Scaling to extreme data sizes remains active, with new positive-feature and hierarchical approaches partly mitigating quadratic complexity (Scetbon et al., 2020, Motamed, 2020).

Sinkhorn-approximated losses thus provide a principled, flexible, and computationally tractable bridge between optimal transport theory and kernel-based machine learning, with rapidly expanding theoretical and practical scope.