Sinkhorn-Approximated Losses

• Sinkhorn-approximated losses are a class of optimal transport-based loss functions that use entropic regularization to produce differentiable and statistically robust metrics.
• They leverage fixed-point Sinkhorn iterations to efficiently approximate the regularized OT cost, ensuring convergence of the dual potentials under marginal constraints.
• Advanced variants extend this framework to online, partial transport, and generalized settings, enabling applications in generative modeling, robust optimization, and distributionally robust learning.

Sinkhorn-approximated losses are a class of optimal transport-based loss functions incorporating entropic regularization to achieve tractable, differentiable, and statistically robust optimization over probability distributions. These losses interpolate between optimal transport (OT) metrics such as Wasserstein distance and kernel-based alternatives like maximum mean discrepancy (MMD), and form the computational backbone of modern generative modeling, robust optimization, and related machine learning frameworks.

1. Mathematical Foundations of Sinkhorn Losses

Sinkhorn-approximated losses originate from the regularized OT problem. Given two probability measures $\mu = \sum_{i=1}^n a_i \delta_{x_i}$ and $\nu = \sum_{j=1}^m b_j \delta_{y_j}$, and a cost matrix $C \in \mathbb{R}^{n\times m}$ with entries $C_{ij}=c(x_i,y_j)$, the entropic OT cost is

$$W_\varepsilon(\mu, \nu) = \min_{\pi \in \Pi(a, b)} \sum_{i,j} C_{ij} \pi_{ij} + \varepsilon\sum_{i,j} \pi_{ij}(\log \pi_{ij} - 1),$$

where $$\Pi(a, b) = \{ \pi \in \mathbb{R}_+^{n \times m} : \pi 1_m = a, \pi^T 1_n = b \}$$ is the set of admissible couplings.

The Sinkhorn divergence, used to debias entropic OT and ensure metric properties, is

$$S_\varepsilon(\mu, \nu) = W_\varepsilon(\mu, \nu) - \frac{1}{2} W_\varepsilon(\mu, \mu) - \frac{1}{2} W_\varepsilon(\nu, \nu).$$

Tuning $\varepsilon$ interpolates between the geometric OT regime ($\varepsilon \to 0$) and the MMD regime ($\varepsilon \to \infty$), where the coupling approaches the independent product $\nu = \sum_{j=1}^m b_j \delta_{y_j}$0 and $\nu = \sum_{j=1}^m b_j \delta_{y_j}$1 degenerates to an energy or kernel-based divergence.

2. Computational Algorithms: Sinkhorn Iterations

The central computational procedure for evaluating Sinkhorn losses is a fixed-point iteration referred to as the Sinkhorn algorithm. Defining the Gibbs kernel $\nu = \sum_{j=1}^m b_j \delta_{y_j}$2, the optimal plan is sought as $\nu = \sum_{j=1}^m b_j \delta_{y_j}$3 subject to marginal constraints encoded by the iterative updates: $\nu = \sum_{j=1}^m b_j \delta_{y_j}$4 All vector divisions are element-wise. After $\nu = \sum_{j=1}^m b_j \delta_{y_j}$5 iterations, the coupling is $\nu = \sum_{j=1}^m b_j \delta_{y_j}$6, yielding an approximate cost $\nu = \sum_{j=1}^m b_j \delta_{y_j}$7 (Genevay et al., 2017). This scheme admits robust implementation on modern GPUs with batch sizes $\nu = \sum_{j=1}^m b_j \delta_{y_j}$8 in the range $\nu = \sum_{j=1}^m b_j \delta_{y_j}$9–$C \in \mathbb{R}^{n\times m}$0 and iteration count $C \in \mathbb{R}^{n\times m}$1 typically $C \in \mathbb{R}^{n\times m}$2–$C \in \mathbb{R}^{n\times m}$3 for $C \in \mathbb{R}^{n\times m}$4.

For generalized OT problems including partial transport or unbalanced couplings, the Sinkhorn framework adapts via proximal/divide updates and clipping/min operations on dual scalings $C \in \mathbb{R}^{n\times m}$5 to enforce penalized marginal constraints (Bai, 2024).

3. Differentiability, Gradient Computation, and Implicit Differentiation

The entropic regularization confers infinite differentiability ($C \in \mathbb{R}^{n\times m}$6) on both the Sinkhorn cost and Sinkhorn divergence in the simplex interior (Luise et al., 2018). Efficient gradient computation leverages either (i) backpropagation through unrolled Sinkhorn iterations, or (ii) implicit differentiation of the KKT conditions: $C \in \mathbb{R}^{n\times m}$7 where $C \in \mathbb{R}^{n\times m}$8 is the optimal plan and $C \in \mathbb{R}^{n\times m}$9 dual potentials. Solving the resulting sparse linear system yields vector-Jacobian products for $C_{ij}=c(x_i,y_j)$0 analytic gradients with provable error bounds (Eisenberger et al., 2022). Implicit differentiation is advantageous for high $C_{ij}=c(x_i,y_j)$1 or $C_{ij}=c(x_i,y_j)$2, providing superior memory efficiency over unrolled automatic differentiation (AD).

4. Statistical and Learning-Theoretic Properties

Entropic regularization produces strictly convex transport losses with improved sample complexity and variance. For small $C_{ij}=c(x_i,y_j)$3, the OT bias persists with the curse of dimensionality ($C_{ij}=c(x_i,y_j)$4); for large $C_{ij}=c(x_i,y_j)$5, the Sinkhorn loss exhibits MMD-like sample complexity ($C_{ij}=c(x_i,y_j)$6) (Genevay et al., 2017). In supervised learning, the sharp Sinkhorn loss ($C_{ij}=c(x_i,y_j)$7 without explicit entropy penalty in the final cost) guarantees universal consistency and fast excess risk convergence rates under standard RKHS assumptions (Luise et al., 2018).

Recent work verifies second-order Hadamard differentiability of Sinkhorn divergences, facilitating local quadratic approximations and enabling rigorous coreset construction (Kokot et al., 28 Apr 2025). This functional smoothness underpins compressed representations and efficient subsampling schemes.

5. Advanced Variants: Online, Generalized, and Partial Transport

Algorithms such as Online Sinkhorn (Mensch et al., 2020) permit stochastic streaming estimation of Sinkhorn-approximated losses, maintaining non-parametric mixture representations of scaling potentials $C_{ij}=c(x_i,y_j)$8:

• New sample batches update the mixture weights via stochastic approximation (Robbins–Monro step),
• Theoretical guarantees yield near-optimal $C_{ij}=c(x_i,y_j)$9 error rates for $$W_\varepsilon(\mu, \nu) = \min_{\pi \in \Pi(a, b)} \sum_{i,j} C_{ij} \pi_{ij} + \varepsilon\sum_{i,j} \pi_{ij}(\log \pi_{ij} - 1),$$0 samples.

Generalized and partial transport models (GOPT) introduce penalty functions $$W_\varepsilon(\mu, \nu) = \min_{\pi \in \Pi(a, b)} \sum_{i,j} C_{ij} \pi_{ij} + \varepsilon\sum_{i,j} \pi_{ij}(\log \pi_{ij} - 1),$$1 for mass destruction/creation and enable coupled clipping/min operations inside Sinkhorn iterates. This confers flexibility over the balanced/unbalanced spectrum by adjusting mass constraints in the primal and dual (Bai, 2024).

Convex regularization beyond Shannon entropy is accommodated in generalized Sinkhorn frameworks (Marino et al., 2020), including regularizers $$W_\varepsilon(\mu, \nu) = \min_{\pi \in \Pi(a, b)} \sum_{i,j} C_{ij} \pi_{ij} + \varepsilon\sum_{i,j} \pi_{ij}(\log \pi_{ij} - 1),$$2 (e.g. Tsallis, quadratic), with each instantiation yielding a corresponding dual, complementary slackness, and IPFP-type iterative scaling.

6. Practical Implementation, Optimization, and Applications

In contemporary deep learning, Sinkhorn layers are integrated end-to-end:

• Cost networks compute $$W_\varepsilon(\mu, \nu) = \min_{\pi \in \Pi(a, b)} \sum_{i,j} C_{ij} \pi_{ij} + \varepsilon\sum_{i,j} \pi_{ij}(\log \pi_{ij} - 1),$$3 from learned features,
• Sinkhorn iterations yield the coupling $$W_\varepsilon(\mu, \nu) = \min_{\pi \in \Pi(a, b)} \sum_{i,j} C_{ij} \pi_{ij} + \varepsilon\sum_{i,j} \pi_{ij}(\log \pi_{ij} - 1),$$4 and the regularized OT loss,
• Backpropagation exploits differentiability of matrix/tensor operations.

Loss surrogate terms in physics-informed neural networks, generative adversarial nets (GANs), Schrödinger bridges, and robust optimization pipelines use Sinkhorn divergences to enforce distributional constraints or supply differentiable distribution-matching objectives (Genevay et al., 2017, Nodozi et al., 2023, Wang et al., 2021). Distributionally robust optimization (DRO) with Sinkhorn balls replaces the hard Wasserstein supremum by a smooth log-sum-exp, making worst-case training tractable (Wang et al., 2021).

Specific algorithms, such as CO2-coresets, compress large datasets for regularized Sinkhorn loss minimization via spectral decompositions and MMD-based matching in RKHS, with polylogarithmic coreset size and near-optimal approximation error (Kokot et al., 28 Apr 2025).

7. Computational Complexity, Approximation Strategies, and Scalability

Each Sinkhorn iteration is $$W_\varepsilon(\mu, \nu) = \min_{\pi \in \Pi(a, b)} \sum_{i,j} C_{ij} \pi_{ij} + \varepsilon\sum_{i,j} \pi_{ij}(\log \pi_{ij} - 1),$$5; total cost is $$W_\varepsilon(\mu, \nu) = \min_{\pi \in \Pi(a, b)} \sum_{i,j} C_{ij} \pi_{ij} + \varepsilon\sum_{i,j} \pi_{ij}(\log \pi_{ij} - 1),$$6 for $$W_\varepsilon(\mu, \nu) = \min_{\pi \in \Pi(a, b)} \sum_{i,j} C_{ij} \pi_{ij} + \varepsilon\sum_{i,j} \pi_{ij}(\log \pi_{ij} - 1),$$7 steps. For large $$W_\varepsilon(\mu, \nu) = \min_{\pi \in \Pi(a, b)} \sum_{i,j} C_{ij} \pi_{ij} + \varepsilon\sum_{i,j} \pi_{ij}(\log \pi_{ij} - 1),$$8, strategies include:

• Screening (Screenkhorn) to selectively freeze negligible dual components (Alaya et al., 2019),
• Mini-batching to fit cost matrices in memory,
• Warm-starting dual variables for successive optimization loops,
• Implicit differentiation for scalable backward passes with fixed memory requirements.

Approximation bounds characterize the trade-off between marginal violations and objective error; screened or compressed solves can achieve marginal error below $$W_\varepsilon(\mu, \nu) = \min_{\pi \in \Pi(a, b)} \sum_{i,j} C_{ij} \pi_{ij} + \varepsilon\sum_{i,j} \pi_{ij}(\log \pi_{ij} - 1),$$9 and substantial computational savings given appropriate budgets (Alaya et al., 2019).

---

Sinkhorn-approximated losses synthesize optimal transport theory, entropic regularization, and iterative matrix scaling into a scalable, robust, and fully differentiable loss framework supporting a wide spectrum of modern statistical, optimization, and learning applications. With explicit control via $$\Pi(a, b) = \{ \pi \in \mathbb{R}_+^{n \times m} : \pi 1_m = a, \pi^T 1_n = b \}$$0 and extensions to convex regularizers, partial mass regimes, and sample-streaming/distributionally robust pipelines, Sinkhorn divergences constitute a foundational tool for high-dimensional generative modeling, robust learning, and applied optimal transport in both discrete and continuous settings.