K-OT Regularization in Optimal Transport

• K-OT regularization is a method that incorporates convex constraints into the classical Kantorovich OT framework to enhance tractability and control sample complexity.
• It employs strategies like entropic, quadratic, and structural regularization to influence the solution’s sparsity, smoothness, and convergence speed.
• The approach is widely applied in domain adaptation, generative modeling, and metric learning, with algorithms such as Sinkhorn iterations and mirror descent.

Kantorovich Optimal Transport (K-OT) Regularization is a collection of methodologies that introduce structural regularization terms or constraints into the classical Kantorovich optimal transport framework. These regularizations serve to ensure tractability, overcome statistical and computational obstacles, promote desired plan properties (such as sparsity or smoothness), and tightly control the sample complexity of Wasserstein distance estimation, especially in high-dimensional regimes. K-OT regularization underlies widely used solvers and enables extensions to new application domains in data science and machine learning.

1. The Classical Kantorovich OT Framework

Classical (unregularized) Kantorovich OT seeks a transport plan $\pi\in\mathcal{P}(X\times Y)$ between two probability measures $\mu$ and $\nu$ on Polish spaces $X, Y$, minimizing a cost functional: $$\min_{\pi\in\Pi(\mu,\nu)} \int_{X\times Y} c(x,y) \, d\pi(x,y)$$ where $\Pi(\mu,\nu)$ enforces marginal constraints. For $c(x,y) = \|x - y\|^2$ on $\mathbb{R}^d$, the $2$-Wasserstein distance arises. Solutions can be highly singular (plans supported on low-dimensional sets) and are known to suffer from the curse of dimensionality, requiring $\exp(\Omega(d))$ samples for empirical approximation (Paty et al., 2019).

The dual incorporates potentials $\mu$0: $\mu$1 Brenier's theorem shows that for $\mu$2, the optimal map is the gradient of a convex "Brenier potential" $\mu$3: $\mu$4.

2. Regularization Strategies in Kantorovich OT

Regularization modifies the original problem by penalizing the transport plan $\mu$5 (or the dual potentials) via convex functionals, constraints, or entropic terms. Key motivations are: removing singularity, accelerating algorithms, ensuring existence/uniqueness, promoting sparsity or smoothness, and improving generalization with finite samples.

2.1 Entropic Regularization

The most widely used approach adds negative Shannon entropy: $\mu$6 Yielding strictly convex objectives and unique, diffuse solutions, this leads to the celebrated Sinkhorn algorithm (matrix scaling in the discrete case) (Peyré, 10 May 2025, Peyré et al., 2018, Tupitsa et al., 2022). The Gibbs kernel $\mu$7 enables fast update rules, with per-iteration cost $\mu$8 in the discrete $\mu$9 case: $\nu$0 Algorithmic and theoretical studies (Tupitsa et al., 2022, Kassraie et al., 2024) show:

• $\nu$1 recovers unregularized OT (but yields ill-condition numerics), $\nu$2 gives maximal-entropy (independent coupling).
• Faster convergence and improved sample complexity (polynomial in $\nu$3) compared to the unregularized setting.

2.2 Other Strongly Convex Regularizers

The regularization term can be any strictly convex $\nu$4: $\nu$5 Quadratic regularization ($\nu$6) induces sparsity in the optimal plan (Lorenz et al., 2019, Lorenz et al., 2019), unlike entropic regularization, which produces fully dense couplings. Quadratic-regularized OT admits efficient quasi-Newton and block coordinate algorithms, and closed semismooth duals (piecewise linear-quadratic) leading to rapid convergence for moderate problem sizes.

Regularizers based on Orlicz norms (Young's functions $\nu$7) generalize this framework, include the entropy and various power-like penalties, and admit Fenchel duals and theoretical guarantees (Lorenz et al., 2019, Lorenz et al., 2020).

2.3 Regularization via Structural Constraints

Imposing explicit structural constraints further shapes the solution:

• Cardinality constraints: Control the support of $\nu$8 directly (e.g., at most $\nu$9 nonzeros per column) (Liu et al., 2022). This enables precise plan sparsity, crucial for applications to mixture-of-experts, clustering, and routing.
• Strong convexity and smoothness of the Brenier potential: By enforcing $X, Y$0-strong convexity and $X, Y$1-smoothness (Hessian bounded between $X, Y$2 and $X, Y$3), one regularizes the Monge map, ensuring robust estimation and controlled Lipschitz constants (Paty et al., 2019).

2.4 Divergence Regularization

Recent works interpolate between different penalties:

• Rényi. The $X, Y$4-Rényi divergence regularizes OT, with $X, Y$5 yielding KL/entropic regularization and $X, Y$6 recovering classical OT. This interpolates smoothly between sparse and dense plans, often out-performing KL and Tsallis regularizations (Bresch et al., 2024).
• Quantum entropy. In tensor-valued OT, the von Neumann entropy regularizes the coupling; the resulting quantum Sinkhorn algorithm generalizes classical entropic regularization to the nonscalar, PSD-matrix regime (Peyré et al., 2016).

3. Dual Formulations and Algorithms

The introduction of regularization modifies both the primal and the dual: $X, Y$7

$X, Y$8

where $X, Y$9 is the Fenchel conjugate. For entropic regularization, this dual is smooth and high-dimensional, enabling the use of block coordinate ascent, Nesterov acceleration, and mirror descent (An et al., 2021, Tupitsa et al., 2022, Bresch et al., 2024). For quadratic regularization, the dual is strongly concave and admits efficient semismooth Newton and coordinate-descent solvers (Lorenz et al., 2019, Lorenz et al., 2019).

Structural regularizations, e.g., on plan sparsity, nonconvex $$\min_{\pi\in\Pi(\mu,\nu)} \int_{X\times Y} c(x,y) \, d\pi(x,y)$$0-type constraints, still lead to dual/semi-dual problems that are tractable, often via closed-form projections onto the constraint set (Liu et al., 2022).

In high-dimensional continuous settings, regularization on the Brenier potential leads to quadratic-constrained quadratic programs (QCQPs) separable across spatial clusters/block partitions (Paty et al., 2019).

4. Theoretical Guarantees and Sample Complexity

Regularization profoundly alters statistical properties:

• Existence and uniqueness: Strict convexity/strong regularization yield unique solutions even when classical OT admits multiple (possibly singular) optima (Clason et al., 2019, Lorenz et al., 2019, Lorenz et al., 2020).
• $$\min_{\pi\in\Pi(\mu,\nu)} \int_{X\times Y} c(x,y) \, d\pi(x,y)$$1-convergence: As the regularization vanishes, regularized solutions converge to the unregularized Kantorovich plan (Lorenz et al., 2020, Clason et al., 2019).
• Distortion and regularity bounds: Regularization on the Brenier map enforces Lipschitz constants, thus bounding distortion in high dimension (Paty et al., 2019).
• Sample complexity: Regularized estimators (notably, with smooth strongly convex potentials or entropy) admit polynomial (in $$\min_{\pi\in\Pi(\mu,\nu)} \int_{X\times Y} c(x,y) \, d\pi(x,y)$$2) convergence rates, avoiding the curse of dimensionality (Paty et al., 2019, Kassraie et al., 2024, Tupitsa et al., 2022).
• Interpolation properties: Rényi and Tsallis-regularized OT frameworks admit rigorous interpolation between unregularized OT and the entropy-regularized regime, with explicit control of plan structure by a divergence parameter (Bresch et al., 2024).

5. Algorithmic Implementations and Computational Complexity

K-OT regularization enables scalable algorithms:

• Sinkhorn iteration: The mainstay for entropic OT, converges linearly and is massively parallelizable; cost per iteration $$\min_{\pi\in\Pi(\mu,\nu)} \int_{X\times Y} c(x,y) \, d\pi(x,y)$$3 (Peyré, 10 May 2025, Peyré et al., 2018).
• Mirror descent and accelerated methods: Useful for smooth regularizers, e.g., with quadratic penalty or Tsallis/Rényi divergence (An et al., 2021, Bresch et al., 2024).
• QCQP solvers: Required for smooth strongly-convex Brenier potential regularization; block-separable in practice (Paty et al., 2019).
• Sparse and structured plans: K-OT with sparsity constraints admits dual and semi-dual forms where the gradient computation requires only top-$$\min_{\pi\in\Pi(\mu,\nu)} \int_{X\times Y} c(x,y) \, d\pi(x,y)$$4 selection per column, scaling as $$\min_{\pi\in\Pi(\mu,\nu)} \int_{X\times Y} c(x,y) \, d\pi(x,y)$$5 per iteration (Liu et al., 2022).
• Distributed and barycentric extensions: Strong regularization enables efficient decentralized and parallel barycenter algorithms (Tupitsa et al., 2022).

Tables of regularizer properties, dual forms, and algorithmic complexity are available in the cited literature.

6. Applications and Empirical Performance

Kantorovich OT regularization is critical in modern machine learning and statistics:

• Domain adaptation: Regularized Monge map estimation achieves superior out-of-sample domain alignment and transfer accuracy (Paty et al., 2019).
• Generative modeling: Sinkhorn divergences and regularized Wasserstein losses are used widely in GANs, VAEs, and latent-variable models (Peyré, 10 May 2025).
• Sparse assignment problems: K-OT with explicit cardinality constraints excels in mixture-of-experts models and computational vision tasks (Liu et al., 2022).
• Robustness and metric learning: The ground cost-adversarial interpretation shows that regularization endows OT distances with intrinsic robustness to metric perturbations, critical for adversarial defense and distributional robustness (Paty et al., 2020).
• Quantile and ranking operations: Differentiable sorting/ranking leverages K-OT regularization for gradient-based pipelines (Cuturi et al., 2019).
• Tensor field and geometric data: Quantum-OT enables OT with matrix-valued data, powering geometric morphing, barycenters, and biomedical imaging (Peyré et al., 2016, Bercu et al., 2024).

Empirical benchmarks consistently show regularization improves both sample efficiency and algorithmic speed, with the plan's structure and regularity tunable to application needs.

7. Parameter Selection and Practical Recommendations

Regularization parameters ($$\min_{\pi\in\Pi(\mu,\nu)} \int_{X\times Y} c(x,y) \, d\pi(x,y)$$6, $$\min_{\pi\in\Pi(\mu,\nu)} \int_{X\times Y} c(x,y) \, d\pi(x,y)$$7, $$\min_{\pi\in\Pi(\mu,\nu)} \int_{X\times Y} c(x,y) \, d\pi(x,y)$$8, $$\min_{\pi\in\Pi(\mu,\nu)} \int_{X\times Y} c(x,y) \, d\pi(x,y)$$9, $\Pi(\mu,\nu)$0, or $\Pi(\mu,\nu)$1 as appropriate) must be chosen to balance fidelity, numerical stability, computational speed, and statistical accuracy:

• Entropic regularization: Choose $\Pi(\mu,\nu)$2 as a small fraction of the mean transport cost; decrease for higher fidelity but at increased computational cost (Tupitsa et al., 2022).
• Quadratic/$\Pi(\mu,\nu)$3: Cross-validate $\Pi(\mu,\nu)$4 to match model complexity to sample size and desired sparsity.
• Structural bounds: For smoothness/convexity ($\Pi(\mu,\nu)$5) or support ($\Pi(\mu,\nu)$6), use prior knowledge or hold-out risk minimization (Paty et al., 2019, Liu et al., 2022).
• Rényi divergence: $\Pi(\mu,\nu)$7–$\Pi(\mu,\nu)$8 yields tighter OT plans while maintaining numerical stability (Bresch et al., 2024).
• Continuation/annealing: Solve for large regularization, then gradually decrease (warm start) (Tupitsa et al., 2022, Kassraie et al., 2024).

Choice of parameters and regularizer is application-specific, with cross-validation or debiased divergences (e.g., Sinkhorn) providing principled guidance.

---

References:

• "Regularity as Regularization: Smooth and Strongly Convex Brenier Potentials in Optimal Transport" (Paty et al., 2019)
• "Quadratically regularized optimal transport" (Lorenz et al., 2019)
• "Orlicz-space regularization for optimal transport and algorithms for quadratic regularization" (Lorenz et al., 2019)
• "Interpolating between Optimal Transport and KL regularized Optimal Transport using Rényi Divergences" (Bresch et al., 2024)
• "Sparsity-Constrained Optimal Transport" (Liu et al., 2022)
• "Optimal Transport for Machine Learners" (Peyré, 10 May 2025)
• "Computational Optimal Transport" (Peyré et al., 2018)
• "Entropic regularization of continuous optimal transport problems" (Clason et al., 2019)
• "Regularized Optimal Transport is Ground Cost Adversarial" (Paty et al., 2020)
• "Quantum Optimal Transport for Tensor Field Processing" (Peyré et al., 2016)
• "Imaging with Kantorovich-Rubinstein discrepancy" (Lellmann et al., 2014)
• "Numerical Methods for Large-Scale Optimal Transport" (Tupitsa et al., 2022)
• "Efficient Optimal Transport Algorithm by Accelerated Gradient descent" (An et al., 2021)
• "Regularized estimation of Monge-Kantorovich quantiles for spherical data" (Bercu et al., 2024)
• "ENOT: Expectile Regularization for Fast and Accurate Training of Neural Optimal Transport" (Buzun et al., 2024)
• "Progressive Entropic Optimal Transport Solvers" (Kassraie et al., 2024)
• "Differentiable Ranks and Sorting using Optimal Transport" (Cuturi et al., 2019)