--- title: Wasserstein Distance-Preserving Embeddings url: https://www.emergentmind.com/topics/wasserstein-distance-preserving-embeddings type: topic --- # Wasserstein Distance-Preserving Embeddings A Wasserstein distance-preserving embedding is a mapping from a space of probability measures (distributions) equipped with the $p$-Wasserstein metric into a Euclidean or structured latent space such that the Euclidean (or other) geometry of embeddings closely approximates the original Wasserstein geometry. Unlike kernel mean embeddings which map distributions to points in a Hilbert space by averaging, Wasserstein embeddings directly encode optimal transport-based geometry—capturing structural, multimodal, or hierarchical relationships between distributions that kernel approaches often miss. Such embeddings facilitate scalable, distance-aware learning and analysis of datasets where each object is a probability distribution (e.g., histograms, point clouds, sequence distributions, or sets of images). ## 1. Formal Definition and Theoretical Foundations Let $\mathcal{P}_p(X)$ be the space of Borel probability measures with finite $p$-th moments on a Polish metric space $(X, d_X)$. The $p$-Wasserstein distance is \[ W_p(\mu, \nu) = \left( \inf_{\pi \in \Pi(\mu, \nu)} \int_{X\times X} d_X(x,y)^p\, d\pi(x,y) \right)^{1/p}, \] where $\Pi(\mu,\nu)$ is the set of couplings with marginals $\mu, \nu$. A distance-preserving or isometric embedding $\Phi$ is a map $\Phi:\mathcal{P}_p(X) \to \mathbb{R}^m$ such that \[ \|\Phi(\mu) - \Phi(\nu)\| \approx W_p(\mu, \nu), \quad \text{for all }\mu, \nu\in\mathcal{P}_p(X). \] Sometimes the mapping is only bi-Lipschitz with distortion $C\geq 1$: \[ \frac{1}{C}W_p(\mu, \nu) \leq \|\Phi(\mu) - \Phi(\nu)\| \leq C\, W_p(\mu, \nu). \] For $p=1$ on a finite metric space $X$, it is known that if $X$ admits a stochastic embedding into metric trees with distortion $D$, then $(\mathcal{P}_1(X), W_1)$ bi-Lipschitz embeds into $\ell^1$ with the same distortion $D$ using the Evans–Matsen formula, which enables explicit, linear-complexity isometric embeddings into $\ell^1$ [2110.02115]. More generally, for $p>1$, perfect isometry is not possible in general, but embeddings with small distortion exist for large finite metrics [1905.03329]. ## 2. Canonical Construction Approaches ### 2.1 Template-based Wasserstein Embeddings The template (or dissimilarity) embedding constructs a map from a probability measure $\mu$ to a vector of Wasserstein distances to a selected set of template distributions $T_1,...,T_m$: \[ \phi_W(\mu) := \frac{1}{M} \left[W_p(\mu, T_1), \ldots, W_p(\mu, T_m)\right]^T \in [0,1]^m, \] where $M$ is a normalization constant (e.g., a bound on $W_p$). The quality of the embedding—such as linear separability for classification—follows from the "good dissimilarity" theory, which guarantees that for sufficient $m$, the resulting feature cloud can be separated with low error by a linear classifier provided Wasserstein is a good dissimilarity for the task [1803.00250]. The template set can be chosen by random sampling, clustering (e.g., $k$-means in an RKHS), or heuristically. ### 2.2 Metric MDS with Wasserstein Geometry Classical multidimensional scaling (MDS) is used to construct embeddings such that Euclidean distances align with a given dissimilarity matrix. For distributions $\{\mu_1,...,\mu_N\}$, define the matrix $D_{ij} = W_p(\mu_i,\mu_j)$ (or its square). The MDS embedding seeks $\{z_i\}$ such that \[ \min_{z_1, ..., z_N} \sum_{i1$, exact isometry is unattainable globally; for finite $n$-point metrics, $W_p$-spaces can embed with distortion $O(\log n)$ (universality, metric theory [1905.03329]). - For the template embedding, the margin and linear separability of the resulting feature cloud are controlled by the number of templates and the alignment of Wasserstein geometry with the true task classes [1803.00250]. ### 3.2 Sample Complexity and Concentration - The number of templates $m$ required for faithful (low error, high margin) embedding is $O((M/\gamma)^2\log n)$, with $M$ a Wasserstein upper bound and $\gamma$ the separation margin. - When working with empirical measures $\hat\mu$ estimated from $N$ samples, control over the error $\left| W_p(\mu, \hat\mu) - W_p(\mu', \hat\mu') \right|$ scales as $g_1(K,N,\eta,d)$, with $N=O(\epsilon^{-d/p}\log(1/\delta))$ for $W_p$ in $d$ dimensions [1803.00250, 2302.07373]. ### 3.3 Approximation Error Induced by Compression/Linearization Using linearized optimal transport (LOT) instead of pairwise OT distances introduces an additive distortion $\tau_2$; the embedding error is upper-bounded by the combined error from linearization, regularization (if using Sinkhorn), and finite-sample estimation—all controlled with explicit high-probability inequalities [2302.07373]. For exact translation or scaling families, zero loss is achievable. ## 4. Algorithmic Realizations ### 4.1 Discrete and Entropic OT Computation - Discrete OT for histograms with $s$ points can be solved via network simplex in $O(s^3 \log s)$, or with Sinkhorn regularization $O(s^2 / \lambda^2)$ per pair, both with GPU acceleration options [1803.00250]. - For empirical Gaussian measures, $W_2$ can be computed in closed form (Bures metric, $O(d^3)$ via eigendecomposition) [1803.00250, 2205.07531]. ### 4.2 Efficient Embedding Construction - Template embedding: $O(m n)$ matrix construction followed by linear SVM training. - MDS-based approaches: $O(N^2)$ pairwise OT computations and $O(N^3)$ eigendecomposition; LOT-based approaches reduce pairwise OT to $O(N)$ [2302.07373]. - Neural methods: Training cost dominated by batch size, OT/sinkhorn computation per pair, and encoder/decoder forward/backpass. Batch-based co-embedding and Sinkhorn backpropagation scale efficiently [1905.03329, 2404.09411]. ## 5. Practical Applications and Empirical Outcomes ### Table: Selected Wasserstein Distance-Preserving Embedding Methods and Their Applications | Method (Paper) | Principle | Typical Applications | |--------------------------------------------|-------------------------------|-------------------------------------------------| | Template Wasserstein Embedding [1803.00250] | Distances to reference measures | Distribution classification (scene, point-clouds) | | MDS/Wassmap [2204.06645] | MDS on $W_p$ distance matrix | Image manifolds, synthetic translations/dilations | | LOT Wassmap [2302.07373] | Linearized OT | Large-scale manifold discovery, high-dim OT | | Neural Metric Embedding [1710.07457, 2404.09411] | Deep metric learning | Fast OT-based similarity (images, point clouds) | | Entropic Wasserstein PCA [2303.05119] | Subspace projection via OT | Gene expression analysis, cluster structure | | Wasserstein t-SNE [2205.07531] | Low-dim visualization via $W_2$| Embedding of hierarchical/grouped data | | Stochastic Vision Transformers [2311.18645]| OT-aware attention | Image SSL, OOD detection, calibration | | Cantor–Wasserstein [2206.04198] | Symbolic sequence embeddings | Predictive state geometry, sequence clustering | Empirical studies show that Wasserstein distance-preserving embeddings outperform kernel mean embeddings and other Euclidean geometry-based approaches on real and synthetic distributional classification, manifold learning, and representation tasks—especially in the presence of non-Euclidean or hierarchical structure. For example, in 3D point-cloud classification, Wasserstein template embedding achieves up to $97\%$ accuracy vs $92\%$ for kernel mean methods [1803.00250]. On synthetic translation/dilation manifolds, Wasserstein MDS recovers latent coordinates up to rigid motion with nearly zero stress [2204.06645]. Neural approaches accelerate OT computations by $10^3$–$10^5\times$ in high-throughput settings [1710.07457]. ## 6. Extensions, Limitations, and Open Challenges Theoretical isometry is only achievable in narrow cases: $p=1$ (tree or Cantor geometry) or distributions restricted to translation/dilation subgroups. In general, $p>1$—especially for non-Gaussian and high-dimensional supports—embeddings unavoidably incur distortion, controlled by sample size, manifold curvature, and network/exemplar capacity. Approximate, scalable methods such as linearized OT, entropic regularization, and neural metric learning provide practical compromises, with distortion controlled empirically and in some cases theoretically bounded [2302.07373, 1905.03329]. For symbolic data, Cantor embeddings furnish a bi-Lipschitz mapping into $\mathbb{R}^d$ with uniform distortion constants, enabling effective clustering and visualization with 1D Wasserstein distances [2206.04198]. Current limitations include distortion/dimensional blowup for generic metrics, challenges in scaling to very high-dimensional or continuous measure spaces, and lack of universal approximation bounds for deep-neural OT embeddings away from locally-concentrated or tree-like metric spaces. Ongoing research investigates embedding universality, optimal reference measure selection for LOT and related approaches, explicit distortion rates in neural embeddings, and specialized architectures for structured domains (e.g., graphs, images, spatial-temporal processes). ## 7. Connections to Related Research Areas Wasserstein distance-preserving embeddings bridge optimal transport, metric/representation learning, and dimensionality reduction. They build on, and generalize, kernel mean embeddings (RKHS), classical MDS, and manifold learning, while exploiting the geometric and probabilistic structure of distributions. Applications span distributional supervised learning, OT-based clustering, visualization, generative modeling, robust and uncertainty-aware deep learning, as well as interpretable sequence and hierarchical data analysis. Significant synergies exist with Gromov–Wasserstein geometry for structural data comparison, with distributional regularization in Bayesian and deep models, and with recent advances in scalable OT computation—particularly in settings where computational tractability and geometric fidelity of distributional relationships are critical. Source: https://www.emergentmind.com/topics/wasserstein-distance-preserving-embeddings