---
title: Wasserstein Dependency Measures
url: https://www.emergentmind.com/topics/distance-maximizing-wasserstein-dependency-measures
type: topic
---

# Wasserstein Dependency Measures

Distance-maximizing Wasserstein dependency measures constitute a rigorous framework in which statistical dependence between random variables, groups, or more general random structures is quantified via optimal transport metrics. By formulating dependence as a distance—either from independence or from maximal dependence—in an appropriate metric geometry, these measures provide a conceptually unified and metrically sensitive approach to characterizing, estimating, and optimizing statistical relationships. These frameworks are instantiated in a diversity of contexts, including classical dependence assessment, unsupervised representation learning, reinforcement learning, robust statistics, Bayesian nonparametrics, and the geometric analysis of random measures.

## 1. Core Definitions and Formalism

Let $(X, Y)$ be random variables with joint law $P_{XY}$ on a Polish space $\mathcal X \times \mathcal Y$, and marginals $P_X$, $P_Y$. The $p$-Wasserstein distance between probability measures $\mu, \nu$ on a metric space $(\mathcal M, d)$ is
$$
W_{d,p}(\mu, \nu) = \left[ \inf_{\pi \in \Gamma(\mu, \nu)} \mathbb E_{(x, y) \sim \pi} d(x, y)^p \right]^{1/p}
$$
where $\Gamma(\mu,\nu)$ is the set of couplings with the given marginals.

### Distance to Independence and Maximal Dependence

- **Distance to Independence**: 
  $$
  D_\otimes(X, Y) = W_{d,p}(P_{XY}, P_X \otimes P_Y)
  $$
  $D_\otimes$ is zero if and only if $X$ and $Y$ are independent [2510.06034].

- **Distance to Maximal Dependence**: Given a set of “maximally dependent” couplings $\mathcal C_1$ (e.g., $Y=f(X)$ for measurable maps $f$), the distance is
  $$
  D_\mathcal{F}(X, Y) = \inf_{f \in \mathcal{F}} W_{d,p}(P_{XY}, \mathscr L(X, f(X)))
  $$
  and can be normalized as
  $$
  I_{\max}(X, Y) = 1 - \frac{D_\mathcal{F}(X, Y)}{\sup_{\pi \in \Gamma(P_X, P_Y)} W_{d, p}(\pi, \mathcal C_1)}
  $$
  which yields $I_{\max}(X, Y) = 1$ at maximal dependence (e.g., $Y = X$), and $0$ at independence [2510.06034].

- **Kantorovich–Rubinstein Duality**: For $p=1$, the Wasserstein distance admits dual formulation:
  $$
  W_{d, 1}(\mu,\nu) = \sup_{\text{Lip}(f) \leq 1} \left\{ \mathbb E_\mu[f] - \mathbb E_\nu[f] \right\}.
  $$

## 2. Theoretical Properties and Characterizations

- **Independence and Maximal Dependence**:
  - $D_\otimes(X, Y) = 0$ if and only if $P_{XY} = P_X \otimes P_Y$ (independence).
  - Normalized indices (e.g., $I_{\max}(X, Y)$, $I_W$ for random measures) attain $1$ exactly at maximal dependence and $0$ at independence [2109.06646, 2510.06034].
- **Metric Invariance**:
  - Wasserstein distances are invariant under isometries of the ground metric. In copula-based settings, indices can be made invariant under monotone reparametrizations of the marginals [2510.06034].
- **Robustness**:
  - Joint distances (e.g., $D_\otimes$) are Lipschitz continuous under weak convergence of probability measures; they respond linearly to small contamination with independent noise [2510.06034].
- **Sample Complexity**:
  - For empirical plug-in estimators, $D_\otimes$ achieves parametric rate $O(n^{-1/2})$ in low dimensions; rates degrade with ambient dimension $m$ as $O(n^{-\min(p,2)/(2m)})$ [2510.06034].

## 3. Optimization and Computation

### Convex and Dual Formulations

Distance-maximizing Wasserstein dependency measures often reduce to convex or saddle-point problems:

- **Piecewise-Algebraic Structure**: For discrete variables, the distance to independence is the solution to a bilevel linear program, whose minimizer has an explicit piecewise algebraic structure governed by the geometry of the Lipschitz polytope and its dual unit ball. For the case of a Segre variety independence model, the explicit solution is computable via duality and polyhedral geometry [2003.06725].
- **Numerical Quadrature**: For dependent random measures, the Wasserstein index of dependence $I_W$ reduces to a one-dimensional quadrature on tail integrals of the Lévy measures, making it practical for high-dimensional ($d \ge 2$) settings [2109.06646].

### Efficient Algorithms

- **Linear Programming**: For discrete measures with finite support, Wasserstein distances are computable via network-flow LPs; complexity is $O(N^3\log N)$ for $N$ atoms [2510.06034].
- **Entropic Regularization**: Sinkhorn’s algorithm solves regularized OT with complexity $O(N^2/ \varepsilon^2)$ per $\varepsilon$-accuracy [1901.08949, 2510.06034].
- **Frank–Wolfe with Eigen-Decomposition**: For subspace-robust Wasserstein distances, maximization over low-rank projections is carried out using Frank–Wolfe and eigen-decomposition, keeping computation tractable in moderate dimensions ($d \lesssim 300$) [1901.08949].

## 4. Specialized Distance-Maximizing Constructions

### Wasserstein Distance Maximizing Intrinsic Control

In reinforcement learning, the “Wasserstein Distance Maximizing Intrinsic Control” (WIC) objective,
$$
\max_\pi \mathbb E_\omega \left[ W_d^1 \big(\delta_{s_0}, \mu_\pi(\cdot \mid \omega, s_0)\big) \right]
$$
formalizes the goal of skill learning by maximizing the expected Wasserstein distance between the start-state Dirac measure and the skill-conditioned state visitation measure. This differs from mutual information–based methods (VIC/DIAYN) by explicitly incentivizing covering maximal state-space distance, rather than mere distinguishability. The dual is estimated via a neural 1-Lipschitz critic, which provides a shaped reward for policy optimization. Empirical results in various Atari environments demonstrate that WIC yields coverage and returns superior to KL-based diversity baselines, especially in environments with sparse or exploratory tasks [2110.15331].

### Subspace Robust Wasserstein Distances

Robustness and interpretability in high dimensions are enhanced by the subspace-robust Wasserstein (SRW) distance:
$$
SRW_k(\mu, \nu) = \left[ \min_{\pi \in \Pi(\mu, \nu)} \sum_{j=1}^k \lambda_j(V_\pi) \right]^{1/2}
$$
where $V_\pi$ is the second-moment matrix under coupling $\pi$ and $\lambda_j$ its ordered eigenvalues. By focusing on the $k$ largest variance directions, SRW down-weights noise and reveals the intrinsic dimensionality of dependence. It induces an increasing and concave $k \mapsto SRW_k^2(\mu, \nu)$, with an observable “elbow” at the effective subspace dimension of dependency [1901.08949].

### Wasserstein Dependency Measures for Representation Learning

For unsupervised representation learning, replacing the KL-divergence in mutual information objectives with the Wasserstein distance yields the Wasserstein Dependency Measure (WDM):
$$
I_{\mathcal W}(X; Y) = W_1(p(x, y), p(x)p(y))
$$
Practical estimation leverages 1-Lipschitz neural critics with gradient penalty. Unlike KL-based bounds, which suffer from exponential sample complexity in high mutual information, WDM-based estimation provides stable and complete representation learning, as confirmed in multi-factor synthetic and image-based benchmark tasks [1903.11780].

## 5. Wasserstein Dependence Indices for Random Measures

The Wasserstein index of dependence for random measures, $I_W$, is defined for infinitely active, completely random vectors (CRVs) on a Polish space, parameterized by Lévy measures. Given $\nu$ (the joint Lévy measure), $\nu^\parallel$ (comonotonic dependence, i.e., on the diagonal), and $\nu^\perp$ (independence, i.i.d.), the index is
$$
I_W(\bm{\tilde\mu}) = 1 - \frac{W_*(\nu, \nu^\parallel)^2}{W_*(\nu^\perp, \nu^\parallel)^2}
$$
where $W_*$ is an extended Wasserstein metric between possibly infinite-mass Lévy measures, allowing coupling mass to/from the null vector. $I_W$ uniquely characterizes independence ($I_W=0$) and maximal dependence ($I_W=1$), and is jointly sensitive to $d \ge 2$ random measures without reduction to pairwise metrics. Numerical computation involves one-dimensional quadratures over transformations of the Lévy measure’s marginal and sum distributions, enabling practical criterion-based tuning of dependence in prior specification and fair comparison of Bayesian nonparametric models [2109.06646].

## 6. Practical Applications and Empirical Results

Applications of distance-maximizing Wasserstein dependency measures include:

- **Skill discovery and exploration in RL**: Wasserstein-based objectives provide effective intrinsic control mechanisms, outperforming mutual-information approaches in diverse environments [2110.15331].
- **Robust high-dimensional inference**: Subspace robust distances stabilize dependency measurement and clustering under noise and irrelevant features [1901.08949].
- **Representation learning**: Wasserstein dependency objectives yield more expressive features for downstream prediction, especially when standard mutual information maximization underfits due to insufficient sample complexity [1903.11780].
- **Random measure modeling in Bayesian nonparametrics**: The Wasserstein index of dependence enables prior specification and model comparison on a common dependence scale, confirmed by posterior comparisons when priors are matched for $I_W$ [2109.06646].
- **Algebraic and geometric study of dependence**: In discrete models, the explicit polyhedral and algebraic structure of the distance-to-independence functional allows detailed combinatorial and analytic investigation of dependency [2003.06725].
- **General statistical dependency measure**: Wasserstein-based indices systematically generalize classical correlation by quantifying both weak and strong, linear and nonlinear, dependence across a broad range of metric spaces [2510.06034].

## 7. Limitations and Open Directions

Challenges include:
- The need to solve nontrivial saddle point problems or to navigate the combinatorial complexity of polyhedral structures in large finite spaces.
- The bias-variance tradeoff in entropic regularization and the selection of regularization parameters.
- The enforcement of strict 1-Lipschitzness in deep neural critics remains practically limited; improved regularization techniques are sought.
- Dependence indices are sensitive to the choice of ground metric $d$, which must be defensibly specified for application domains.
- Computation of the full algebraic structure of dependency measures becomes demanding at scale, and further work seeks to optimize such algorithms and assess empirical degeneracy in practice [1901.08949, 2003.06725, 1903.11780, 2109.06646, 2510.06034].

Distance-maximizing Wasserstein dependency measures thus provide a unified, flexible, and metrically faithful approach to quantifying and optimizing dependence in diverse statistical and machine learning contexts, underpinned by principle, computable convex duality, and distinct geometric and probabilistic interpretations.

Source: https://www.emergentmind.com/topics/distance-maximizing-wasserstein-dependency-measures