Kernel Quantile Embeddings

• Kernel Quantile Embeddings (KQEs) are an advanced framework that generalizes kernel mean embeddings by incorporating directional quantile information.
• KQEs induce flexible statistical metrics that aggregate quantile discrepancies, recovering kernelized sliced Wasserstein distances under weaker kernel conditions.
• Empirical benchmarks show that KQEs offer near-linear complexity and competitive performance in two-sample testing compared to traditional MMD methods.

Kernel Quantile Embeddings (KQEs) extend the classical approach of embedding probability distributions into reproducing kernel Hilbert spaces (RKHS) by encapsulating generalized quantile information, rather than mere mean structure. This framework generalizes kernel mean embeddings and facilitates the construction of probability metrics with weaker kernel requirements than maximum mean discrepancy (MMD), recovers kernelized forms of sliced Wasserstein distances, and admits computationally efficient estimators with near-linear complexity. KQEs provide a rich, directionally parameterized representation of distributions, demonstrating theoretical guarantees and competitive empirical performance in two-sample testing and related statistical tasks (Naslidnyk et al., 26 May 2025).

1. Mathematical Framework and Definition

Given a measurable, continuous, bounded kernel $k\!:\! X\times X \rightarrow \mathbb{R}$ on a Borel space $X$, let $\mathcal{H}$ denote the RKHS with feature map $\psi(x) = k(x,\,\cdot)$ and inner product $\langle \cdot, \cdot \rangle_{\mathcal{H}}$. For a probability measure $P$ on $X$, the classical kernel mean embedding is $$\mu_P = \mathbb{E}_{X\sim P} [\psi(X)] \in \mathcal{H}$$; $k$ is mean-characteristic if $P \mapsto \mu_P$ is injective.

Quantiles are generalized to RKHS via directional projections. For $$u \in S_{\mathcal{H}} = \{ u \in \mathcal{H} : \|u\|_{\mathcal{H}} = 1 \}$$, define the pushforward measure $u_{\sharp}P$ on $\mathbb{R}$ by $u(x) = \langle u, \psi(x) \rangle_{\mathcal{H}}$. The one-dimensional $\alpha$-quantile of $u_{\sharp}P$ is denoted $\rho_{u_{\sharp}P}^\alpha$. The kernel quantile embedding (KQE) for $P$, quantile level $\alpha \in [0,1]$, along direction $u \in S_{\mathcal{H}}$, is

$$\rho_P^{\alpha,u}(\cdot) = \rho_{u_{\sharp}P}^\alpha \; u(\cdot) \in \mathcal{H},$$

or equivalently $$\rho_P^{\alpha,u}(x) = \rho_{u_{\sharp}P}^\alpha \, \langle u, \psi(x) \rangle_{\mathcal{H}}$$. The full KQE of $P$ is the collection $$Q_k(P) = \{\, \rho_P^{\alpha,u} : \alpha \in [0,1],\, u \in S_{\mathcal{H}}\,\}$$ (Naslidnyk et al., 26 May 2025).

2. Associated Probability Metrics

KQEs induce a family of statistical distances aggregating quantile discrepancies over directions and quantile levels. Fix a probability measure $\nu$ on $[0,1]$ to weight quantile levels. For two measures $P, Q$ and direction $u \in S_{\mathcal{H}}$, define the $L^p$-quantile difference

$$\tau_p(P, Q; \nu, u) = \left( \int_0^1 \| \rho_P^{\alpha,u} - \rho_Q^{\alpha,u} \|_\mathcal{H}^p\; \nu(d\alpha) \right)^{1/p}.$$

To aggregate across $S_{\mathcal{H}}$, define:

• Expected KQD (e-KQD):

$$\text{e-KQD}_p(P, Q; \nu, \gamma) = \left( \mathbb{E}_{u \sim \gamma} [\, \tau_p(P, Q; \nu, u)^p ] \right)^{1/p},$$

where $\gamma$ is a probability measure on $S_{\mathcal{H}}$.

• Supremum KQD (sup-KQD):

$$\text{sup-KQD}_p(P, Q; \nu) = \left( \sup_{u \in S_{\mathcal{H}}} \tau_p(P, Q; \nu, u)^p \right)^{1/p}.$$

These KQD metrics subsume kernel mean discrepancies and recover kernelized variants of (max-)sliced Wasserstein distances. Specifically, for $X\subset \mathbb{R}^d$, linear kernel $k(x,y) = x^\top y$, Lebesgue $\nu$, and uniform $\gamma$ on $S^{d-1}$, e-KQD coincides with the sliced Wasserstein distance $SW_p(P, Q)$ and sup-KQD with max-sliced Wasserstein (Naslidnyk et al., 26 May 2025).

3. Theoretical Guarantees and Properties

Assume: (A1) $X$ is Hausdorff, separable, and $\sigma$-compact; (A2) $k$ is continuous and separating ($k(x, \cdot)\neq k(y, \cdot)$ for $x\neq y$).

• Quantile-characteristic Kernels: If $k$ meets A1 and A2, then the map $P\mapsto Q_k(P)$ is injective (Cramér–Wold in RKHS), so $k$ is quantile-characteristic. Every mean-characteristic kernel is quantile-characteristic, but not conversely.
• Metric Properties: With $\nu$ of full support on $[0,1]$, sup-KQD$_p$ defines a metric on $\mathcal{P}_X$. If $\gamma$ also has full support on $S_{\mathcal{H}}$, then e-KQD$_p$ is a metric.
• Finite-Sample Consistency: For data $x_{1:n}\sim P$, let $P_n = \frac{1}{n}\sum\delta_{x_i}$. The empirical directional quantile $$\rho_{P_n}^{\alpha,u}(x) = [u(x_{1:n})]_{\lceil \alpha n \rceil} \, u(x)$$ (with $[\cdot]_j$ denoting the $j$-th order statistic) converges in $\mathcal{H}$-norm to $\rho_P^{\alpha,u}$ at $O_p(n^{-1/2})$ rate, under mild $u_{\sharp}P$ density assumptions. Similarly, empirical e-KQD estimation converges at rate $O(l^{-1/2} + n^{-1/2})$ for $l$ Monte Carlo directions and sample size $n$, under regularity conditions on $\nu$ and kernel moments (Naslidnyk et al., 26 May 2025).

4. Algorithmic Implementation and Computational Complexity

Estimating Directional Quantiles: Given data $x_{1:n}$, for $u\in S_{\mathcal{H}}$, compute projections $$u(x_i) = \langle u,\,\psi(x_i)\rangle_{\mathcal{H}}$$, sort, and set $$\rho_{P_n}^{\alpha,u}(\cdot) = [u(x_{1:n})]_{\lceil \alpha n \rceil} u(\cdot)$$.

Sampling Directions: Uniform sampling on $S_{\mathcal{H}}$ is undefined in infinite-dimensional $\mathcal{H}$. Instead, choose a reference measure $\xi$ on $X$, define $C[f](x) = \int k(x,y)f(y)\xi(dy)$, sample $f\sim N(0,C)$, and set $u = f/\|f\|_\mathcal{H}$. A finite-sample proxy uses $z_{1:m}\sim \xi$, empirical covariance $C_m$, and random $\lambda_j \sim N(0,1)$: $f(x) = (1/\sqrt{m})\sum_{j=1}^m \lambda_j k(x,z_j)$.

e-KQD Estimator Workflow:

• For $i=1,\ldots,l$ directions: sample $z_{1:m}\sim\xi$, $\lambda_{1:m}\sim N(0,I_m)$; form $f_i$, normalize to $u_i$; sort $u_i(x_{1:n})$, $u_i(y_{1:n})$; compute $$\tau_{p,i}^p = \sum_{j=1}^n |[u_i(x)]_j - [u_i(y)]_j|^p f_\nu(j/n)$$.
• Return $(1/l \sum_i \tau_{p,i}^p)^{1/p}$.

Complexity Table:

Step	Complexity	Comment
$f_i(x_{1:n})$, $f_i(y_{1:n})$	$O(nm)$	$m$ kernel terms
Norm $\|f_i\|_\mathcal{H}}$ (Gram)	$O(m^2)$	$m \times m$ matrix
Sorting	$O(n\log n)$	Each direction
Total for $l$ directions	$O(l\cdot \max\{nm,\,m^2,\,n\log n\})$

Selecting $l=m=O(\log n)$ yields total cost $O(n\log^2 n)$ (“near-linear”) (Naslidnyk et al., 26 May 2025).

5. Empirical Results and Benchmarks

Two-sample testing is performed via permutation thresholding of the test statistic, at significance level $0.05$, comparing e-KQD and sup-KQD to quadratic MMD (U-statistic), linear MMD, MMD–Multi (incomplete U-statistic), sliced Wasserstein, and Sinkhorn.

Key experimental findings:

• Power decay under increasing dimension ($d$ up to $512$): e-KQD exhibits the most gradual test power decay, outperforming all fast MMD approximations.
• Laplace vs Gaussian in $\mathbb{R}$, identical first-two moments, polynomial kernel (degree 3): MMD fails as the kernel is not mean-characteristic, but all KQDs succeed, indicating that the kernel is quantile-characteristic.
• Galaxy MNIST ($12288$ dims): e-KQD-centered performs similarly to MMD, while near-linear e-KQD and sup-KQD outperform MMD–Multi.
• CIFAR-10 vs CIFAR-10.1 ($3072$ dims): near-linear KQDs significantly outperform fast MMD approximations at comparable computational cost (Naslidnyk et al., 26 May 2025).

6. Connections, Interpretation, and Extensions

KQEs systematically extend kernel mean embeddings by substituting the mean with the comprehensive quantile structure along arbitrary directions in $\mathcal{H}$. Under mild kernel conditions, KQEs uniquely characterize probability distributions, generalizing the role of mean-characteristic kernels. The induced KQD metrics interpolate between kernel mean discrepancy and (max-)sliced Wasserstein distances, providing a flexible continuum of statistical distances for high-dimensional inference. The near-linear time estimators and favorable empirical results suggest practical utility for large-scale hypothesis testing and related statistical machine learning tasks (Naslidnyk et al., 26 May 2025). A plausible implication is the potential for KQEs to serve as a universal nonparametric tool when classical kernel mean embeddings are insufficient, especially under non mean-characteristic kernels.