Kernel Mean Embedding

• Kernel mean embedding is a technique that represents probability measures as elements in a reproducing kernel Hilbert space, capturing the full distributional information.
• It underpins methods such as maximum mean discrepancy for nonparametric hypothesis testing and conditional mean embedding for computing nonparametric conditional expectations.
• Its theoretical guarantees, variance-aware estimators, and scalable approximations (e.g., Nyström) make it practical for applications in functional data analysis, privacy, and quantum computation.

Kernel mean embedding (KME) is a functional analytic technique that represents probability measures as elements in a reproducing kernel Hilbert space (RKHS), thereby enabling the extension of kernel methods, originally designed for pointwise data, to directly operate on distributions. Central to nonparametric hypothesis testing, statistical inference on structured objects, and distributional representation in machine learning, KME underpins a broad array of modern kernel-based approaches and has motivated advances in both theory and scalable computation.

1. Mathematical Formulation and Properties

Given a separable topological space $\mathcal{X}$, a positive-definite kernel $k:\mathcal{X} \times \mathcal{X} \to \mathbb{R}$ with associated RKHS $\mathcal{H}_k$, and a Borel probability measure $P$ on $\mathcal{X}$ satisfying $\mathbb{E}_{P}[\sqrt{k(X, X)}] < \infty$, the kernel mean embedding $\mu_P$ is defined as

$$\mu_P := \mathbb{E}_{X \sim P}[k(\cdot, X)] \in \mathcal{H}_k.$$

This Bochner integral is characterized by the reproducing property: for any $f \in \mathcal{H}_k$,

$$\langle f, \mu_P \rangle_{\mathcal{H}_k} = \mathbb{E}_{X \sim P}[f(X)].$$

Hence, $k:\mathcal{X} \times \mathcal{X} \to \mathbb{R}$0 preserves all RKHS-representable expectations and can be interpreted as a "mean feature" of $k:\mathcal{X} \times \mathcal{X} \to \mathbb{R}$1.

An essential property is injectivity: if the kernel $k:\mathcal{X} \times \mathcal{X} \to \mathbb{R}$2 is characteristic on a class of measures, then the mapping $k:\mathcal{X} \times \mathcal{X} \to \mathbb{R}$3 is injective (i.e., $k:\mathcal{X} \times \mathcal{X} \to \mathbb{R}$4 implies $k:\mathcal{X} \times \mathcal{X} \to \mathbb{R}$5), so $k:\mathcal{X} \times \mathcal{X} \to \mathbb{R}$6 determines $k:\mathcal{X} \times \mathcal{X} \to \mathbb{R}$7, and the associated RKHS norm defines a metric on probability measures. Many common kernels, such as the Gaussian and Laplace RBFs on $k:\mathcal{X} \times \mathcal{X} \to \mathbb{R}$8, are characteristic—even on infinite-dimensional spaces such as $k:\mathcal{X} \times \mathcal{X} \to \mathbb{R}$9 or $\mathcal{H}_k$0 (Hayati et al., 2020, Simon-Gabriel et al., 2016).

For two probability measures $\mathcal{H}_k$1, the inner product and squared RKHS norm between their embeddings admit the closed-form

$\mathcal{H}_k$2

encoding global similarity of distributions via pairwise kernel evaluations (Hayati et al., 2020, Muandet et al., 2016). Empirical estimation from finite samples replaces population expectations by averages, yielding an estimator that converges in RKHS norm at the $\mathcal{H}_k$3 rate, which is both upper- and minimax lower-bounded (i.e., optimal and dimension-free) (Tolstikhin et al., 2016).

2. Statistical Inference, Maximum Mean Discrepancy, and Testing

The maximum mean discrepancy (MMD) is the RKHS distance between embeddings: $\mathcal{H}_k$4 For characteristic $\mathcal{H}_k$5, $\mathcal{H}_k$6 if and only if $\mathcal{H}_k$7, making it a metric on the space of measures (Hayati et al., 2020, Simon-Gabriel et al., 2016). Empirically, MMD is efficiently estimable via U-statistics, forming the basis for nonparametric two-sample tests, independence tests (e.g., HSIC for joint embeddings), and goodness-of-fit procedures (Muandet et al., 2016).

MMD-based inference proceeds by constructing test statistics—quantifying the discrepancy between model and observed (or between two samples)—and calibrating p-values via permutation or bootstrap, since the asymptotic distribution is often intractable, especially for complex or infinite-dimensional sample spaces (Hayati et al., 2020). MMD’s performance is robust, with type I error controlled and higher power relative to various alternatives in high-dimensional and functional-data settings.

For conditional distributions, the conditional mean embedding framework embeds $\mathcal{H}_k$8 as an RKHS-valued operator, allowing the computation of nonparametric conditional expectations and providing algebraic "kernel rules" (sum, product, Bayes) analogous to probability calculus (Muandet et al., 2016, Shimizu et al., 2024).

3. Computational Methods and Variance-Aware Estimation

Directly working with KMEs can be computationally burdensome for large $\mathcal{H}_k$9 due to the need to store and manipulate all $P$0 kernel sections $P$1. Nyström-type low-rank approximations address this by projecting the empirical mean embedding onto a subspace spanned by a randomly chosen subset of $P$2 data points. The Nyström estimator,

$P$3

interpolates the empirical embedding and, under mild spectral decay of the kernel covariance operator, achieves the same $P$4 statistical rate at reduced computational cost ($P$5) (Chatalic et al., 2022). Theoretical analysis delineates conditions on $P$6 to recover the minimax-optimal rate, and practical guidelines include choosing $P$7 for exponential eigenvalue decay.

Variance-aware estimation refines traditional concentration inequalities for empirical KMEs by replacing the worst-case variance bound with the intrinsic RKHS variance,

$P$8

which can be much smaller for concentrated $P$9 or large bandwidths. Empirical, unbiased estimators of $\mathcal{X}$0 from data yield sharper, fully empirical finite-sample deviation bounds, directly benefiting statistical power in testing and estimation (Wolfer et al., 2022).

4. Functional Data Analysis and Infinite-Dimensional Extensions

KME theory extends to probability measures over infinite-dimensional separable Hilbert spaces, enabling the treatment of functional data. For random elements in $\mathcal{X}$1 or higher-order function spaces, Gaussian and sum-product kernels remain characteristic, so that distributions of functional-valued data can be uniquely represented (Hayati et al., 2020).

Applications in functional data analysis (FDA) include:

• Function-on-scalar regression: pseudo-likelihood based on KME yields the OLS estimator, and MMD-based tests maintain nominal error with superior power compared to $\mathcal{X}$2-norm and elliptical-region alternatives.
• Functional one-way ANOVA: the KME/MMD approach yields closed-form statistics in terms of mean and covariance operators, outperforming $\mathcal{X}$3, F-type, and generalized permutation-based tests.
• Equality of covariance operators: pairwise MMD between multivariate/funtional Gaussians is available in closed-form via log-determinant expressions, facilitating efficient hypothesis testing of covariance structure.

Critically, all these FDA scenarios exploit KME’s ability to encode local “small-ball” concentration effects and handle high- or infinite-dimensional function spaces without resorting to problematic density definitions (Hayati et al., 2020).

5. Extensions: Algebraic Structure, Topology, and Generalizations

The mathematical structure of KME interlinks several concepts:

• Universality, characteristicness, and strict positive definiteness: For $\mathcal{X}$4 to induce injective KME it must be universal (RKHS dense in $\mathcal{X}$5), and these notions are equivalent on locally compact Hausdorff spaces—see (Simon-Gabriel et al., 2016).
• Topology: When $\mathcal{X}$6 is continuous and characteristic, the RKHS metric $\mathcal{X}$7 metrizes weak/narrow convergence of probability measures, and the Bochner spaces $\mathcal{X}$8 equipped with kernel mean embedding topology support both weak and strong formulations for stochastic kernels. This makes KMEs foundational for robust optimal control and stochastic policy learning, with explicit Hilbert-norm-based continuity and approximation bounds (Saldi et al., 19 Feb 2025).
• Algebra-valued generalizations: Extending beyond scalar-valued measures, KME in reproducing kernel Hilbert $\mathcal{X}$9-modules (RKHMs) or over von Neumann algebras allows representation and discrimination of matrix-valued or operator-valued measures, relevant for structured multivariate data, higher-order interactions, and quantum mechanical applications (Hashimoto et al., 2020, Hashimoto et al., 2021). Injectivity and universality properties have been generalized to these settings, enabling operator-theoretic versions of major kernel methodologies.

KME also admits closed-form expressions for a broad range of kernel-distribution pairs, including Gaussian, Matérn, Brownian, and polynomial kernels with uniform, Gaussian, or other standard distributions. Algebraic operations (product, mixture, pushforward, Stein kernels) facilitate the construction of new embeddings without recourse to fresh integration (Briol et al., 26 Apr 2025).

6. Applications in Inference, Privacy, and Learning

The representational utility of KME is central in:

• Two-sample and independence tests: Via MMD and Hilbert-Schmidt Independence Criterion (HSIC), powering nonparametric detection of discrepancies between distributions and independence in high or infinite dimensions (Muandet et al., 2016).
• Likelihood-free inference: The pseudo-likelihood $\mathbb{E}_{P}[\sqrt{k(X, X)}] < \infty$0 is a strictly proper scoring rule for characteristic kernels, supporting statistical model selection and estimation when classical likelihoods are unavailable (Hayati et al., 2020).
• Differential privacy: The RKHS-norm facilitates the release of synthetic KMEs consistent with privacy guarantees, enabling third parties to compute population-level statistics via linear functionals on the noised (and eventually consistent) embedding (Balog et al., 2017).
• Robust and Bayesian inference: Robust parameter estimation (e.g., under Huber contamination) leverages variance-aware KME rates for enhanced performance; Bayesian treatments learn kernel parameters via marginal pseudolikelihood, yielding full uncertainty quantification for the embedding and tuning via automatic relevance determination (Wolfer et al., 2022, Flaxman et al., 2016).
• Functional and nonlinear filtering: KME-based distributed nonlinear filters propagate and update probability distributions over state spaces by iterating the embedding and leveraging consensus strategies, with demonstrated superiority over classical cubature Kalman filters in state estimation scenarios (Guo et al., 2023).
• Quantum computation: Quantum mean embedding explicitly encodes the KME as a quantum state, theoretically enabling linear-time computation of kernel inner products (compared to quadratic scaling classically) for many distributional learning tasks (Kübler et al., 2019).

7. Outlook and Limitations

KMEs provide a rigorous, nonparametric, and theoretically grounded bridge between probability theory and kernel-based algorithms, powerful for high-dimensional, structured, or functional data. Open research directions include developing sharper variance-aware bounds under low-dimensional structures, fully adaptive kernel selection, scalable implementations for conditional and operator-valued embeddings, and deeper theoretical connections with optimal transport, measure theory, and topological learning (Wolfer et al., 2022, Saldi et al., 19 Feb 2025, Briol et al., 26 Apr 2025).

Limitations include the computational cost of fully empirical estimators (e.g., variance-aware bounds scale quadratically), the challenge of kernel and bandwidth selection in practice, and difficulties in high-dimensional output contexts for conditional mean embeddings. Work continues on scalable approximations (e.g., low-rank, sparse, and neural-parameterized variants), theoretical characterization under non-i.i.d. data, and robust privacy-preserving implementations.

---

References and Core Contributions

• (Hayati et al., 2020) foundational extension of KME/MMD to infinite-dimensional Hilbert spaces and functional data analysis
• (Wolfer et al., 2022) variance-aware estimation, tighter concentration bounds, robust parametric applications
• (Chatalic et al., 2022) Nyström approximation and scalable embedding construction
• (Simon-Gabriel et al., 2016) equivalence of universal, characteristic, and strictly PD kernels; metrization of weak convergence
• (Muandet et al., 2016) comprehensive review: conditional KME, sum/product/Bayes rules, applications in regression, independence testing, MCMC, RL
• (Briol et al., 26 Apr 2025) systematic collection of closed-form KME expressions; algebraic operations and practical library
• (Hashimoto et al., 2021, Hashimoto et al., 2020) extension to RKHM and non-commutative algebras
• (Guo et al., 2023) KME-based distributed nonlinear filtering
• (Balog et al., 2017) differential privacy guarantees via synthetic KME release
• (Saldi et al., 19 Feb 2025) rigorous topology for KME on stochastic kernels and implications for control-theoretic learning
• (Flaxman et al., 2016) Bayesian learning of kernel embeddings, shrinkage estimators, posterior uncertainty quantification
• (Kübler et al., 2019) quantum mean embedding and potential computational speedup

These results establish kernel mean embedding as a mathematically rigorous, practical, and far-reaching paradigm for nonparametric statistical analysis and machine learning with probability distributions.