Kernel Mean Embedding (KME)

• Kernel Mean Embedding (KME) is a framework that represents probability distributions as elements in a reproducing kernel Hilbert space (RKHS).
• It enables nonparametric methods such as two-sample testing, regression, integration, and control by leveraging geometric operations in the RKHS.
• KME facilitates scalable computation and advanced inference techniques, including characteristic kernel analysis, sparse approximations, and Bayesian regularization.

Kernel mean embedding (KME) is the representation of a probability measure $P$ by the RKHS mean element

$$\mu_P := \int_{\mathcal X} k(\cdot,x)\,dP(x)=\mathbb E_{X\sim P}[\phi(X)],$$

where $k$ is a positive definite kernel, $\mathcal H_k$ its RKHS, and $\phi(x)=k(x,\cdot)$ the canonical feature map. In this construction, distributions become points in Hilbert space, expectations of RKHS functions become inner products, and geometric operations on $\mathcal H_k$ induce nonparametric procedures for testing, regression, numerical integration, probabilistic inference, and control (Muandet et al., 2016).

1. Formal definition and RKHS representation

The basic KME construction extends the standard kernel feature map from observations to probability measures. If $k:\mathcal X\times\mathcal X\to\mathbb R$ is positive definite and $\mathcal H_k$ is the associated RKHS, then

$$\mu_P=\int_{\mathcal X} k(x,\cdot)\,dP(x)=\mathbb E_{X\sim P}[\phi(X)].$$

A sufficient existence condition is

$\mathbb E_{X\sim P}\big[\sqrt{k(X,X)}\big]<\infty,$

and a common stronger condition is bounded kernel diagonal, $$\mu_P := \int_{\mathcal X} k(\cdot,x)\,dP(x)=\mathbb E_{X\sim P}[\phi(X)],$$0. The reproducing property yields

$$\mu_P := \int_{\mathcal X} k(\cdot,x)\,dP(x)=\mathbb E_{X\sim P}[\phi(X)],$$1

so the embedding is an expectation-preserving representation of the measure (Muandet et al., 2016).

A useful equivalent notation treats the embedding as a function,

$$\mu_P := \int_{\mathcal X} k(\cdot,x)\,dP(x)=\mathbb E_{X\sim P}[\phi(X)],$$2

with $$\mu_P := \int_{\mathcal X} k(\cdot,x)\,dP(x)=\mathbb E_{X\sim P}[\phi(X)],$$3. This function-form and its integrated scalar counterpart both recur in Bayesian quadrature, worst-case error analysis, and MMD computations (Briol et al., 26 Apr 2025).

Several classical kernels recover familiar moment objects. For the linear kernel, $$\mu_P := \int_{\mathcal X} k(\cdot,x)\,dP(x)=\mathbb E_{X\sim P}[\phi(X)],$$4. For polynomial kernels, $$\mu_P := \int_{\mathcal X} k(\cdot,x)\,dP(x)=\mathbb E_{X\sim P}[\phi(X)],$$5 contains moments up to the kernel degree. For translation-invariant kernels $$\mu_P := \int_{\mathcal X} k(\cdot,x)\,dP(x)=\mathbb E_{X\sim P}[\phi(X)],$$6, the embedding is a filtered characteristic function via Bochner’s theorem. In particular, a Dirac measure satisfies $$\mu_P := \int_{\mathcal X} k(\cdot,x)\,dP(x)=\mathbb E_{X\sim P}[\phi(X)],$$7, so the standard pointwise feature map is the special case of KME applied to a point mass (Muandet et al., 2016).

2. Characteristicness, MMD, and conditional embeddings

The central identifiability notion is characteristicness: $$\mu_P := \int_{\mathcal X} k(\cdot,x)\,dP(x)=\mathbb E_{X\sim P}[\phi(X)],$$8 is characteristic if

$$\mu_P := \int_{\mathcal X} k(\cdot,x)\,dP(x)=\mathbb E_{X\sim P}[\phi(X)],$$9

This makes KME a faithful representation of distributions. Universal kernels on compact domains are characteristic, and for translation-invariant kernels on $k$0, characteristicness is tied to the support of the Fourier transform or spectral measure covering all of $k$1. The survey explicitly lists Gaussian, Laplacian, Matérn, rational quadratic, certain spline kernels, and various kernels on groups and semigroups as characteristic examples (Muandet et al., 2016).

The induced discrepancy is the maximum mean discrepancy,

$k$2

where

$k$3

Equivalently, MMD is the integral probability metric over the RKHS unit ball. For characteristic kernels, $k$4 if and only if $k$5, which is why KME underlies kernel two-sample testing, goodness-of-fit testing, and minimum-distance estimation (Muandet et al., 2016, Briol et al., 26 Apr 2025).

KME also extends to dependence measures. The Hilbert-Schmidt Independence Criterion (HSIC) is the squared Hilbert-Schmidt norm of a cross-covariance operator and vanishes under independence when the product kernel is characteristic. In this sense, the KME framework provides both a geometry for marginal distributions and an operator-theoretic language for dependence (Muandet et al., 2016).

A major generalization is the conditional mean embedding (CME), which represents $k$6 by an operator-valued map: $k$7 Its regularized empirical form is

$k$8

This enables kernel sum, product, and Bayes’ rules, but it is also an inverse problem because $k$9 is ill-posed in infinite dimensions; accordingly, CME estimation is substantially harder than marginal KME estimation and has slower convergence rates (Muandet et al., 2016).

3. Estimation theory, minimax rates, and Bayesian regularization

Given i.i.d. samples $\mathcal H_k$0, the empirical embedding is

$\mathcal H_k$1

It is unbiased, consistent, and attains the familiar $\mathcal H_k$2 RKHS error rate under bounded-kernel conditions (Muandet et al., 2016).

That rate is not merely a property of the empirical estimator; it is minimax-optimal. For continuous translation-invariant kernels on $\mathcal H_k$3, the $\mathcal H_k$4 rate is minimax in both RKHS norm and $\mathcal H_k$5-norm over the class of discrete measures and the class of measures with infinitely differentiable densities. A common misconception is that smoother kernels or smoother densities should improve the estimation exponent; the minimax analysis shows that, for KME estimation itself, the exponent remains $\mathcal H_k$6 (Tolstikhin et al., 2016).

Finite-sample behavior can nevertheless improve in favorable instances. A variance-aware analysis introduces the intrinsic RKHS variance

$\mathcal H_k$7

which can be much smaller than the crude uniform bound based on $\mathcal H_k$8. The resulting Bernstein-type bounds adapt to low-variance settings, and the paper gives an unbiased empirical estimator $\mathcal H_k$9 together with empirical-Bernstein confidence radii. For translation-invariant kernels, this yields a fully data-driven confidence bound that is never much worse than the classical distribution-agnostic bound but can be substantially tighter when $\phi(x)=k(x,\cdot)$0 (Wolfer et al., 2022).

Another line of regularization is Bayesian. The Bayesian Kernel Embedding model places a GP prior directly on the unknown mean embedding: $\phi(x)=k(x,\cdot)$1 This construction is chosen so that sample paths lie in $\phi(x)=k(x,\cdot)$2 almost surely. Combined with a conjugate Gaussian likelihood for the empirical embedding evaluations, it yields a closed-form posterior over $\phi(x)=k(x,\cdot)$3. The posterior mean is closely related to shrinkage estimators for KME, while the marginal pseudolikelihood provides a closed-form objective for kernel hyperparameter learning in unsupervised settings such as MMD and HSIC (Flaxman et al., 2016).

4. Closed forms, sparse approximations, and scalable computation

Many kernel methods require not only $\phi(x)=k(x,\cdot)$4 but explicit formulas for $\phi(x)=k(x,\cdot)$5, $\phi(x)=k(x,\cdot)$6, and $\phi(x)=k(x,\cdot)$7. This is particularly important in Bayesian quadrature, where

$\phi(x)=k(x,\cdot)$8

and in RKHS quadrature, where the worst-case error depends explicitly on $\phi(x)=k(x,\cdot)$9 and $\mathcal H_k$0. Closed forms eliminate nested numerical integration and can improve both numerical stability and statistical performance (Briol et al., 26 Apr 2025).

A recent dictionary of closed-form KMEs collates formulas for Gaussian, Matérn, Wendland, fractional Brownian motion, power-series, spherical stationary, periodic Sobolev, and Stein kernels. A central example is the Gaussian kernel with Gaussian measure: $\mathcal H_k$1 and

$\mathcal H_k$2

The same reference emphasizes practical closure rules: products of kernels with product measures, sums with mixtures, change of variables via pushforwards, matrix-valued lifts from scalar embeddings, and Stein-kernel constructions with $\mathcal H_k$3 (Briol et al., 26 Apr 2025).

When closed forms are unavailable, scalable approximations become central. One approach is sparse approximation of the empirical kernel mean: $\mathcal H_k$4 For radial kernels, support selection is reduced to the $\mathcal H_k$5-center problem, giving a linear-time construction of a sparse kernel mean together with an automatic sparsity-selection scheme. This preserves the representer form of the empirical KME while reducing downstream costs (Cortés et al., 2015).

A second approach is Nyström compression. Given landmarks $\mathcal H_k$6 sampled from the data, the approximation is

$\mathcal H_k$7

High-probability bounds show that, under effective-dimension conditions, one can retain the standard $\mathcal H_k$8 statistical rate with $\mathcal H_k$9; in particular, polynomial or logarithmic effective-dimension growth permits subsample sizes on the order of $k:\mathcal X\times\mathcal X\to\mathbb R$0 up to logarithmic factors (Chatalic et al., 2022).

5. Generalizations to stochastic processes, operator-valued measures, and stochastic kernels

Standard KME treats a stochastic process as a path-valued random variable, but this ignores filtration. Higher-order KMEs address this by conditioning on the filtration and thereby capturing information flow through time. This yields higher-order MMDs, empirical estimators with consistency guarantees, a filtration-sensitive kernel two-sample test, universal kernels on stochastic processes, and applications to calibration, optimal stopping, and causal discovery from multidimensional trajectories (Salvi et al., 2021).

KME has also been generalized from RKHSs to reproducing kernel Hilbert $k:\mathcal X\times\mathcal X\to\mathbb R$1-modules (RKHMs). In that setting, the embedding becomes

$k:\mathcal X\times\mathcal X\to\mathbb R$2

where $k:\mathcal X\times\mathcal X\to\mathbb R$3 is an $k:\mathcal X\times\mathcal X\to\mathbb R$4-valued finite regular Borel measure and $k:\mathcal X\times\mathcal X\to\mathbb R$5 is a $k:\mathcal X\times\mathcal X\to\mathbb R$6-algebra or von Neumann algebra. The reproducing identity is

$k:\mathcal X\times\mathcal X\to\mathbb R$7

and for matrix-valued $k:\mathcal X\times\mathcal X\to\mathbb R$8 the theory recovers injectivity results for transition-invariant and radial kernels together with an exact equivalence between injectivity and universality in the finite-dimensional algebra case (Hashimoto et al., 2021).

A closely related noncommutative extension embeds von Neumann-algebra-valued measures into RKHMs and defines an $k:\mathcal X\times\mathcal X\to\mathbb R$9-valued analogue of MMD,

$\mathcal H_k$0

with applications to matrix-valued cross-covariance measures and positive operator-valued measures in quantum mechanics. In the finite-dimensional matrix case, the paper proves the RKHM analogue of the classical injectivity–universality equivalence and shows that the framework can preserve pairwise interaction structure that scalar RKHS embeddings collapse into a single number (Hashimoto et al., 2020).

A different extension treats an entire stochastic kernel $\mathcal H_k$1 as a Bochner-measurable $\mathcal H_k$2-valued function. This yields weak and strong KME topologies on spaces of stochastic kernels. The weak form is equivalent, on stochastic kernels, to the Borkar $\mathcal H_k$3-topology and the Young narrow topology when the limit remains stochastic, whereas the strong form is the $\mathcal H_k$4-norm topology of pointwise MMD discrepancies and is intended for robustness and learning-theoretic analysis of models. The paper is explicit that weak KME and $\mathcal H_k$5 are relatively compact but not closed, while Young narrow is closed but lacks relative compactness (Saldi et al., 19 Feb 2025).

6. Applications, scope, and current issues

Because KME turns distributions into geometric objects, it supports learning directly on distributional inputs. The survey lists multiple-instance learning, group anomaly detection, distribution regression, support measure machines, probabilistic modeling, causal discovery, reinforcement learning, and deep generative modeling as representative application areas (Muandet et al., 2016). A concrete example is multiple-instance regression, where KME is applied not to the distribution of original instances but to the empirical distribution of instance-level predicted labels within each bag: $\mathcal H_k$6 This target-wise embedding yields consistent gains over mean or median aggregation while preserving within-bag predictive heterogeneity (Uriot, 2019).

KME also appears in domain-specific statistical procedures. In functional data analysis on infinite-dimensional separable Hilbert spaces, KMEs of Gaussian laws admit explicit formulas and support tests for function-on-scalar regression, functional one-way ANOVA, and equality of covariance operators (Hayati et al., 2020). For spatial point patterns, a finite-dimensional approximate KME tailored to $\mathcal H_k$7 converts first-order pattern comparison into coordinate-wise Euclidean mean testing, avoiding bootstrap or permutation calibration for the per-coordinate tests (Rustamov et al., 2019).

In numerical integration and optimization, KME is foundational for Bayesian quadrature, kernel quadrature, and semi-analytic MMD computations (Briol et al., 26 Apr 2025). It has also been used as a distribution-preserving compression mechanism in stochastic programming and control: the empirical scenario distribution is approximated by a sparse RKHS expansion, zero-weight scenarios are removed, and the optimization is re-solved on the reduced scenario set to decrease conservativeness (Zhu et al., 2020).

More recent work pushes KME into dynamical inference and control. KME-dynamics replaces low-order moment matching by matching probability measures through their RKHS embeddings along a tempered Bayesian path, yielding interacting particle systems that include the Kalman–Bucy filter as the quadratic-kernel special case (Wang et al., 2024). In nonlinear stochastic optimal control, tensorized KMEs are used to identify Markov transition operators of controlled diffusions from data and then inserted into a convex Hamilton–Jacobi–Bellman recursion to produce optimal feedback laws without explicit model identification (Bevanda et al., 2024).

Several limitations recur across this literature. Characteristicness is essential if identifiability is required; otherwise KME may identify only an equivalence class of distributions (Muandet et al., 2016). Kernel and bandwidth choice remain delicate, and this difficulty motivates Bayesian hyperparameter learning, variance-aware confidence bounds, and tractable closed-form dictionaries (Flaxman et al., 2016, Wolfer et al., 2022, Briol et al., 26 Apr 2025). Conditional embeddings remain inverse problems, and noncommutative or infinite-dimensional generalizations still have incomplete injectivity theory beyond the finite-dimensional matrix-valued case (Muandet et al., 2016, Hashimoto et al., 2021, Hashimoto et al., 2020). Taken together, these results suggest that KME is best understood not as a single estimator, but as a unifying representation layer linking distributional geometry, kernelized operators, and data-driven inference across a wide range of stochastic models.