Maximum Mean Discrepancy (MMD)

• Maximum Mean Discrepancy is a kernel-based metric that quantifies the distance between distributions by measuring the RKHS norm of their mean embeddings.
• It underpins nonparametric two-sample testing by allowing distribution comparisons from samples without explicit density estimation.
• MMD serves as a likelihood-free learning objective in generative modeling and domain adaptation, offering efficient and scalable computational frameworks.

Maximum Mean Discrepancy (MMD) is a kernel-based way to compare two probability distributions using samples rather than explicit likelihoods. In the reproducing kernel Hilbert space (RKHS) formulation, it is the distance between the kernel mean embeddings of the two distributions, and for characteristic kernels it is zero if and only if the distributions are equal. Because it can be estimated from samples alone, without explicit density estimation and without a min-max discriminator game, MMD has become a standard nonparametric two-sample statistic and a general distribution-matching objective in implicit generative modeling, likelihood-free inference, domain adaptation, online change detection, and related optimization problems (Alon et al., 2021).

1. Formal definition and RKHS interpretation

At the most general level, MMD is an integral probability metric (IPM). For distributions $P$ and $Q$ and a function class $F$,

$\MMD(P,Q;F):=\sup_{f\in F}\left|\mathbb{E}[f(X)]-\mathbb{E}[f(Y)]\right|,$

where $X\sim P$ and $Y\sim Q$. In the RKHS setting, $F$ is taken to be the unit ball of a reproducing kernel Hilbert space $\mathcal H_k$, giving

$\MMD(P,Q;\mathcal H_k) :=\sup_{f\in \mathcal H_k,\ \|f\|_{\mathcal H_k}\le 1} \left|\mathbb{E}[f(X)]-\mathbb{E}[f(Y)]\right| = \|\mu_P-\mu_Q\|_{\mathcal H_k},$

with kernel mean embeddings $\mu_P:=\mathbb{E}[k(\cdot,X)]$ and $Q$0 (Gao et al., 2020).

The squared MMD admits the standard kernel expansion

$Q$1

In generative-model training, the optimization is often performed on this squared form; one source explicitly notes that, more precisely, MMD is the square root of this quantity, but the optimization is done on the squared form (Alon et al., 2021). A common choice is the Gaussian RBF kernel

$Q$2

with bandwidth $Q$3 (Alon et al., 2021).

A fundamental property is characteristicness. For characteristic kernels, $Q$4 if and only if $Q$5, so minimizing MMD means matching the full distributions, not just finitely many moments (Alon et al., 2021). In the language of discrepancy theory, MMD is the norm of the difference of two measures when embedded into an RKHS. On a compact Polish space $Q$6 with continuous symmetric positive definite kernel $Q$7, the discrepancy

$Q$8

has the closed form

$Q$9

which is precisely the RKHS embedding viewpoint (2002.01189).

The spectral interpretation is equally important. If

$F$0

then

$F$1

so the discrepancy is the weighted squared distance between Mercer coefficients of the two measures (2002.01189). This formulation makes explicit that kernel choice determines which distributional features are emphasized.

2. Estimation and two-sample testing

The canonical statistical use of MMD is the two-sample problem

$F$2

Given IID samples

$F$3

a standard unbiased estimator is the $F$4-statistic

$F$5

with

$F$6

and a standard biased estimator is

$F$7

(Gao et al., 2020, Kalinke et al., 2022).

A central difficulty in classical MMD testing is null calibration. Under $F$8, the standard quadratic-time MMD statistic has a degenerate asymptotic behavior, converging to an infinite weighted sum of centered $F$9-type random variables, which makes direct thresholding difficult and often necessitates bootstrap or permutation calibration (Danafar et al., 2013). This calibration issue has driven a substantial literature on studentization, regularization, and alternative asymptotics.

One response is Regularized Maximum Mean Discrepancy (RMMD), defined by

$\MMD(P,Q;F):=\sup_{f\in F}\left|\mathbb{E}[f(X)]-\mathbb{E}[f(Y)]\right|,$0

RMMD reduces to squared MMD when $\MMD(P,Q;F):=\sup_{f\in F}\left|\mathbb{E}[f(X)]-\mathbb{E}[f(Y)]\right|,$1, is asymptotically normal under both null and alternative, and is claimed to be non-degenerate under $\MMD(P,Q;F):=\sup_{f\in F}\left|\mathbb{E}[f(X)]-\mathbb{E}[f(Y)]\right|,$2; the cited analysis identifies $\MMD(P,Q;F):=\sup_{f\in F}\left|\mathbb{E}[f(X)]-\mathbb{E}[f(Y)]\right|,$3 as the choice yielding the highest power (Danafar et al., 2013).

A second response is direct studentization in moderate and high dimension. A recent high-dimensional analysis develops a studentized MMD statistic $\MMD(P,Q;F):=\sup_{f\in F}\left|\mathbb{E}[f(X)]-\mathbb{E}[f(Y)]\right|,$4 whose null limit is standard normal as both the ambient dimension $\MMD(P,Q;F):=\sup_{f\in F}\left|\mathbb{E}[f(X)]-\mathbb{E}[f(Y)]\right|,$5 and sample sizes $\MMD(P,Q;F):=\sup_{f\in F}\left|\mathbb{E}[f(X)]-\mathbb{E}[f(Y)]\right|,$6 diverge, for a wide range of kernels including Gaussian and Laplacian kernels and also energy distance as a special case (Gao et al., 2021). That work also provides Berry–Esseen-type error bounds and a power theory showing that lower-order discrepancies are easier to detect than purely higher-moment differences (Gao et al., 2021).

A third response is martingale reformulation. The martingale MMD (mMMD) statistic sequentializes the witness estimation,

$\MMD(P,Q;F):=\sup_{f\in F}\left|\mathbb{E}[f(X)]-\mathbb{E}[f(Y)]\right|,$7

with self-normalization

$\MMD(P,Q;F):=\sup_{f\in F}\left|\mathbb{E}[f(X)]-\mathbb{E}[f(Y)]\right|,$8

and obtains a limiting standard Gaussian distribution under $\MMD(P,Q;F):=\sup_{f\in F}\left|\mathbb{E}[f(X)]-\mathbb{E}[f(Y)]\right|,$9, consistency against any fixed alternative, and resampling-free Gaussian thresholding, while retaining quadratic-time computation (Chatterjee et al., 13 Oct 2025).

The testing literature also highlights that apparent failures of MMD can be failures of kernel choice, power optimization, or bootstrap validity rather than failures of the metric itself. In adversarial-example detection, one paper argues that previous negative results arose because the Gaussian kernel has limited representation power for complex high-dimensional data, test power optimization was neglected, and adversarial data may be non-IID. Its Semantic-aware MMD (SAMMD) replaces the Gaussian kernel by a deep kernel based on semantic features, maximizes an asymptotic test-power criterion, and uses the wild bootstrap rather than permutation bootstrap when dependence is present; on this basis, it reports that MMD is aware of adversarial attacks (Gao et al., 2020).

3. Optimization in generative modeling and likelihood-free inference

MMD has a distinct role outside testing: it is a likelihood-free learning objective. In implicit generative models, likelihood $X\sim P$0 is often unavailable or intractable to compute, so maximum likelihood estimation is not practical. MMD instead compares a generated distribution $X\sim P$1 and a target distribution $X\sim P$2 through expectations of kernel test functions and can be estimated from samples alone (Alon et al., 2021).

A central theoretical question is whether MMD-based learning can globally optimize its non-convex objective. For three representative settings—Gaussian mean estimation, low-rank Gaussian covariance estimation, and symmetric Gaussian-mixture mean estimation—the optimization landscape has been shown to be benign: there are no bad local minima, and all other critical points are strict saddles or maxima (Alon et al., 2021). For $X\sim P$3 with known covariance and unknown mean, the objective is quasi-convex in $X\sim P$4 and has a single stationary point at $X\sim P$5, which is the global minimizer. For the low-rank covariance model

$X\sim P$6

the global minima are at $X\sim P$7, the origin is a global maximum, and all other stationary points are strict saddles. For the symmetric two-Gaussian mixture

$X\sim P$8

the only global minima are at $X\sim P$9, the global maximum is at $Y\sim Q$0, and all other stationary points are strict saddles (Alon et al., 2021). Combined with standard strict-saddle convergence results, these landscape statements yield a global optimization guarantee for MMD-based learning in those models (Alon et al., 2021).

Likelihood-free inference motivates a different line of work: improving the sample complexity of MMD estimation itself. Standard U- and V-statistic MMD estimators converge at the root-$Y\sim Q$1 Monte Carlo rate on the simulator side. An optimally weighted estimator instead uses weights $Y\sim Q$2 on simulated points,

$Y\sim Q$3

chosen by a kernel quadrature system on the latent base space $Y\sim Q$4. Under smoothness assumptions on the generator $Y\sim Q$5 and Matérn-kernel regularity,

$Y\sim Q$6

which improves over the usual $Y\sim Q$7 rate (Bharti et al., 2023). The method is especially advantageous for smooth simulators with low-to-moderate latent dimension $Y\sim Q$8, and it has been used in minimum distance estimation, approximate Bayesian computation, generalized Bayesian inference, and composite goodness-of-fit testing (Bharti et al., 2023).

MMD also underlies measure quantisation. In that setting, the target measure $Y\sim Q$9 is approximated by a discrete empirical measure supported on a selected point set, with the goal of minimizing

$F$0

Sequential selection schemes include myopic greedy minimization, a non-myopic algorithm that selects $F$1 points jointly by solving an integer quadratic program, and mini-batch variants that reduce computational cost while preserving consistency under IID or Markov-chain candidate sampling (Teymur et al., 2020). Applications include optimization of nodes in Bayesian cubature and thinning of Markov chain output (Teymur et al., 2020).

4. Computational acceleration and scalable computation

The quadratic cost of naive MMD computation has motivated several acceleration schemes. For exact pairwise evaluation with $F$2 samples in $F$3 dimensions, classical MMD requires $F$4 time (Zhao et al., 2014). FastMMD exploits shift-invariant kernels and Bochner’s theorem to rewrite the MMD quadratic form as an expectation over sinusoidal discrepancies. With $F$5 random Fourier frequencies, the complexity becomes

$F$6

and for spherically invariant kernels such as the Gaussian, Fastfood reduces this further to

$F$7

The same work gives uniform convergence bounds for both biased and unbiased estimators and interprets MMD geometrically as an ensemble of circular discrepancy (Zhao et al., 2014).

An alternative route is to represent the kernel witness implicitly by a neural network. Neural Tangent Kernel MMD (NTK-MMD) trains a network briefly on the linear objective

$F$8

so that, in the short-time or lazy-training regime, the resulting statistic is the classical squared MMD with kernel equal to the neural tangent kernel at initialization (Cheng et al., 2021). The network-computed statistic approximates exact NTK-MMD with linear-in-time error, admits online or one-pass SGD implementations, and avoids explicit storage of a large Gram matrix (Cheng et al., 2021).

Streaming settings require a different kind of scalability. Maximum Mean Discrepancy on Exponential Windows (MMDEW) compares recent observations to earlier observations in a data stream using exponentially sized windows and stored kernel summaries. A key lemma shows that the MMD between neighboring windows can be computed in $F$9 time from stored within-window and cross-window kernel sums, and the full online algorithm has runtime $\mathcal H_k$0 per insert and memory $\mathcal H_k$1 (Kalinke et al., 2022). This makes MMD practical for online change detection with formal level-$\mathcal H_k$2 thresholds under bounded kernels (Kalinke et al., 2022).

Uniform generalization guarantees for MMD objectives have also been obtained. A recent concentration theory treats MMD, energy distance, distance covariance, and HSIC under a unified kernel-based framework and proves finite-sample uniform concentration inequalities for empirical MMD objectives over generator classes with finite covering number (Ni et al., 2024). For MMD-based minimum-distance estimation and MMD-GAN, these results yield explicit excess-risk bounds in terms of Gaussian complexity, kernel boundedness and Lipschitz constants, and the confidence parameter $\mathcal H_k$3 (Ni et al., 2024).

5. Distribution shift, cross-domain learning, and applied variants

In predictive modeling under covariate shift, MMD is used directly as a regularizer between the labeled training set $\mathcal H_k$4 and the unlabeled test set $\mathcal H_k$5. Three formulations have been proposed: MMD Representation, which aligns training and test embeddings through

$\mathcal H_k$6

MMD Mask, which applies MMD to the joint space of feature values and missingness indicators; and MMD Hybrid, which alternates between input-space masking alignment and embedding-space alignment (Ouyang et al., 2021). These methods are designed to address distribution shift, missingness shift, or both, and empirical results indicate that representation matching helps most for ordinary covariate shift, masking helps most for missingness shift, and the hybrid formulation is best when both are present (Ouyang et al., 2021).

In domain adaptation, MMD is often used to align source and target distributions, but its interaction with class geometry is nontrivial. One analysis shows that minimizing the class-wise MMD equals maximizing the source-target class-wise variance terms and minimizing the source-target class-wise within-class scatters with implicit class-dependent weights

$\mathcal H_k$7

which can degrade feature discriminability (Wang et al., 2020). On that basis, discriminative MMD variants introduce explicit trade-off parameters on intra-class compactness and inter-class separation (Wang et al., 2020). A related development, Discriminative Joint Probability MMD (DJP-MMD), replaces the usual weighted sum of marginal and conditional discrepancies by a direct joint-probability discrepancy based on $\mathcal H_k$8, together with an explicit discriminability term $\mathcal H_k$9 that pushes apart different classes across domains (Zhang et al., 2019).

Security and heterogeneous-space extensions further broaden the scope of MMD. Secure Maximum Mean Discrepancy (SMMD) embeds MMD-based transfer learning in an additively homomorphic encryption pipeline so that source and target parties can align hidden representations without revealing raw data, hidden features, or pairwise kernel values (Zhang et al., 2020). Generalized MMD (GMMD) extends MMD to probability measures on different spaces by introducing two cycle-consistent maps $\MMD(P,Q;\mathcal H_k) :=\sup_{f\in \mathcal H_k,\ \|f\|_{\mathcal H_k}\le 1} \left|\mathbb{E}[f(X)]-\mathbb{E}[f(Y)]\right| = \|\mu_P-\mu_Q\|_{\mathcal H_k},$0 and $\MMD(P,Q;\mathcal H_k) :=\sup_{f\in \mathcal H_k,\ \|f\|_{\mathcal H_k}\le 1} \left|\mathbb{E}[f(X)]-\mathbb{E}[f(Y)]\right| = \|\mu_P-\mu_Q\|_{\mathcal H_k},$1 and penalizing both geometric distortion and MMD between pushforwards,

$\MMD(P,Q;\mathcal H_k) :=\sup_{f\in \mathcal H_k,\ \|f\|_{\mathcal H_k}\le 1} \left|\mathbb{E}[f(X)]-\mathbb{E}[f(Y)]\right| = \|\mu_P-\mu_Q\|_{\mathcal H_k},$2

(Zhang et al., 2021). This construction connects MMD-style marginal matching with cycle-consistent cross-space correspondence learning (Zhang et al., 2021).

MMD has also been used to reconstruct the analysis step of particle filtering. In the ensemble transport filter, a transport map $\MMD(P,Q;\mathcal H_k) :=\sup_{f\in \mathcal H_k,\ \|f\|_{\mathcal H_k}\le 1} \left|\mathbb{E}[f(X)]-\mathbb{E}[f(Y)]\right| = \|\mu_P-\mu_Q\|_{\mathcal H_k},$3 is learned so that the pushed-forward forecast ensemble matches a particle-filter posterior in MMD, and a variance penalty term is added to prioritize minimizing the discrepancy between the expectations of highly informative statistics for the approximated and reference posteriors (Zeng et al., 2024). The reported effect is improved robustness, spread, and coverage relative to pure MMD optimization in nonlinear and higher-dimensional filtering regimes (Zeng et al., 2024).

6. Relation to optimal transport, Wasserstein dynamics, and broader IPM theory

MMD occupies a specific position within the broader family of discrepancy measures. One rigorous link is to entropically regularized optimal transport. For a symmetric positive definite kernel $\MMD(P,Q;\mathcal H_k) :=\sup_{f\in \mathcal H_k,\ \|f\|_{\mathcal H_k}\le 1} \left|\mathbb{E}[f(X)]-\mathbb{E}[f(Y)]\right| = \|\mu_P-\mu_Q\|_{\mathcal H_k},$4 and the cost correspondence $\MMD(P,Q;\mathcal H_k) :=\sup_{f\in \mathcal H_k,\ \|f\|_{\mathcal H_k}\le 1} \left|\mathbb{E}[f(X)]-\mathbb{E}[f(Y)]\right| = \|\mu_P-\mu_Q\|_{\mathcal H_k},$5, the debiased Sinkhorn divergence satisfies

$\MMD(P,Q;\mathcal H_k) :=\sup_{f\in \mathcal H_k,\ \|f\|_{\mathcal H_k}\le 1} \left|\mathbb{E}[f(X)]-\mathbb{E}[f(Y)]\right| = \|\mu_P-\mu_Q\|_{\mathcal H_k},$6

so the large-$\MMD(P,Q;\mathcal H_k) :=\sup_{f\in \mathcal H_k,\ \|f\|_{\mathcal H_k}\le 1} \left|\mathbb{E}[f(X)]-\mathbb{E}[f(Y)]\right| = \|\mu_P-\mu_Q\|_{\mathcal H_k},$7 limit of Sinkhorn divergence is exactly half the squared MMD or discrepancy (2002.01189). The same analysis proves

$\MMD(P,Q;\mathcal H_k) :=\sup_{f\in \mathcal H_k,\ \|f\|_{\mathcal H_k}\le 1} \left|\mathbb{E}[f(X)]-\mathbb{E}[f(Y)]\right| = \|\mu_P-\mu_Q\|_{\mathcal H_k},$8

and

$\MMD(P,Q;\mathcal H_k) :=\sup_{f\in \mathcal H_k,\ \|f\|_{\mathcal H_k}\le 1} \left|\mathbb{E}[f(X)]-\mathbb{E}[f(Y)]\right| = \|\mu_P-\mu_Q\|_{\mathcal H_k},$9

showing that Sinkhorn regularization interpolates between optimal transport geometry and kernel mean embedding geometry (2002.01189).

A second geometric development treats MMD itself as an energy on Wasserstein space. For fixed target $\mu_P:=\mathbb{E}[k(\cdot,X)]$0, define

$\mu_P:=\mathbb{E}[k(\cdot,X)]$1

Its first variation is the witness function $\mu_P:=\mathbb{E}[k(\cdot,X)]$2, and the Wasserstein gradient flow is

$\mu_P:=\mathbb{E}[k(\cdot,X)]$3

The associated particle dynamics are

$\mu_P:=\mathbb{E}[k(\cdot,X)]$4

and the MMD energy decreases monotonically according to

$\mu_P:=\mathbb{E}[k(\cdot,X)]$5

(Arbel et al., 2019). Because the objective is not displacement convex in the full Wasserstein sense, global convergence requires additional conditions; the same work proposes gradient-noise injection as a regularization mechanism that preserves the original MMD optimum while improving empirical and theoretical convergence behavior (Arbel et al., 2019).

Within IPM theory, MMD is a particular choice of function class rather than the only one. The Radon-Kolmogorov-Smirnov (RKS) test is an IPM over a Radon bounded variation function class, not an RKHS, and is therefore described as a non-kernel MMD or IPM rather than a kernel MMD (Paik et al., 2023). Its witness is always a ridge spline of degree $\mu_P:=\mathbb{E}[k(\cdot,X)]$6, i.e., a single neuron in a neural network, and its empirical strengths differ from Gaussian-kernel MMD: it is especially suitable when $\mu_P:=\mathbb{E}[k(\cdot,X)]$7 and $\mu_P:=\mathbb{E}[k(\cdot,X)]$8 differ in only a few directions, whereas Gaussian kernel MMD and energy distance perform better for diffuse global differences such as variance inflation in all directions (Paik et al., 2023). This comparison clarifies that MMD is best understood as one prominent member of a larger IPM family, distinguished by RKHS geometry, kernel mean embeddings, and the characteristic-kernel identifiability property.