---
title: Maximum Mean Discrepancy Split Criterion
url: https://www.emergentmind.com/topics/maximum-mean-discrepancy-mmd-split-criterion
type: topic
---

# Maximum Mean Discrepancy Split Criterion

The Maximum Mean Discrepancy (MMD) split criterion is a data-partitioning principle designed to maximize distributional separation between subsets, primarily motivated by domain shift and unsupervised/data-driven clustering challenges. The MMD statistic serves as a nonparametric measure of divergence between probability distributions and underpins principled split criteria for tasks including out-of-distribution model selection and unsupervised functional data clustering. Recent major developments formalize both split-maximization and recursive clustering approaches based on the maximization of MMD and its weighted variants, delivering theoretically-grounded algorithms with empirical and oracle guarantees [2405.19461][2507.11158].

## 1. Definition of Maximum Mean Discrepancy and its Empirical Estimation

Let $P$ and $Q$ be probability distributions over a space $\mathcal{X}$ and let $\kappa:\mathcal{X}\times\mathcal{X}\to\mathbb{R}$ denote a positive-definite kernel with feature map $\phi:\mathcal{X}\to\mathcal{H}$ into an RKHS $\mathcal{H}$. The squared Maximum Mean Discrepancy is defined as
$$
\mathrm{MMD}^2(P,Q)=\|\mu_P-\mu_Q\|^2_{\mathcal H}=\Big\|\mathbb{E}_{x\sim P}[\phi(x)] - \mathbb{E}_{y\sim Q}[\phi(y)]\Big\|^2_{\mathcal H}.
$$
For i.i.d. samples $\{x_i\}_{i=1}^n$ from $P$ and $\{y_j\}_{j=1}^m$ from $Q$, the unbiased empirical estimator is
$$
\mathrm{MMD}^2(P,Q)=\frac{1}{n^2}\sum_{i,i'}\kappa(x_i,x_{i'})+\frac{1}{m^2}\sum_{j,j'}\kappa(y_j,y_{j'})-\frac{2}{nm}\sum_{i,j}\kappa(x_i,y_j).
$$
[2405.19461][2507.11158]

For data splitting, a weighted-MMD form is introduced:
$$
d_w(A,B)=\frac{n_1 n_2}{n_1+n_2}\mathrm{MMD}^2(\widehat{P}_A,\widehat{P}_B),
$$
where $A$ and $B$ are discrete data subsets of size $n_1$ and $n_2$. The prefactor penalizes severely imbalanced splits [2507.11158].

## 2. MMD as a Data Splitting and Clustering Objective

### Split Maximization under Domain Shift

For model selection robust to domain shift, the optimal train/validation partition maximizes the MMD between the two sets in feature space. Let $S$ index the dataset, $V\subset S$ designate the (fractional) validation subset, and $T=S\setminus V$ the training subset. With constraints ensuring class/domain representativity ($|V_{(y,d)=g}|=h|S_{(y,d)=g}|$ for all $g$), the split maximizes
$$
\max_{T,V}\;\|\mu_{\mathbb P_T}-\mu_{\mathbb P_V}\|_{\mathcal H},
$$
subject to cardinality constraints [2405.19461].

### Binary Splitting for Maximum Cluster Separation

For unlabelled data, the task is to find a partition $(A,B)$ of a dataset $\mathcal{E}$ so that the weighted-MMD $d_w(A,B)$ is maximized. The "BS" procedure recursively seeks the point addition that maximizes $d_w$, continuing until overall maximal separation is found [2507.11158].

## 3. Algorithmic Realization: Kernel k-means and Recursive Binary Splitting

### Constrained Kernel k-means for MMD Splits

Maximizing the between-set MMD is equivalent (via the law of total variance in the RKHS) to minimizing the within-cluster variance for $k=2$ kernel k-means, under the same constraints. The optimal partition is found by:
- Alternating centroid updates in feature space and constrained assignments.
- Assignments are posed as linear programs: let $U\in\{0,1\}^{n\times 2}$ assign each point to one cluster, constrained to have fixed (class/domain) cluster sizes.
- By Hoffmann’s total-unimodularity result, LP relaxation is exact for this setup.
- For large $n$, the kernel matrix can be approximated efficiently (Nyström approximation) [2405.19461].

### Recursive Binary Splitting and Clustering

The BS algorithm for functional data is defined as follows:
1. Initialize one cluster as empty ($C_1^{(0)}=\emptyset$), the other as the full set ($C_2^{(0)}={\cal E}$).
2. Iteratively move a point maximizing the incremental $d_w$ until all $n-1$ possible splittings are examined.
3. Select the split (iteration) where weighted-MMD is maximized.
4. Efficient update formulas ensure $O(n^2)$ complexity for $n$ data points [2507.11158].

This procedure forms the backbone of both single-split and recursive (multiway) clusterings.

## 4. Extension to Unknown or Fixed Number of Clusters

With unknown $K$, an additional "single cluster check" (SCC) is implemented:
- Let $V$ be the ratio of maximum to minimum weighted-MMD observed over all possible splits.
- Null $H_0$ ("single cluster") is accepted if $|V-1| < |V-R/H|$ for ratios $R,\,H$ computed from extremal MMDs; otherwise, reject and split.
- The CURBS-I algorithm combines SCC and recursive BS, producing ($\widehat K$) clusters when further splits are unwarranted [2507.11158].

For fixed $K=J$ ("CURBS-II"), after recursive bisection, closest-pair merging via $d_w$ controls the final cluster count. This merging phase preserves the order-preserving property (POP): no cluster ever splits a true class.

## 5. Theoretical Guarantees and Oracle Analysis

In the oracle model, every observation’s true population label is known and used to define “representative” empirical measures per cluster:
$$
\widetilde P_S = \sum_{j=1}^K \left(\frac{|S\cap{\cal D}_j|}{|S|}\right)\widehat P_j.
$$
- Under mixture models, MMD contracts strictly unless clusters are pure.
- CURBS-I* (oracle variant) perfectly recovers true clusters for balanced sizes as $n\to\infty$.
- CURBS-II* guarantees Perfect-Order-Preserving (POP) clustering for all $J$: if the true number of clusters is $K$, setting $J<K$ merges whole classes, $J>K$ only splits classes [2507.11158].

## 6. Empirical Validation and Applications

Comparative studies across domain generalization (DG), unsupervised domain adaptation (UDA), and functional-data clustering demonstrate:
- Cluster-based splits (linear or RBF kernel) close approximately 50% of the gap (in normalized accuracy) between random split and oracle split, outperforming metadata-based or leave-one-domain-out strategies.
- Average test domain accuracy gains are +3 pp over random splitting [2405.19461].
- Direct analysis yields Spearman rank-correlation $\rho=0.63$ (p $\ll$ 10⁻³) and Pearson $r\approx0.61$ (p $\approx3.8\times10^{-4}$) between achieved MMD and held-out test accuracy, supporting the notion that maximizing MMD between train/validation benefits out-of-distribution model selection.
- In functional data clustering, the weighted-MMD/BS framework yields near-perfect Rand indices—empirically outperforming alternative methods, even where clusters differ by only subtle scale changes [2507.11158].

## 7. Practical Considerations and Implementational Choices

- Kernels: Gaussian, Laplacian, and linear choices, with practical selection by the median pairwise distance heuristic.
- Computational complexity: $O(n^2)$ pairwise kernel updates are needed, but BS incremental-update formulas and kernel approximations (e.g., Nyström) improve scalability.
- The algorithms require no access to explicit metadata beyond (class, domain) labels when available for splitting or representative-measure computation for oracle results, and can be deployed for both unlabelled and semi-supervised settings.
- The linear programming formulation allows enforcement or relaxation of class/domain balance constraints as required [2405.19461][2507.11158].

Source: https://www.emergentmind.com/topics/maximum-mean-discrepancy-mmd-split-criterion