Conditional Maximum Mean Discrepancy (CMMD)

• Conditional Maximum Mean Discrepancy (CMMD) is a kernel-based metric that quantifies differences between conditional distributions via RKHS embeddings and the Hilbert–Schmidt norm.
• It is applied non-parametrically in generative modeling, image classification, domain adaptation, and causal analysis by comparing conditional embedding operators.
• CMMD variants use operator-smoothing and regularization techniques to manage bias-variance trade-offs while ensuring identifiability under characteristic kernels.

Searching arXiv for recent and foundational papers on Conditional Maximum Mean Discrepancy and related kernel-embedding formulations. Conditional Maximum Mean Discrepancy (CMMD) is a kernel-based discrepancy for comparing conditional distributions through reproducing kernel Hilbert space (RKHS) embeddings. In its original formulation, CMMD is the Hilbert–Schmidt norm between two conditional embedding operators, so that two conditionals are identified by the vanishing of this operator difference under characteristic-kernel and regularity assumptions (Ren et al., 2016). Subsequent work has broadened the term to a family of conditional-distribution metrics indexed by an operator-smoothing level, with three named special cases—CMMD$_0$, CMMD$_1$, and CMMD$_2$—as well as closely related pointwise and averaged variants such as MCMD and AMCMD (Moskvichev et al., 4 May 2026, Goff et al., 23 Oct 2025, Broadbent et al., 14 Apr 2025). Across this literature, CMMD functions as a non-parametric tool for generative modeling, image classification, domain adaptation, distribution compression, model averaging, and causal transferability analysis.

1. Operator-theoretic foundation

Let $X \in \mathcal X$ and $Y \in \mathcal Y$ be random variables, with RKHSs $\mathcal F$ and $\mathcal G$ induced by kernels $k_x$ and $k_y$. The basic construction uses the cross-covariance and auto-covariance operators

$$C_{XX}=E[\phi_x(X)\otimes \phi_x(X)]-\mu_x\otimes \mu_x,\qquad C_{YX}=E[\phi_y(Y)\otimes \phi_x(X)]-\mu_y\otimes \mu_x,$$

and defines the conditional embedding operator by

$_1$0

This operator satisfies $_1$1 and $_1$2 (Ren et al., 2016).

For two conditional distributions $_1$3 and $_1$4 with embedding operators $_1$5 and $_1$6, the population CMMD is

$_1$7

If $_1$8 is characteristic and mild conditions hold, $_1$9 if and only if the conditional laws agree (Ren et al., 2016).

The empirical estimator is obtained from finite samples by replacing the population operators with regularized kernel estimators. Given samples

$_2$0

and Gram matrices $_2$1 together with cross-Gram terms, one defines

$_2$2

and the empirical CMMD expands into a trace expression involving only Gram matrices: $_2$3 with $_2$4 and $_2$5 (Ren et al., 2016).

A classification-specialized formulation embeds labels or predicted soft labels in an RKHS and computes the discrepancy between a “true” conditional distribution $_2$6 and a predicted conditional distribution $_2$7. In that setting, the same operator form is written as a CMMD loss between source and target conditional embeddings and is used directly as a training criterion (Ren et al., 2020).

2. Levels, variants, and related discrepancies

A later unifying treatment writes the difference of conditional mean operators as

$_2$8

and defines a level-$_2$9 CMMD by

$X \in \mathcal X$0

This places several previously separate constructions into one hierarchy (Moskvichev et al., 4 May 2026).

Variant	Definition	Emphasis
CMMD$X \in \mathcal X$1	$X \in \mathcal X$2	conditional mean operators
CMMD$X \in \mathcal X$3	$X \in \mathcal X$4	conditional mean embeddings
CMMD$X \in \mathcal X$5	$X \in \mathcal X$6	joint mean embeddings
MCMD	$X \in \mathcal X$7	fixed query point $X \in \mathcal X$8
AMCMD	$X \in \mathcal X$9	averaged CME discrepancy

The hierarchy obeys explicit inequalities: $Y \in \mathcal Y$0 If $Y \in \mathcal Y$1, then $Y \in \mathcal Y$2 (Moskvichev et al., 4 May 2026). The accompanying interpretation is that multiplication by $Y \in \mathcal Y$3 smooths differences in low-density directions of the covariate distribution.

The pointwise MCMD fixes a query $Y \in \mathcal Y$4 and compares two conditional distributions only at that covariate value: $Y \in \mathcal Y$5 Because the kernel on $Y \in \mathcal Y$6 is characteristic, MCMD is nonnegative, symmetric, satisfies the triangle inequality, and vanishes if and only if the two conditional laws at $Y \in \mathcal Y$7 coincide (Goff et al., 23 Oct 2025).

AMCMD introduces an additional weighting law $Y \in \mathcal Y$8 on the covariate space: $Y \in \mathcal Y$9 When $\mathcal F$0 differs from the observed marginals, AMCMD can focus on particular regions of $\mathcal F$1. Under characteristic $\mathcal F$2, mutual absolute continuity, and regular versions of the conditionals, AMCMD is zero if and only if the two conditional laws agree for $\mathcal F$3-almost every $\mathcal F$4; with bounded Radon–Nikodym derivatives it also satisfies the triangle inequality and defines a metric on conditional laws (Broadbent et al., 14 Apr 2025).

3. Estimation, convergence, and testing

The empirical estimation of CMMD and its variants is dominated by kernel ridge-type inversions. For CMMD$\mathcal F$5, CMMD$\mathcal F$6, CMMD$\mathcal F$7, and general level-$\mathcal F$8 estimators, the standard constructions use Gram matrices such as $\mathcal F$9 and regularized inverses $\mathcal G$0 or $\mathcal G$1, with closed-form trace expressions for the resulting discrepancies (Moskvichev et al., 4 May 2026).

For pointwise MCMD, one estimates conditional embeddings by regularized least squares: $\mathcal G$2 and then computes

$\mathcal G$3

Under boundedness assumptions on $\mathcal G$4 and $\mathcal G$5 with $\mathcal G$6, the conditional mean embedding error is $\mathcal G$7, and the plug-in MCMD estimator is consistent with the same induced rate (Goff et al., 23 Oct 2025).

For AMCMD, a closed-form estimator is obtained by expanding the RKHS norm and rearranging traces: $\mathcal G$8 with computational cost $\mathcal G$9. Under bounded universal $k_x$0, bounded $k_x$1, smoothness of the true conditional operators, and regularization $k_x$2 with $k_x$3, this estimator converges at rate

$k_x$4

(Broadbent et al., 14 Apr 2025).

A distinct development is the doubly robust estimator for CMMD. It introduces an indicator $k_x$5 and propensity $k_x$6, forms pseudo-outcomes that combine inverse-propensity weighting with regression-style conditional mean models, and assembles an operator $k_x$7 whose Hilbert–Schmidt norm yields DR-CMMD$k_x$8 and whose trace form yields DR-CMMD$k_x$9. The doubly robust property is that consistency is retained if either the propensity estimator converges or the conditional mean regression model converges (Moskvichev et al., 4 May 2026).

CMMD also supports hypothesis testing. When $k_y$0, one can pool the samples, randomly split them into two groups, recompute the discrepancy, and obtain an empirical null by bootstrap. When $k_y$1, the same work proposes conditional resampling: for each sample, resample a Bernoulli label using $k_y$2, reassign the sample to $k_y$3 or $k_y$4, recompute CMMD, and approximate the null distribution that way (Moskvichev et al., 4 May 2026).

4. Deep-learning objectives: generative modeling and image classification

The first explicit deep-learning use of CMMD was in Conditional Generative Moment-Matching Networks (CGMMN), which train a conditional generator $k_y$5 by stochastic gradient descent so that the generated conditional distribution matches the data conditional distribution under empirical CMMD (Ren et al., 2016). The loss depends on the generated outputs only through Gram matrices on the response variables, and gradients are computed by back-propagation through the trace form. The paper evaluates CGMMN on predictive modeling, contextual generation, and Bayesian dark knowledge, and reports competitive performance across these tasks (Ren et al., 2016).

In image classification, CMMD is used differently: the discrepancy is computed between the true label conditional distribution and the model’s predicted label conditional distribution. The main issue identified for this setting is that a fixed Gaussian or Laplace kernel on raw inputs often fails to distinguish within-class from between-class samples. To address this, the Kernel Learning Network (KLN) replaces $k_y$6 by $k_y$7, where $k_y$8 is a learnable injective map implemented by the encoder of an auto-encoder and $k_y$9 is a fixed characteristic kernel on the latent representation. The resulting objective is

$$C_{XX}=E[\phi_x(X)\otimes \phi_x(X)]-\mu_x\otimes \mu_x,\qquad C_{YX}=E[\phi_y(Y)\otimes \phi_x(X)]-\mu_y\otimes \mu_x,$$0

and, in the semi-supervised case,

$$C_{XX}=E[\phi_x(X)\otimes \phi_x(X)]-\mu_x\otimes \mu_x,\qquad C_{YX}=E[\phi_y(Y)\otimes \phi_x(X)]-\mu_y\otimes \mu_x,$$1

where

$$C_{XX}=E[\phi_x(X)\otimes \phi_x(X)]-\mu_x\otimes \mu_x,\qquad C_{YX}=E[\phi_y(Y)\otimes \phi_x(X)]-\mu_y\otimes \mu_x,$$2

Training proceeds by mini-batch stochastic gradient descent or Adam on disjoint mini-batches $$C_{XX}=E[\phi_x(X)\otimes \phi_x(X)]-\mu_x\otimes \mu_x,\qquad C_{YX}=E[\phi_y(Y)\otimes \phi_x(X)]-\mu_y\otimes \mu_x,$$3 and $$C_{XX}=E[\phi_x(X)\otimes \phi_x(X)]-\mu_x\otimes \mu_x,\qquad C_{YX}=E[\phi_y(Y)\otimes \phi_x(X)]-\mu_y\otimes \mu_x,$$4, latent features $$C_{XX}=E[\phi_x(X)\otimes \phi_x(X)]-\mu_x\otimes \mu_x,\qquad C_{YX}=E[\phi_y(Y)\otimes \phi_x(X)]-\mu_y\otimes \mu_x,$$5, current pseudo-labels $$C_{XX}=E[\phi_x(X)\otimes \phi_x(X)]-\mu_x\otimes \mu_x,\qquad C_{YX}=E[\phi_y(Y)\otimes \phi_x(X)]-\mu_y\otimes \mu_x,$$6, the corresponding Gram matrices, and the total loss (Ren et al., 2020).

Theoretical support for KLN comes from two statements: a characteristic kernel composed with an injective map remains characteristic, and the auto-encoder penalty encourages the encoder to have a left inverse on the data manifold. This suggests that the learned compound kernel preserves the discriminative guarantees associated with characteristic kernels while improving class separation (Ren et al., 2020).

On MNIST, SVHN, CIFAR-10, and CIFAR-100, the reported supervised test error rates are lower for KLN than for CGMMN-CNN:

Dataset	CGMMN-CNN error (%)	KLN error (%)
MNIST	0.45	0.39
SVHN	2.01	1.56
CIFAR-10	6.61	5.15
CIFAR-100	24.84	22.63

The same study reports semi-supervised errors of $$C_{XX}=E[\phi_x(X)\otimes \phi_x(X)]-\mu_x\otimes \mu_x,\qquad C_{YX}=E[\phi_y(Y)\otimes \phi_x(X)]-\mu_y\otimes \mu_x,$$7 on MNIST with 100 labels, $$C_{XX}=E[\phi_x(X)\otimes \phi_x(X)]-\mu_x\otimes \mu_x,\qquad C_{YX}=E[\phi_y(Y)\otimes \phi_x(X)]-\mu_y\otimes \mu_x,$$8 on SVHN with 1,000 labels, and $$C_{XX}=E[\phi_x(X)\otimes \phi_x(X)]-\mu_x\otimes \mu_x,\qquad C_{YX}=E[\phi_y(Y)\otimes \phi_x(X)]-\mu_y\otimes \mu_x,$$9 on CIFAR-10 with 4,000 labels. Ablations replacing the jointly learned encoder with identity, PCA, or a pre-trained auto-encoder produce much worse accuracy, and kernel histograms show substantially cleaner same-class versus different-class separation under KLN (Ren et al., 2020).

5. Domain adaptation, compression, and model averaging

In domain adaptation, CMMD arises as a special case of a unified joint MMD (JMMD) when the label kernel is chosen as the Kronecker-delta kernel

$_1$00

Under that choice, the unified trace formula becomes a sum of classwise MMD terms and recovers the standard class-conditional alignment objective (Wang et al., 2021).

The same work identifies two drawbacks. First, class-conditional alignment may degrade the feature–label dependence that benefits classification, because it minimizes the discrepancy between joint feature-label embeddings without explicitly preserving the domain-specific dependence norms. Second, CMMD and its weighted variant are sensitive to label-distribution shift: if $_1$01, the resulting classwise mean alignment can be incorrect. The proposed remedies are a modified MMD matrix

$_1$02

which incorporates Hilbert–Schmidt independence criterion terms to promote dependence, and a robust label-kernel $_1$03 whose entries implement per-class reweighting by the ratio $_1$04 (Wang et al., 2021).

For distribution compression, AMCMD supplies a direct target for preserving conditional distributions rather than only the joint distribution. This leads to Average Conditional Kernel Herding (ACKH), a greedy linear-time algorithm, and Average Conditional Kernel Inducing Points (ACKIP), a jointly optimized alternative with the same linear dependence on the original sample size. The same paper introduces joint-distribution baselines JKH and JKIP; empirically, ACKIP outperforms both joint distribution compression and greedy conditional compression, while JKIP consistently outperforms JKH (Broadbent et al., 14 Apr 2025).

For conditional generative model averaging, CMMD is used in a sample-based, likelihood-free setting. Given candidate generators $_1$05 and mixture weights $_1$06, the empirical cMMD loss is

$_1$07

StaticMA constrains the weights to be fixed in $_1$08 and solves a convex quadratic program over the simplex, while MoEMA parameterizes input-adaptive weights through a softmax neural-network gate and optimizes the same objective by SGD or Adam. The framework establishes in-sample and out-of-sample asymptotic optimality for StaticMA and MoEMA, together with consistency of the estimated adaptive weight function under regularity conditions (Gong et al., 5 Jul 2026).

6. Causal transferability, empirical behavior, and practical considerations

MCMD is used as an atomic component in the Structural Causal Model Distance (SCMD). For two structural causal models, SCMD sums pairwise interventional discrepancies, each of which reduces to MCMD when no additional adjustment set is required. This construction is designed to capture differences in causal effects in addition to observational distributional differences (Goff et al., 23 Oct 2025).

Two toy cases in the SCMD study illustrate the role of conditional discrepancies. In a linear-Gaussian case with the same graph but different slope parameters, MCMD detects the parametric shift directly through the change in $_1$09. In a second case with reversed graph structure but the same observational joint distribution, both $_1$10 and $_1$11 are nonzero, so SCMD is positive although observational MMD is zero (Goff et al., 23 Oct 2025).

Several recurring interpretive points emerge across the literature. One is terminological: CMMD does not denote a single unique construction. Depending on the paper, it may refer to an operator norm between conditional mean operators, an average of pointwise conditional mean embedding differences, a pointwise conditional discrepancy, or a special label-kernel instance of a joint-distribution discrepancy (Moskvichev et al., 4 May 2026). A second is that more smoothing is not uniformly preferable. In synthetic experiments for level-$_1$12 CMMD, extra smoothing hurts power in one setting and helps up to a point in another, making the bias–variance trade-off induced by $_1$13 explicit (Moskvichev et al., 4 May 2026). A third is that conditional alignment objectives can have side effects: in domain adaptation they may reduce discriminability or mis-handle label shift unless additional corrections are introduced (Wang et al., 2021).

The practical recommendations in the cited work are correspondingly technical. Characteristic kernels are central because they ensure identifiability of conditional laws; typical choices include Gaussian, Laplacian, and Matérn kernels, while mixtures of Gaussian kernels at multiple scales are used in KLN (Moskvichev et al., 4 May 2026, Ren et al., 2020). Regularization $_1$14 ensures stable inversion of empirical covariance operators, with conditions such as $_1$15 and $_1$16 used for consistency statements (Moskvichev et al., 4 May 2026). Naïve estimators are typically $_1$17 because of matrix inversions, motivating low-rank approximations such as Nyström and random Fourier features, distributed or mini-batch variants, and primal-form computations when $_1$18 is finite-dimensional (Ren et al., 2016, Moskvichev et al., 4 May 2026).

Taken together, these developments position CMMD not as a single formula but as a kernel-embedding framework for conditional-distribution comparison. Its common core is the use of RKHS structure to represent and compare conditional laws non-parametrically; its differences lie in whether one compares operators, pointwise embeddings, averaged embeddings, or joint embeddings, and in how those objects are estimated and optimized for a particular task.