Conditional Multi-Kernel MMD

• CMK-MMD is a discrepancy measure that extends CMMD by integrating multiple kernel representations to compare conditional embedding operators.
• It leverages distinct kernels on conditioning and output spaces to capture heterogeneous scales, improving robustness and sensitivity in modeling.
• Its practical implementation relies on empirical Gram matrix estimation and inversion, which supports applications in generative and predictive modeling.

Searching arXiv for recent and foundational papers on CMMD / multi-kernel MMD / kernel discrepancies. Conditional Multi-Kernel Max Mean Discrepancy (CMK-MMD) is a multi-kernel extension of conditional maximum mean discrepancy (CMMD), a kernel-embedding criterion for comparing conditional distributions through the distance between conditional embedding operators. In the formulation introduced by "Conditional Generative Moment-Matching Networks" (Ren et al., 2016), CMMD measures the squared Hilbert–Schmidt norm between two conditional mean embeddings, one associated with a target conditional distribution and one with a model. The original paper does not explicitly define or optimize a multi-kernel CMMD, but it states the ingredients that make such an extension straightforward: distinct kernels on conditions and outputs, an empirical trace form built from Gram matrices, and linear dependence of the loss on the chosen kernel. CMK-MMD therefore denotes a natural conditional analogue of multi-kernel MMD, in which several kernels are combined on the output space, on the conditioning space, or on both (Ren et al., 2016).

1. Conditional discrepancy as an operator norm

CMK-MMD inherits its basic structure from the kernel mean embedding view of conditional distributions. Let $\mathcal{F}$ be the RKHS for $X$ with kernel $k_X$, and $\mathcal{G}$ the RKHS for $Y$ with kernel $k_Y$. For a conditional distribution $P(Y\mid X)$, the conditional mean embedding is

$\mu_{Y\mid x} := \mathbb{E}_{Y\mid x}[\phi(Y)],$

and is represented by a linear operator

$$C_{Y\mid X} : \mathcal{F} \to \mathcal{G}, \qquad \mu_{Y\mid x} = C_{Y\mid X}\phi(x).$$

Under the assumptions quoted from Song et al. (2009), the conditional embedding operator is

$C_{Y\mid X} = C_{YX} C_{XX}^{-1},$

where $X$0 and $X$1 are covariance operators (Ren et al., 2016).

CMMD is then defined by comparing two such operators, for example $X$2 and $X$3, through

$X$4

The foundational theorem stated in the CGMMN paper says that, under universality and regularity assumptions, equality of conditional embeddings implies equality of conditional distributions in the sense that for every fixed $X$5, $X$6 (Ren et al., 2016).

Empirically, with paired samples $X$7, the regularized conditional embedding estimator is

$X$8

where $X$9, $k_X$0, and $k_X$1. This estimator is always an element of the tensor product space, even when the theoretical Hilbert–Schmidt existence conditions may fail, which is why the paper treats it as a practical approximation (Ren et al., 2016).

The empirical CMMD between a data set $k_X$2 and a synthetic set $k_X$3 admits the explicit trace form

$k_X$4

with $k_X$5 and $k_X$6. This formula is the computational core from which CMK-MMD is obtained (Ren et al., 2016).

2. From CMMD to CMK-MMD

The multi-kernel extension follows from the fact that CMMD already uses two distinct kernels: $k_X$7 on conditions and $k_X$8 on outputs. The CGMMN paper states that one may use Gaussian kernels and that it also tried combining several Gaussian kernels with different bandwidths, although this “didn't make noticeable difference” in its reported experiments (Ren et al., 2016). That statement is the direct basis for CMK-MMD.

For multi-kernel structure on the output space, one replaces the output kernel by

$k_X$9

with nonnegative weights $\mathcal{G}$0. Because CMMD depends linearly on the output kernel through the output Gram matrices,

$\mathcal{G}$1

Substituting these matrices into the CMMD trace formula yields a loss that is linear in the kernel combination (Ren et al., 2016).

A natural definition therefore is

$\mathcal{G}$2

For multi-kernel structure only in $\mathcal{G}$3, the empirical loss reduces to

$\mathcal{G}$4

For multi-kernel structure in both $\mathcal{G}$5 and $\mathcal{G}$6, the same trace expression is used after replacing the input Gram matrices $\mathcal{G}$7 and the output Gram matrices $\mathcal{G}$8 by weighted sums of their per-kernel counterparts (Ren et al., 2016).

This construction clarifies an important conceptual difference from unconditional MMD. In the equal-sample-size remark in the CGMMN paper, ordinary MMD corresponds to uniformly weighted comparison of output Gram matrices, whereas CMMD introduces non-uniform weights determined by the similarity of the conditioning variables $\mathcal{G}$9. CMK-MMD preserves that conditional weighting while adding multiple notions of similarity through kernel combinations (Ren et al., 2016).

A plausible implication is that CMK-MMD should be understood less as a wholly new discrepancy than as a kernel-design layer placed on top of CMMD’s operator-valued comparison.

3. Optimization in conditional generative modeling

CMK-MMD is most naturally understood through the CGMMN training objective. CGMMN defines an implicit conditional generator

$Y$0

where $Y$1 is drawn from the simple prior

$Y$2

The model is trained by minimizing empirical CMMD between real pairs $Y$3 and generated pairs $Y$4, with $Y$5 taken from data and $Y$6 (Ren et al., 2016).

The paper emphasizes a structural simplification that remains relevant for CMK-MMD: the inverse matrices $Y$7 and $Y$8 depend only on the condition samples $Y$9, not on model parameters. Consequently, gradients propagate only through the output Gram matrices. For any parameter $k_Y$0,

$k_Y$1

with $k_Y$2 computed by standard backpropagation and $k_Y$3 obtained by differentiating the Gram-matrix objective. For Gaussian RBF output kernels, these derivatives have closed forms (Ren et al., 2016).

The training algorithm is minibatch SGD: split data into minibatches, generate a synthetic minibatch using the same condition inputs and fresh noise, evaluate the CMMD loss, and update parameters with Adam. The dominant cost is inversion of minibatch input Gram matrices, which is $k_Y$4 in minibatch size $k_Y$5; the paper therefore uses moderate minibatches such as $k_Y$6–$k_Y$7 (Ren et al., 2016).

In a CMK-MMD implementation, only small changes are required relative to the single-kernel objective described in the paper: maintain multiple output Gram matrices, or multiple input and output Gram matrices, aggregate them with fixed or tuned weights, and backpropagate through the summed loss. This is explicitly described in the synthesis portion of the provided material as the practical route from CMMD to CMK-MMD (Ren et al., 2016).

4. Kernel design, aggregation, and spectral interpretations

Later work on MMD gives a broader framework for interpreting and extending CMK-MMD. The following works are especially relevant.

Work	Main contribution	Relevance to CMK-MMD
(Ren et al., 2016)	CMMD and CGMMN	Foundational conditional operator loss
(Takhanov, 2021)	MMD as comparison of local moments	Spectral interpretation of kernel choice
(Schrab, 4 Mar 2025)	Adaptive kernel pooling	Statistic-level multi-kernel combination
(Chatterjee et al., 2023)	Mahalanobis aggregation of MMDs	Covariance-aware multi-kernel aggregation
(Cheng et al., 2017)	Anisotropic, reference-based MMD	Geometry-aware kernel construction
(Naslidnyk, 25 Feb 2026)	MMD for conditional expectations	Operator-based conditional estimation

The practical motivation for multi-kernel structure is strongly tied to bandwidth sensitivity. "A Practical Introduction to Kernel Discrepancies: MMD, HSIC & KSD" states that for translation-invariant radial kernels, $k_Y$8 as $k_Y$9 or $P(Y\mid X)$0, regardless of whether $P(Y\mid X)$1. It therefore proposes a parameter-free kernel collection built from Gaussian and Laplace kernels over bandwidths obtained from the $P(Y\mid X)$2 and $P(Y\mid X)$3 quantiles of pairwise distances, and combines per-kernel statistics by mean pooling, maximum pooling, or fuse pooling with variance normalization (Schrab, 4 Mar 2025). This directly motivates CMK-MMD constructions in which several conditional kernels are pooled rather than a single bandwidth being fixed a priori.

"Boosting the Power of Kernel Two-Sample Tests" develops a different aggregation principle: form a vector of MMD estimates over multiple kernels and combine them through a Mahalanobis quadratic form using the inverse null covariance matrix. The resulting test is universally consistent and has non-trivial asymptotic Pitman efficiency, because aggregation is performed in a covariance-aware way rather than through a simple average (Chatterjee et al., 2023). This suggests a statistic-level extension of CMK-MMD in which several conditional discrepancies are combined after estimating their joint variability.

A second line of interpretation is spectral. "How many moments does MMD compare?" shows that, for kernels represented through pseudo-differential operators, MMD can be interpreted as comparing a certain number of local moments determined by the decay of singular values. In finite-rank form, the discrepancy compares exactly $P(Y\mid X)$4 local moments; for general symbols, an effective number $P(Y\mid X)$5 of local moments dominates when the singular values decay rapidly (Takhanov, 2021). This suggests that CMK-MMD is not merely a bandwidth-robustification device: a kernel mixture changes which conditional local moments of $P(Y\mid X)$6 are emphasized.

A third line concerns kernel geometry. "Two-sample Statistics Based on Anisotropic Kernels" constructs MMD statistics from anisotropic kernels using local covariance matrices and reference points, yielding geometry-adaptive tests with reduced computational cost when $P(Y\mid X)$7. The paper explicitly notes that such constructions can be imported into multi-kernel and conditional settings, including kernels whose local structure depends on covariates or conditioning regions (Cheng et al., 2017). For CMK-MMD, this suggests replacing isotropic RBF components by anisotropic, context-dependent base kernels.

5. Applications and neighboring conditional constructions

Within the original CGMMN paper, CMMD is the central criterion in three application classes: predictive modeling, contextual generation, and Bayesian dark knowledge. In predictive modeling, the objective is to learn $P(Y\mid X)$8 for tasks such as classification and regression; in contextual generation, labels are treated as conditions and images as outputs; in Bayesian dark knowledge, the teacher produces samples from a predictive distribution and a smaller student CGMMN is trained to match that conditional distribution (Ren et al., 2016).

These applications clarify why conditional discrepancy is needed. The goal is not to align an unconditional output marginal $P(Y\mid X)$9, but the distribution of outputs given each condition. The CGMMN paper explicitly contrasts CMMD with unconditional MMD and notes that CMMD aligns outputs using non-uniform weights induced by input similarity. A plausible implication is that CMK-MMD is most useful when the output space exhibits heterogeneous scales or structures across conditions, because a single output kernel may underresolve some regimes while oversmoothing others (Ren et al., 2016).

The same operator perspective appears in later work on conditional expectations. "Scalable Kernel-Based Distances for Statistical Inference and Integration" studies MMD-based estimators for conditional expectations through Conditional Bayesian Quadrature (CBQ). There, for each parameter value $\mu_{Y\mid x} := \mathbb{E}_{Y\mid x}[\phi(Y)],$0, Stage 1 uses Bayesian quadrature weights that minimize an MMD between $\mu_{Y\mid x} := \mathbb{E}_{Y\mid x}[\phi(Y)],$1 and a weighted empirical approximation, and Stage 2 performs GP regression across $\mu_{Y\mid x} := \mathbb{E}_{Y\mid x}[\phi(Y)],$2 with heteroscedastic noise determined by the Stage-1 BQ variance, which is exactly an MMD squared (Naslidnyk, 25 Feb 2026). This is not a CMK-MMD construction in name, but it shows that conditional operator estimation, kernel quadrature, and MMD-based uncertainty quantification naturally coexist.

A broader methodological implication is that CMK-MMD sits at the intersection of three strands: conditional embeddings, multi-kernel aggregation, and kernel-based inference with operator-valued structure. That intersection is explicit in the supplied material even when the exact term “CMK-MMD” is not formalized in every source.

6. Assumptions, limitations, and common misconceptions

The principal theoretical assumptions are those required for CMMD to characterize conditional distributions: universality of the RKHS on $\mu_{Y\mid x} := \mathbb{E}_{Y\mid x}[\phi(Y)],$3, inclusion of the conditional expectations $\mu_{Y\mid x} := \mathbb{E}_{Y\mid x}[\phi(Y)],$4 and $\mu_{Y\mid x} := \mathbb{E}_{Y\mid x}[\phi(Y)],$5 in $\mu_{Y\mid x} := \mathbb{E}_{Y\mid x}[\phi(Y)],$6, and Hilbert–Schmidt regularity of the conditional embedding operators. The CGMMN material also mentions the smoothness condition that $\mu_{Y\mid x} := \mathbb{E}_{Y\mid x}[\phi(Y)],$7 is Hilbert–Schmidt. In practice these conditions may fail, but the empirical estimator $\mu_{Y\mid x} := \mathbb{E}_{Y\mid x}[\phi(Y)],$8 remains well defined in the tensor product space and is used as an approximation (Ren et al., 2016).

A first misconception is that CMK-MMD is a canonically established standalone discrepancy in the foundational paper. It is not. The original paper introduces CMMD, not an explicitly optimized CMK-MMD, and only remarks that combining several Gaussian kernels with different bandwidths did not make noticeable difference in those experiments (Ren et al., 2016). The term is therefore best understood as a principled extension rather than a closed historical object.

A second misconception is that multi-kernel structure automatically improves performance. The supplied material does not support that conclusion in general. The CGMMN experiments reported no noticeable difference for simple Gaussian-kernel mixtures (Ren et al., 2016), whereas later general MMD work argues that kernel pooling can substantially improve robustness to bandwidth misspecification (Schrab, 4 Mar 2025). The objective view is that benefit is dataset-dependent and tied to whether multiple kernels actually expose distinct informative scales.

A third misconception is that CMMD merely compares conditional means. The operator formulation compares conditional mean embeddings, and equality of these embeddings implies equality of full conditional distributions under the stated assumptions (Ren et al., 2016). At the same time, the spectral analysis of MMD in later work indicates that any chosen kernel effectively emphasizes a finite or approximately finite set of local moments, governed by singular-value decay (Takhanov, 2021). This does not reduce CMK-MMD to ordinary moment matching in a low-order Euclidean sense; it specifies the RKHS-encoded moment structure that the discrepancy sees.

Finally, computational cost remains a structural limitation. In CGMMN, the $\mu_{Y\mid x} := \mathbb{E}_{Y\mid x}[\phi(Y)],$9 inversion of minibatch input Gram matrices is the dominant cost (Ren et al., 2016). Multi-kernel variants inherit this burden unless they exploit approximations, incomplete statistics, reference sets, or linear-time constructions of the kind studied in later MMD literature (Schrab, 4 Mar 2025, Cheng et al., 2017, Chatterjee et al., 2023). A plausible implication is that scalable CMK-MMD will usually require both kernel adaptation and estimator adaptation rather than only adding more kernels.

In this sense, CMK-MMD is best regarded as an operator-valued conditional discrepancy whose essential content is already present in CMMD: conditional kernel embeddings, input-dependent weighting, and Gram-matrix optimization. The multi-kernel extension adds a controlled way to vary the geometry of similarity in the conditioning and output spaces, while later work supplies the statistical, spectral, and computational tools needed to make that extension practical and theoretically interpretable (Ren et al., 2016).