Central Moment Discrepancy (CMD)

• Central Moment Discrepancy (CMD) is a metric that quantifies differences between probability distributions by aligning their central moments up to a specified order.
• CMD leverages higher-order statistical moments to enhance feature alignment in neural networks, improving unsupervised domain adaptation and neural style transfer compared to traditional methods.
• Its efficient computation and minimal hyperparameter tuning make CMD a practical tool for achieving robust, domain-invariant representations in bounded feature spaces.

Central Moment Discrepancy (CMD) is a theoretically grounded metric for quantifying and minimizing differences between probability distributions by explicitly aligning their central moments up to a prescribed order. Developed to address the challenge of domain-invariant representation learning in neural networks, CMD has proven effective in unsupervised domain adaptation and, by extension, in neural style transfer. The central principle is to exploit the statistical informativeness of higher-order centralized moments to better enforce distributional alignment, enabling robust transfer across domains and more faithful representation matching in learning systems (Zellinger et al., 2017, Kalischek et al., 2021).

1. Formal Definition and Theoretical Foundations

Let $P$ and $Q$ be probability distributions on the compact cube $[a,b]^N$, and let $X \sim P$, $Y \sim Q$. For a truncation order $K \geq 2$, the $k$-th central moment vector of $X$ is defined coordinate-wise as

$$c_k(P) = \Bigl(E[(X_j - E[X_j])^k]\Bigr)_{j=1}^N \in \mathbb{R}^N,$$

where $E[X] = (E[X_1], ... , E[X_N])$.

The empirical Central Moment Discrepancy up to order $Q$0 between samples $Q$1 and $Q$2 is

$Q$3

where

$Q$4

In the context of general compactly supported distributions on $Q$5 and weights $Q$6, a more general "primal" form is stated as

$Q$7

with $Q$8 and, for $Q$9, $[a,b]^N$0, where $[a,b]^N$1 denotes all monomials of total degree $[a,b]^N$2 in the components (Kalischek et al., 2021).

CMD possesses key metric properties: it is nonnegative, symmetric, satisfies the triangle inequality, and the condition $[a,b]^N$3 on compact supports. Furthermore, convergence with respect to CMD entails convergence in distribution, as agreement in all centralized moments on a compact set uniquely determines the distribution (Zellinger et al., 2017).

2. Algorithmic Implementation

CMD is designed for efficient mini-batch computation within neural network training pipelines. For domain adaptation, in each iteration with hidden activations $[a,b]^N$4 and $[a,b]^N$5 from source and target domains, respectively, the following pseudo-code computes $[a,b]^N$6:

$k$6 The computed quantity is then scaled by a penalty $[a,b]^N$7 and added to the task-specific loss (e.g., cross-entropy). All operations are differentiable and incur linear complexity $[a,b]^N$8 per batch (Zellinger et al., 2017).

For neural style transfer on convolutional feature maps, CMD is computed between feature activations $[a,b]^N$9 and $X \sim P$0 from output and style images, channel-wise and up to order $X \sim P$1, ensuring activations reside in a known range via, e.g., sigmoid nonlinearity (Kalischek et al., 2021).

3. Comparison with Other Distribution Matching Methods

CMD is specifically contrasted with Maximum Mean Discrepancy (MMD) and other moment-matching methods:

Method	Statistic / Moments Matched	Computational Cost	Kernel/Bandwidth Needed
CMD	All central moments up to $X \sim P$2	$X \sim P$3 (linear)	None
MMD (Gaussian)	Weighted moment sums/all orders	$X \sim P$4 (quadratic)	Yes ($X \sim P$5)
Gram matrix	Non-central second moment	$X \sim P$6 per layer	None
AdaIN, MM	Mean and covariance only	$X \sim P$7	None
OT (Gaussian)	Mean and covariance	$X \sim P$8 (or cubic)	None

CMD explicitly matches each moment up to order $X \sim P$9, avoiding kernel selection and expensive Gram matrix operations. MMD, by contrast, requires careful tuning of kernel bandwidth and involves quadratic complexity in sample size. Gram-based methods and AdaIN only match first and second moments, leaving higher-order characteristics unconstrained. CMD’s explicit moment-order matching provides more complete alignment in practice and theory (Zellinger et al., 2017, Kalischek et al., 2021).

4. Empirical Performance and Applications

CMD has been empirically validated in unsupervised domain adaptation and neural style transfer.

• Office Dataset: With VGG16 features and a 256 neuron adaptation layer, CMD ($Y \sim Q$0, $Y \sim Q$1) achieves 79.9% accuracy, outperforming fine-tuned VGG16 (75.5%) and AdaBN (76.7%), exceeding all prior methods on 4/6 tasks and matching on the remaining two.
• Amazon Reviews Dataset: CMD achieves 79.8% mean classification accuracy ($Y \sim Q$2, $Y \sim Q$3), compared to 78.1% for MMD and 75.2% for the source-only baseline, reaching the state of the art on 9/12 adaptation tasks (Zellinger et al., 2017).
• Neural Style Transfer: CMD-based losses, implemented via the dual or primal form, yielded stylizations rated preferable over methods such as AdaIN, Gram/MMD, MM, OST, and WCT in user studies (CMD-based transfer chosen in 21.7% of cases, highest among six methods). Ablation studies attribute improved texture and color reproduction to explicit matching of higher moments up to $Y \sim Q$4 (Kalischek et al., 2021).

CMD is also noted for computational efficiency; with $Y \sim Q$5, overhead is minor compared to classical Gatys implementations, while OT and other methods incur higher costs per iteration.

5. Metric and Convergence Properties

CMD is a metric on the space of probability distributions supported on a compact interval. The condition $Y \sim Q$6 implies $Y \sim Q$7, as all central moments agree and moment sequences determine the distribution on compact supports. CMD metrizes weak convergence: if $Y \sim Q$8, then $Y \sim Q$9 holds. This metric property distinguishes CMD from non-characteristic kernels (such as the quadratic Gram in MMD), ensuring that empirical minimization of CMD yields convergent distributional alignment (Zellinger et al., 2017).

In the context of polynomial integral-probability metrics (IPMs), CMD corresponds to maximizing the difference of expectations over the class of polynomial test functions up to degree $K \geq 2$0, fully characterized by their central moments. As $K \geq 2$1, CMD metrizes convergence on compact sets (Kalischek et al., 2021).

6. Practical Hyperparameter Selection and Sensitivity

CMD requires minimal hyperparameter tuning. Recommendations derived from empirical sensitivity analyses include:

• Set $K \geq 2$2 to incorporate mean, variance, skewness, kurtosis, and a fifth-order shape descriptor, capturing essential distributional features.
• Default $K \geq 2$3 balances CMD and task loss.
• Reducing $K \geq 2$4 to $K \geq 2$5 yields $K \geq 2$6 of peak performance in domain adaptation.
• CMD displays stable performance for $K \geq 2$7 (accuracy fluctuates by less than 0.5%) and for $K \geq 2$8 in $K \geq 2$9 (Zellinger et al., 2017).
• Ensure all activations to be matched are bounded within a known range $k$0—typically by using activation functions such as $k$1 or a sigmoid, to correctly normalize each term via $k$2.

In neural style transfer, elementwise sigmoid ensures feature activations remain in $k$3, simplifying the normalization for CMD and stabilizing computation (Kalischek et al., 2021).

7. Application-Specific Contexts and Limitations

CMD is broadly applicable in tasks requiring precise distribution alignment in feature space. In domain-adaptive neural networks, CMD functions as a robust regularizer by reducing discrepancies between hidden activations of source and target domains. In neural style transfer, CMD is preferred for more faithfully capturing complex style elements by matching higher-order statistics beyond means and variances. Empirical studies demonstrate that odd moments control brightness/contrast, even moments govern texture and color, and higher moments further enrich stylization. A plausible implication is that matching further moments (higher $k$4) could support even more granular control over distributional alignment, though with diminishing returns beyond $k$5 in typical tasks.

CMD is most naturally applied to distributions on compact sets; ensuring boundedness of input features or activations is critical to preserve theoretical properties and normalization. CMD does not require kernel tuning or expensive matrix computations, but for very high-dimensional settings, the computation of higher-order moments may become more costly, especially if full multivariate moments are included rather than marginal powers.

---

Central Moment Discrepancy constitutes a metric, theoretically justified approach for aligning finite sets of central moments in empirical learning systems, outperforming classical moment-matching methods in tasks including domain adaptation and neural style transfer, while maintaining practical tractability and minimal reliance on hyperparameter tuning (Zellinger et al., 2017, Kalischek et al., 2021).