Enhanced LMMD (ELMMSD)

• The paper introduces ELMMSD which enhances standard LMMD by jointly matching mean and variance structures using a squared-kernel approach in RKHS.
• It incorporates label smoothing to assign soft class weights, improving noise robustness and the alignment of source-target label distributions.
• Experimental results show ELMMSD reduces domain gaps and boosts transfer learning accuracy, particularly in bearing fault diagnosis scenarios.

Enhanced Local Maximum Mean Squared Discrepancy (ELMMSD) is a domain adaptation loss designed to improve subdomain alignment for transfer learning scenarios with distribution discrepancies and limited labeled data, notably in resource-constrained environments such as bearing fault diagnosis. It extends standard Local Maximum Mean Discrepancy (LMMD) by (1) jointly matching mean and variance structure in the Reproducing Kernel Hilbert Space (RKHS) using a squared-kernel approach, and (2) incorporating class-probability–weighted "label smoothing," leading to greater noise robustness and improved alignment of label distributions between source and target subdomains (Kavianpour et al., 13 Jan 2025).

1. Problem Context and Standard LMMD

LMMD targets class-conditional domain adaptation where one deals with a labeled source domain $\mathcal{D}_s = \{(x_i^s, y_i^s)\}_{i=1}^{n_s}$ and an unlabeled target domain $\mathcal{D}_t = \{x_j^t\}_{j=1}^{n_t}$ across $n_c$ classes. The objective is to align source $p_s(x|y=c)$ and target $p_t(x|y=c)$ subdomains through RKHS-embedded feature statistics.

Maximum Mean Discrepancy (MMD) measures domain shift via the squared RKHS norm between empirical means for each class:

$$L_\mathrm{LMMD} = \frac{1}{n_c} \sum_{c=1}^{n_c} \|\mu_s^c - \mu_t^c\|_\mathcal{H}^2$$

with empirical calculation over kernel similarities for pseudo-labeled target and labeled source instances.

2. ELMMSD: Enhancements over LMMD

ELMMSD introduces two modifications to standard LMMD:

2.1 Mean and Variance Matching via MMSD

The first enhancement replaces the base kernel $k(x, x')$ with its tensor-square $k^2(x, x')$, thereby capturing not just mean but also second-order (variance/covariance) structure in RKHS. This is termed Maximum Mean Squared Discrepancy (MMSD). In practice, a multi-kernel strategy is used:

$$k^2(\cdot, \cdot) = \sum_{u=1}^U \mu_u k_u^2(\cdot, \cdot)$$

where $\{k_u\}$ span multiple Gaussian bandwidths, improving sensitivity to diverse data characteristics.

2.2 Soft Label Integration via Label Smoothing

The second enhancement leverages label smoothing to reduce overconfidence in class-response and to mitigate label noise. For source labels $y_i^s \in \mathbb{R}^{n_c}$, smoothed vectors are constructed:

$$S(y_i^s) = (1-\epsilon) y_i^s + \frac{\epsilon}{n_c} \mathbf{1}$$

($\epsilon$ typically $0.1$). These smoothed probabilities, together with softmax outputs for target data, are used as class-membership weights ($\omega_i^{sc}$ for source, $\omega_j^{tc}$ for target), resulting in soft rather than hard assignment of samples to classes in the discrepancy calculation.

3. Formal Definition and Implementation

ELMMSD is instantiated at two fully connected layers of the neural network (denoted $FC_1$ and $FC_2$). For each layer $\ell \in \{1,2\}$:

$$\hat{d}_{z_\ell} = \frac{1}{n_c} \sum_{c=1}^{n_c} \left[ \sum_{i,j=1}^{n_s} \omega_i^{sc} \omega_j^{sc} k^2(\tilde{z}_i^{\ell s}, \tilde{z}_j^{\ell s}) + \sum_{i,j=1}^{n_t} \omega_i^{tc} \omega_j^{tc} k^2(z_i^{\ell t}, z_j^{\ell t}) - 2 \sum_{i=1}^{n_s} \sum_{j=1}^{n_t} \omega_i^{sc} \omega_j^{tc} k^2(\tilde{z}_i^{\ell s}, z_j^{\ell t}) \right]$$

where $\tilde{z}_i^{\ell s}$ is the layer-$\ell$ source output, $z_j^{\ell t}$ is the target counterpart, and class weights are as given above.

The final ELMMSD loss across both layers is

$L_\mathrm{ELMMSD} = \hat{d}_{z_1} + \hat{d}_{z_2}$

Interpretively, the three loss terms per class $c$ penalize mismatches in intra-class compactness (source/target) and mean location (cross-domain class centers), with weights proportional to the soft class probabilities, thereby improving sensitivity and robustness relative to LMMD.

4. Integration into the Progressive Knowledge Distillation Framework

ELMMSD is incorporated into a composite training loss with three main elements:

• Source cross-entropy with label smoothing:

$$L_{cls} = -\sum_{c=1}^{n_c} [(1-\epsilon) y_i^s[c] + \epsilon/n_c] \log p(c|x_i^s)$$

• Subdomain-adaptation loss:

$$L_{SDA} = L_{cls} + \lambda_{SDA}(e) L_\mathrm{ELMMSD}$$

with a scheduling function $\lambda_{SDA}(e)$ increasing over epochs.

• Knowledge-distillation losses (with temperature scaling and ramped trade-off parameter $\lambda_e$):

$$L_\mathrm{total}(e) = (1 - \lambda_e) L_{SDA} + \lambda_e [L_{KD}^{T} + L_{KD}^{S}]$$

where $L_{KD}^{T}$ and $L_{KD}^{S}$ are KL divergences between teacher/student softened predictions on target and source, respectively.

The optimization proceeds by alternating forward passes, computation of smoothed logits and pseudo-labels, ELMMSD/loss evaluation at two layers, and updating weights per mini-batch, together with ramping $\lambda_{SDA}$ and $\lambda_e$.

5. Experimental Validation and Ablative Results

Ablation studies demonstrate that replacing ELMMSD with alternative adaptation losses (e.g., DANN, LMMD, MMSD with hard labels) results in a reduction of student accuracy on the CWRU dataset from 97–98% (ELMMSD) to 93–96%. ELMMSD also achieves the largest decrease in both global $A$-distance and class-conditional $A_L$-distance, indicating more effective domain and subdomain alignment.

Empirically, label smoothing alone increases teacher accuracy by $\sim$0.4% (from ~99.2% to 99.6% on the A1→A4 task), highlighting the contribution of soft-label assignment to robust subdomain matching.

Dynamic scheduling of $\lambda_{SDA}$ and use of multi-kernel $k^2$ yield improved convergence and accuracy compared to fixed-weight baselines. ELMMSD outperforms MMSD by 1–2% and LMMD by 3–4%, while maintaining a highly compact student model (0.028 MB, 32M FLOPs) with only $\approx 2\%$ accuracy penalty relative to the full teacher (Kavianpour et al., 13 Jan 2025).

6. Key Hyperparameters and Practical Considerations

Essential ELMMSD hyperparameters include:

• Label smoothing $\epsilon=0.1$
• KD temperature $\tau=20$
• Trade-off scheduling parameters $\alpha_1, \alpha_2$ (e.g., $0.1 \rightarrow 0.9$ for $\lambda_e$)
• Multi-kernel squared bandwidths: $\{0.001, 0.01, 1, 10, 100\}$
• Classification loss trade-off $\lambda_{CLS}=0.8$

Applications involve deploying student models for rapid, low-resource bearing fault diagnosis under variable conditions, emphasizing the compactness and efficiency of student models enabled by this method.

7. Summary and Interpretation

ELMMSD extends Local Maximum Mean Discrepancy by simultaneously aligning first and second moments in RKHS using squared kernels, and by softening source and target class allocations with smoothed label probabilities. This dual enhancement yields more reliable and noise-robust subdomain alignment, which directly translates into higher transfer diagnostic accuracy in resource-constrained settings. Empirical evidence highlights its ability to shrink domain and subdomain gaps beyond conventional adaptation baselines and to support effective knowledge distillation with minimal loss in student model performance (Kavianpour et al., 13 Jan 2025).