Cross-modal Knowledge Distillation (CMKD)

• Cross-modal Knowledge Distillation (CMKD) is a technique that transfers knowledge from richer, multi-modal teacher models to constrained student models, mitigating modality mismatch.
• It employs both feature-level and response-level distillation methods to align heterogeneous representations, such as LiDAR to image or text to speech channels.
• Applications span 3D perception, speech recognition, and audio-visual understanding, while recent advances address challenges like modality noise and teacher overfitting.

Cross-modal Knowledge Distillation (CMKD) denotes a family of learning strategies in which knowledge is transferred between models that operate on different modalities, or under different modality-availability regimes, so that information available during training can improve a model that must ultimately operate with fewer or weaker modalities at inference. In the most common formulation, a teacher is trained on a stronger, richer, or privileged modality and supervises a student trained on a weaker or deployment-constrained modality; however, the literature also includes asymmetric unimodal-to-multimodal transfer, teacher-free collaborative training, and source-to-target transfer through learned priors rather than per-sample teacher outputs (Hong et al., 2022, Zhao et al., 2020, Ferrod et al., 30 May 2025).

1. Definition and scope

CMKD is typically motivated by modality mismatch between training and deployment. In monocular $3$D object detection, LiDAR is available during training but not at inference, so a LiDAR-based detector serves as a privileged teacher for an image-only student (Hong et al., 2022). In speech recognition, large text-only LLMs can supervise acoustic models, even though the teacher and student differ both in modality and, in some settings, in output units (Lee et al., 2021). In speech LLMs, a text-only LLM can supervise a speech LLM so that spoken-query behavior is aligned with text-conditioned reasoning (Wang et al., 18 Sep 2025). A related but distinct setting appears when multimodal paired data are scarce: an audio-only SELD model trained on much larger audio data can teach an audio-visual student trained on limited paired audio-video data (Wang et al., 17 Aug 2025).

Most CMKD work assumes paired cross-modal samples during training, because teacher and student supervision must be related at the instance level. A notable extension is Cross-Modal Knowledge Generalization, which addresses the case where the target dataset contains only the weak modality and the superior modality is completely unavailable; in that setting, cross-modal knowledge learned on a source dataset is transferred as a learned prior over student parameters rather than as direct teacher supervision on target samples (Zhao et al., 2020). This suggests that CMKD is not confined to one teacher–student protocol, but is better understood as a broader mechanism for transferring task-relevant structure across heterogeneous observation spaces.

2. Terminology and conceptual boundaries

The acronym “CMKD” is not uniform across the literature. In the audio-classification papers “CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation” and “FastAST,” CMKD explicitly means Cross-Model Knowledge Distillation, not cross-modal distillation: teacher and student process the same audio spectrogram modality, but belong to different model families such as CNNs and ASTs (Gong et al., 2022, Behera et al., 2024). That distinction is central, because these papers address architectural heterogeneity rather than modality transfer.

Within true cross-modal usage, the phrase “cross-modal” itself has multiple concrete meanings. In speech LLMs, it refers specifically to transfer across input modalities, with a text teacher supervising a speech-conditioned student through text-to-text and speech-to-text channels (Wang et al., 18 Sep 2025). In low-resource audio-visual SELD, it refers to transfer from an audio-only teacher into an audio-visual student, so the knowledge crosses from one modality regime to another rather than from multimodal teacher to unimodal student (Wang et al., 17 Aug 2025). In supervised cross-modal hashing, SODA reformulates multi-label annotations as a textual modality and distills from an image–label teacher stage into an image–text student stage; there, the transferred object is a teacher-guided Hamming space rather than conventional soft class logits (Sun et al., 7 Oct 2025). These cases show that CMKD is defined less by a single architecture than by a transfer relation across heterogeneous modalities or modality sets.

A recurrent misconception is that a stronger or multimodal teacher should automatically be a better cross-modal teacher. Case studies on AV-MNIST and NYU Depth V2 show the opposite can occur: on AV-MNIST, a multimodal teacher with 91.61% accuracy yielded a student at 69.73%, whereas a weaker unimodal teacher at 84.57% yielded 70.10%; on NYU Depth V2, a multimodal teacher at 51.00 mIoU did not outperform unimodal KD for the RGB student (Xue et al., 2022). This suggests that teacher accuracy and transferability are distinct quantities.

3. Canonical transfer mechanisms

A standard CMKD formulation combines a task loss with a distillation loss,

$$\mathcal{L} = \rho \mathcal{L}_{task} + (1-\rho)\mathcal{L}_{kd},$$

with $\mathcal{L}_{kd}$ often instantiated as a KL divergence between teacher and student outputs (Xue et al., 2022). Many systems remain close to this template but differ in what is distilled, how modalities are aligned, and whether the teacher supervises outputs, features, relations, or learned priors.

At the response level, CMKD often uses softened class or token distributions. Automatic Cued Speech recognition distills from a DeepSpeech2 acoustic teacher into a visual Cued Speech student using frame-level KL divergence and sequence-level cosine similarity on logits, with additional uncertainty-based multi-task weighting for balancing teacher and label supervision (Wang et al., 2021). Language-model-to-acoustic-model distillation in ASR uses a multi-task formulation with separate supervised and KD heads, precisely to avoid the label-interpolation limitations of earlier LM-to-ASR distillation (Lee et al., 2021). Speech LLM CMKD likewise uses response-level hard targets and KL supervision, but distinguishes a text-to-text channel for preserving textual competence from a speech-to-text channel for aligning speech-conditioned reasoning to a text teacher (Wang et al., 18 Sep 2025).

Feature-level CMKD is equally prominent. Monocular $3$D detection distills LiDAR BEV features into an image-based detector through

$$\mathcal{L}_{feat} = MSE(\hat{F}^{bev}_{I},F^{bev}_{L}),$$

and supplements this with confidence-weighted response distillation over $3$D boxes and classes (Hong et al., 2022). Audio-visual SELD combines response KD with multi-stage, multi-scale feature KD, aligning intermediate representations through hierarchical context loss in addition to output matching (Wang et al., 17 Aug 2025). A dimensional-structure approach for optical-to-radar transfer enforces output-channel decorrelation and intermediate-feature uniformity, interpreting CMKD as transfer of representational structure rather than only semantic outputs (Si et al., 2023). A more recent frequency-decoupled framework separates low- and high-frequency feature components, applying strong MSE alignment to low-frequency features and relaxed LogMSE alignment to high-frequency ones, on the grounds that low-frequency structure is more cross-modally consistent (Liu et al., 25 Nov 2025).

Other works distill relations rather than raw outputs or features. SODA transfers semantic affinity structure and a unified Hamming/code space from an image–label teacher stage into image–text hashing, so that the student is guided by a teacher-derived semantic prior rather than by conventional class distributions (Sun et al., 7 Oct 2025). Cross-modality contrastive distillation replaces direct $L_2$ matching with contrastive relational transfer, using both positive and negative correspondences in label-free settings (Lin et al., 2024). In the teacher-unavailable target setting, cross-modal knowledge can even be encoded as a learned regularizer $\mathcal{R}(\boldsymbol{\theta};\boldsymbol{\phi})$ over student parameters, with the effect of source-domain distillation transferred as a prior (Zhao et al., 2020).

4. Representative application regimes

CMKD has been applied across speech, vision, audio, retrieval, segmentation, and multimodal reasoning. The transfer direction is highly task-dependent.

Domain	Transfer direction	Representative work
Speech LLMs	text LLM $\rightarrow$ speech LLM	(Wang et al., 18 Sep 2025)
ASR and speech recognition	LM $\rightarrow$ AM; audio speech $$\mathcal{L} = \rho \mathcal{L}_{task} + (1-\rho)\mathcal{L}_{kd},$$0 visual Cued Speech	(Lee et al., 2021, Wang et al., 2021)
$$\mathcal{L} = \rho \mathcal{L}_{task} + (1-\rho)\mathcal{L}_{kd},$$1D perception	LiDAR $$\mathcal{L} = \rho \mathcal{L}_{task} + (1-\rho)\mathcal{L}_{kd},$$2 monocular image; depth $$\mathcal{L} = \rho \mathcal{L}_{task} + (1-\rho)\mathcal{L}_{kd},$$3 RGB	(Hong et al., 2022, Zhao et al., 2020)
Audio-visual understanding	audio-only $$\mathcal{L} = \rho \mathcal{L}_{task} + (1-\rho)\mathcal{L}_{kd},$$4 audio-visual student; audio $$\mathcal{L} = \rho \mathcal{L}_{task} + (1-\rho)\mathcal{L}_{kd},$$5 vision or vision $$\mathcal{L} = \rho \mathcal{L}_{task} + (1-\rho)\mathcal{L}_{kd},$$6 audio	(Wang et al., 17 Aug 2025, Xia et al., 2023)
RGBD segmentation	joint RGB/Depth training with single-modality deployment	(Ferrod et al., 30 May 2025)
Cross-modal retrieval and hashing	image–label teacher stage $$\mathcal{L} = \rho \mathcal{L}_{task} + (1-\rho)\mathcal{L}_{kd},$$7 image–text student stage	(Sun et al., 7 Oct 2025)

This coverage has widened further in work that treats CMKD as a general framework rather than a task-specific recipe. Frequency-decoupled CMKD is evaluated on audio $$\mathcal{L} = \rho \mathcal{L}_{task} + (1-\rho)\mathcal{L}_{kd},$$8 visual classification, text $$\mathcal{L} = \rho \mathcal{L}_{task} + (1-\rho)\mathcal{L}_{kd},$$9 visual classification, and RGB $\mathcal{L}_{kd}$0 Depth semantic segmentation, including CREMA-D, AVE, VGGSound, CrisisMMD, and NYU-Depth V2 (Liu et al., 25 Nov 2025). Teacher-free alternatives have also broadened the deployment model: DisCoM-KD learns one deployable classifier per modality under train/test modality mismatch, while CroDiNo-KD simultaneously trains RGB-only and Depth-only segmentation models without a multimodal teacher (Ienco et al., 2024, Ferrod et al., 30 May 2025).

5. Empirical behavior and failure modes

Empirically, CMKD can improve both robustness and task accuracy, but the gains are uneven and often concentrate in regimes where the modality gap has been explicitly modeled. In speech LLMs, the central diagnosis is that extending a text LLM into a speech LLM can induce both catastrophic forgetting and modality inequivalence. On VoiceBench, LLaMA-3.1-8B-Instruct scored 81.43 overall, while LLaMA-Omni (T$\mathcal{L}_{kd}$1T) dropped to 74.26, and LLaMA-Omni (S$\mathcal{L}_{kd}$2T) dropped further to 37.51; for the main Qwen2.5-Omni student, combining S2T KD + T2T KD improved speech-input overall VoiceBench from 75.08 to 77.19 and text-input overall VoiceBench from 78.60 to 79.86 (Wang et al., 18 Sep 2025). This is direct evidence that CMKD can simultaneously recover degraded text competence and improve spoken-query reasoning.

In monocular $\mathcal{L}_{kd}$3D detection, LiDAR-to-image CMKD shows that both feature and response distillation matter. Starting from a depth-pretrained baseline at 17.60 / 13.48 / 11.81, adding both feature and response distillation yields 23.53 / 16.33 / 14.44, and adding unlabeled paired image-LiDAR data further reaches 30.17 / 21.54 / 19.44 (Hong et al., 2022). In low-resource audio-visual SELD, the proposed framework reports 22%~36% relative gains in the overall metric over the baseline, with PointMix plus CMKD producing 0.284 on DCASE 2023 when the student alone scores 0.360 and the teacher scores 0.295 (Wang et al., 17 Aug 2025). In unconstrained videos, robust CMKD substantially improves retrieval: on UCF51, the proposed method reaches 72.9 accuracy and 91.4 / 82.3 for mAP@100 / mAP@500, compared with 66.5 and 70.7 / 60.6 for the unimodal baseline (Xia et al., 2023).

The same literature also documents sharp failure modes. The modality focusing study shows that stronger teachers are not reliably better cross-modal teachers, as in AV-MNIST and NYU Depth V2 (Xue et al., 2022). Robust CMKD for unconstrained videos attributes such failures to irrelevant modality noise and differentiated semantic correlation, arguing that temporal synchronization does not guarantee semantic consistency in the wild (Xia et al., 2023). For widely differing modalities such as face $\mathcal{L}_{kd}$4 speech and text $\mathcal{L}_{kd}$5 image, hard-constrained losses like exact $\mathcal{L}_{kd}$6 matching are reported to “easily lead to overfitting,” motivating margin-relaxed feature distillation and shared-classifier alignment instead of exact imitation (Zhao et al., 22 Jul 2025). These results suggest that CMKD is most effective when the transfer mechanism explicitly models what is shared, what is modality-specific, and how reliable a given cross-modal relation is.

6. Theory, diagnostics, and research directions

Several papers now argue that the central question in CMKD is not merely how to distill, but when and what to transfer. The Modality Focusing Hypothesis formalizes this in terms of decisive features: if $\mathcal{L}_{kd}$7 denotes the proportion of modality-general decisive features among all decisive features, then cross-modal KD is expected to perform better as $\mathcal{L}_{kd}$8 increases (Xue et al., 2022). A complementary information-theoretic criterion, the Cross-modal Complementarity Hypothesis, states that CMKD is effective when

$\mathcal{L}_{kd}$9

that is, when mutual information between teacher and student representations exceeds the mutual information between the student representation and the labels (Xie et al., 15 Oct 2025). In label-free CMKD, contrastive theory makes the modality gap explicit by bounding downstream target error in terms of the total variation distance between source and target latent distributions, implying that smaller source–target discrepancy should yield more reliable cross-modal transfer (Lin et al., 2024).

Recent method design increasingly reflects these diagnostics. Frequency-decoupled CMKD reports that low-frequency features exhibit markedly higher cross-modal similarity than high-frequency ones: on CREMA-D, original features score 0.84, low-frequency features 0.91, and high-frequency features -0.02; on AVE, the corresponding values are 0.74, 0.85, and -0.01 (Liu et al., 25 Nov 2025). This suggests that CMKD should align some representational components strongly and others only weakly. Teacher-free alternatives go further by rejecting the premise that a pretrained multimodal teacher is always the right abstraction. DisCoM-KD argues that representation disentanglement plus adversarial alignment can replace sequential teacher–student training under modality mismatch (Ienco et al., 2024), while CroDiNo-KD shows that joint collaborative training of RGB-only and Depth-only models can outperform teacher-based CMKD for RGBD segmentation while using 73M training parameters instead of teacher-dependent setups at 178M or 328M (Ferrod et al., 30 May 2025).

Open issues are recurrent across the literature. Many methods still require paired multimodal data during training (Lin et al., 2024, Ferrod et al., 30 May 2025, Xia et al., 2023). Some influential papers remain under-specified at the implementation level, such as the missing exact KL formulation over variable-length decoding in speech LLM CMKD (Wang et al., 18 Sep 2025). Source-to-target prior transfer mitigates teacher unavailability but remains sensitive to domain shift and uses a simple diagonal parameter-wise regularizer (Zhao et al., 2020). Theoretical guarantees are also narrow: Gaussian, linear, or asymptotic analyses clarify mechanism but do not yet yield a unified theory for modern nonlinear multimodal systems (Xie et al., 15 Oct 2025, Lin et al., 2024). A plausible implication is that future CMKD work will continue moving from raw imitation toward selective, diagnostically grounded transfer: deciding which teacher modality to use, which parts of the representation are transferable, and whether the teacher–student relation is informative enough to justify distillation at all.