---
title: Cross-modal Knowledge Distillation (CMKD)
url: https://www.emergentmind.com/topics/cross-modal-knowledge-distillation-cmkd-2d8e0f3e-884c-4f17-9b2b-f797f21a6e5e
type: topic
---

# Cross-modal Knowledge Distillation (CMKD)

Cross-modal Knowledge Distillation (CMKD) denotes a family of learning strategies in which knowledge is transferred between models that operate on different modalities, or under different modality-availability regimes, so that information available during training can improve a model that must ultimately operate with fewer or weaker modalities at inference. In the most common formulation, a teacher is trained on a stronger, richer, or privileged modality and supervises a student trained on a weaker or deployment-constrained modality; however, the literature also includes asymmetric unimodal-to-multimodal transfer, teacher-free collaborative training, and source-to-target transfer through learned priors rather than per-sample teacher outputs [2211.07171], [2004.00176], [2505.24361].

## 1. Definition and scope

CMKD is typically motivated by modality mismatch between training and deployment. In monocular \(3\)D object detection, LiDAR is available during training but not at inference, so a LiDAR-based detector serves as a privileged teacher for an image-only student [2211.07171]. In speech recognition, large text-only language models can supervise acoustic models, even though the teacher and student differ both in modality and, in some settings, in output units [2110.10429]. In speech large language models, a text-only LLM can supervise a speech LLM so that spoken-query behavior is aligned with text-conditioned reasoning [2509.14930]. A related but distinct setting appears when multimodal paired data are scarce: an audio-only SELD model trained on much larger audio data can teach an audio-visual student trained on limited paired audio-video data [2508.12334].

Most CMKD work assumes paired cross-modal samples during training, because teacher and student supervision must be related at the instance level. A notable extension is Cross-Modal Knowledge Generalization, which addresses the case where the target dataset contains only the weak modality and the superior modality is completely unavailable; in that setting, cross-modal knowledge learned on a source dataset is transferred as a learned prior over student parameters rather than as direct teacher supervision on target samples [2004.00176]. This suggests that CMKD is not confined to one teacher–student protocol, but is better understood as a broader mechanism for transferring task-relevant structure across heterogeneous observation spaces.

## 2. Terminology and conceptual boundaries

The acronym “CMKD” is not uniform across the literature. In the audio-classification papers “CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation” and “FastAST,” CMKD explicitly means **Cross-Model Knowledge Distillation**, not cross-modal distillation: teacher and student process the same audio spectrogram modality, but belong to different model families such as CNNs and ASTs [2203.06760], [2406.07676]. That distinction is central, because these papers address architectural heterogeneity rather than modality transfer.

Within true cross-modal usage, the phrase “cross-modal” itself has multiple concrete meanings. In speech LLMs, it refers specifically to transfer across input modalities, with a text teacher supervising a speech-conditioned student through text-to-text and speech-to-text channels [2509.14930]. In low-resource audio-visual SELD, it refers to transfer from an audio-only teacher into an audio-visual student, so the knowledge crosses from one modality regime to another rather than from multimodal teacher to unimodal student [2508.12334]. In supervised cross-modal hashing, SODA reformulates multi-label annotations as a textual modality and distills from an image–label teacher stage into an image–text student stage; there, the transferred object is a teacher-guided Hamming space rather than conventional soft class logits [2510.09664]. These cases show that CMKD is defined less by a single architecture than by a transfer relation across heterogeneous modalities or modality sets.

A recurrent misconception is that a stronger or multimodal teacher should automatically be a better cross-modal teacher. Case studies on AV-MNIST and NYU Depth V2 show the opposite can occur: on AV-MNIST, a multimodal teacher with **91.61%** accuracy yielded a student at **69.73%**, whereas a weaker unimodal teacher at **84.57%** yielded **70.10%**; on NYU Depth V2, a multimodal teacher at **51.00 mIoU** did not outperform unimodal KD for the RGB student [2206.06487]. This suggests that teacher accuracy and transferability are distinct quantities.

## 3. Canonical transfer mechanisms

A standard CMKD formulation combines a task loss with a distillation loss,
\[
\mathcal{L} = \rho \mathcal{L}_{task} + (1-\rho)\mathcal{L}_{kd},
\]
with \(\mathcal{L}_{kd}\) often instantiated as a KL divergence between teacher and student outputs [2206.06487]. Many systems remain close to this template but differ in what is distilled, how modalities are aligned, and whether the teacher supervises outputs, features, relations, or learned priors.

At the response level, CMKD often uses softened class or token distributions. Automatic Cued Speech recognition distills from a DeepSpeech2 acoustic teacher into a visual Cued Speech student using frame-level KL divergence and sequence-level cosine similarity on logits, with additional uncertainty-based multi-task weighting for balancing teacher and label supervision [2106.13686]. Language-model-to-acoustic-model distillation in ASR uses a multi-task formulation with separate supervised and KD heads, precisely to avoid the label-interpolation limitations of earlier LM-to-ASR distillation [2110.10429]. Speech LLM CMKD likewise uses response-level hard targets and KL supervision, but distinguishes a text-to-text channel for preserving textual competence from a speech-to-text channel for aligning speech-conditioned reasoning to a text teacher [2509.14930].

Feature-level CMKD is equally prominent. Monocular \(3\)D detection distills LiDAR BEV features into an image-based detector through
\[
\mathcal{L}_{feat} = MSE(\hat{F}^{bev}_{I},F^{bev}_{L}),
\]
and supplements this with confidence-weighted response distillation over \(3\)D boxes and classes [2211.07171]. Audio-visual SELD combines response KD with multi-stage, multi-scale feature KD, aligning intermediate representations through hierarchical context loss in addition to output matching [2508.12334]. A dimensional-structure approach for optical-to-radar transfer enforces output-channel decorrelation and intermediate-feature uniformity, interpreting CMKD as transfer of representational structure rather than only semantic outputs [2306.15977]. A more recent frequency-decoupled framework separates low- and high-frequency feature components, applying strong MSE alignment to low-frequency features and relaxed LogMSE alignment to high-frequency ones, on the grounds that low-frequency structure is more cross-modally consistent [2511.19887].

Other works distill relations rather than raw outputs or features. SODA transfers semantic affinity structure and a unified Hamming/code space from an image–label teacher stage into image–text hashing, so that the student is guided by a teacher-derived semantic prior rather than by conventional class distributions [2510.09664]. Cross-modality contrastive distillation replaces direct \(L_2\) matching with contrastive relational transfer, using both positive and negative correspondences in label-free settings [2405.03355]. In the teacher-unavailable target setting, cross-modal knowledge can even be encoded as a learned regularizer \(\mathcal{R}(\boldsymbol{\theta};\boldsymbol{\phi})\) over student parameters, with the effect of source-domain distillation transferred as a prior [2004.00176].

## 4. Representative application regimes

CMKD has been applied across speech, vision, audio, retrieval, segmentation, and multimodal reasoning. The transfer direction is highly task-dependent.

| Domain | Transfer direction | Representative work |
|---|---|---|
| Speech LLMs | text LLM \(\rightarrow\) speech LLM | [2509.14930] |
| ASR and speech recognition | LM \(\rightarrow\) AM; audio speech \(\rightarrow\) visual Cued Speech | [2110.10429], [2106.13686] |
| \(3\)D perception | LiDAR \(\rightarrow\) monocular image; depth \(\rightarrow\) RGB | [2211.07171], [2004.00176] |
| Audio-visual understanding | audio-only \(\rightarrow\) audio-visual student; audio \(\rightarrow\) vision or vision \(\rightarrow\) audio | [2508.12334], [2304.07775] |
| RGBD segmentation | joint RGB/Depth training with single-modality deployment | [2505.24361] |
| Cross-modal retrieval and hashing | image–label teacher stage \(\rightarrow\) image–text student stage | [2510.09664] |

This coverage has widened further in work that treats CMKD as a general framework rather than a task-specific recipe. Frequency-decoupled CMKD is evaluated on audio \(\leftrightarrow\) visual classification, text \(\leftrightarrow\) visual classification, and RGB \(\leftrightarrow\) Depth semantic segmentation, including CREMA-D, AVE, VGGSound, CrisisMMD, and NYU-Depth V2 [2511.19887]. Teacher-free alternatives have also broadened the deployment model: DisCoM-KD learns one deployable classifier per modality under train/test modality mismatch, while CroDiNo-KD simultaneously trains RGB-only and Depth-only segmentation models without a multimodal teacher [2408.07080], [2505.24361].

## 5. Empirical behavior and failure modes

Empirically, CMKD can improve both robustness and task accuracy, but the gains are uneven and often concentrate in regimes where the modality gap has been explicitly modeled. In speech LLMs, the central diagnosis is that extending a text LLM into a speech LLM can induce both **catastrophic forgetting** and **modality inequivalence**. On VoiceBench, **LLaMA-3.1-8B-Instruct** scored **81.43 overall**, while **LLaMA-Omni (T\(\to\)T)** dropped to **74.26**, and **LLaMA-Omni (S\(\to\)T)** dropped further to **37.51**; for the main Qwen2.5-Omni student, combining **S2T KD + T2T KD** improved speech-input overall VoiceBench from **75.08** to **77.19** and text-input overall VoiceBench from **78.60** to **79.86** [2509.14930]. This is direct evidence that CMKD can simultaneously recover degraded text competence and improve spoken-query reasoning.

In monocular \(3\)D detection, LiDAR-to-image CMKD shows that both feature and response distillation matter. Starting from a depth-pretrained baseline at **17.60 / 13.48 / 11.81**, adding both feature and response distillation yields **23.53 / 16.33 / 14.44**, and adding unlabeled paired image-LiDAR data further reaches **30.17 / 21.54 / 19.44** [2211.07171]. In low-resource audio-visual SELD, the proposed framework reports **22%~36% relative gains** in the overall metric over the baseline, with PointMix plus CMKD producing **0.284** on DCASE 2023 when the student alone scores **0.360** and the teacher scores **0.295** [2508.12334]. In unconstrained videos, robust CMKD substantially improves retrieval: on UCF51, the proposed method reaches **72.9** accuracy and **91.4 / 82.3** for mAP@100 / mAP@500, compared with **66.5** and **70.7 / 60.6** for the unimodal baseline [2304.07775].

The same literature also documents sharp failure modes. The modality focusing study shows that stronger teachers are not reliably better cross-modal teachers, as in AV-MNIST and NYU Depth V2 [2206.06487]. Robust CMKD for unconstrained videos attributes such failures to **irrelevant modality noise** and **differentiated semantic correlation**, arguing that temporal synchronization does not guarantee semantic consistency in the wild [2304.07775]. For widely differing modalities such as face \(\rightarrow\) speech and text \(\rightarrow\) image, hard-constrained losses like exact \(l_2\) matching are reported to “easily lead to overfitting,” motivating margin-relaxed feature distillation and shared-classifier alignment instead of exact imitation [2507.16296]. These results suggest that CMKD is most effective when the transfer mechanism explicitly models what is shared, what is modality-specific, and how reliable a given cross-modal relation is.

## 6. Theory, diagnostics, and research directions

Several papers now argue that the central question in CMKD is not merely **how** to distill, but **when** and **what** to transfer. The Modality Focusing Hypothesis formalizes this in terms of decisive features: if \(\gamma\) denotes the proportion of modality-general decisive features among all decisive features, then cross-modal KD is expected to perform better as \(\gamma\) increases [2206.06487]. A complementary information-theoretic criterion, the **Cross-modal Complementarity Hypothesis**, states that CMKD is effective when
\[
I(H_1;H_2) > I(H_2;Y),
\]
that is, when mutual information between teacher and student representations exceeds the mutual information between the student representation and the labels [2510.13182]. In label-free CMKD, contrastive theory makes the modality gap explicit by bounding downstream target error in terms of the total variation distance between source and target latent distributions, implying that smaller source–target discrepancy should yield more reliable cross-modal transfer [2405.03355].

Recent method design increasingly reflects these diagnostics. Frequency-decoupled CMKD reports that low-frequency features exhibit markedly higher cross-modal similarity than high-frequency ones: on CREMA-D, original features score **0.84**, low-frequency features **0.91**, and high-frequency features **-0.02**; on AVE, the corresponding values are **0.74**, **0.85**, and **-0.01** [2511.19887]. This suggests that CMKD should align some representational components strongly and others only weakly. Teacher-free alternatives go further by rejecting the premise that a pretrained multimodal teacher is always the right abstraction. DisCoM-KD argues that representation disentanglement plus adversarial alignment can replace sequential teacher–student training under modality mismatch [2408.07080], while CroDiNo-KD shows that joint collaborative training of RGB-only and Depth-only models can outperform teacher-based CMKD for RGBD segmentation while using **73M** training parameters instead of teacher-dependent setups at **178M** or **328M** [2505.24361].

Open issues are recurrent across the literature. Many methods still require paired multimodal data during training [2405.03355], [2505.24361], [2304.07775]. Some influential papers remain under-specified at the implementation level, such as the missing exact KL formulation over variable-length decoding in speech LLM CMKD [2509.14930]. Source-to-target prior transfer mitigates teacher unavailability but remains sensitive to domain shift and uses a simple diagonal parameter-wise regularizer [2004.00176]. Theoretical guarantees are also narrow: Gaussian, linear, or asymptotic analyses clarify mechanism but do not yet yield a unified theory for modern nonlinear multimodal systems [2510.13182], [2405.03355]. A plausible implication is that future CMKD work will continue moving from raw imitation toward selective, diagnostically grounded transfer: deciding which teacher modality to use, which parts of the representation are transferable, and whether the teacher–student relation is informative enough to justify distillation at all.

Source: https://www.emergentmind.com/topics/cross-modal-knowledge-distillation-cmkd-2d8e0f3e-884c-4f17-9b2b-f797f21a6e5e