Cross-Modal Knowledge Distillation

• Cross-modal knowledge distillation is a paradigm where teacher and student models operate on different modalities, transferring rich, complementary features.
• It employs techniques like logit matching and latent distribution alignment to overcome modality mismatches and enhance model performance.
• This approach has practical applications in video action recognition, remote sensing, and biomedical imaging, yielding efficiency gains and robust results.

Cross-modal knowledge distillation (CMKD) is a teacher–student transfer paradigm in which the teacher and student operate on different modalities, such as audio and video, RGB and depth, images and text, spikes and local field potentials, or text and speech. Its usual purpose is to exploit richer, complementary, or training-only modalities during optimization while preserving a cheaper, smaller, or unimodal model at inference. In contrast to standard knowledge distillation, CMKD must handle modality mismatch in input statistics, semantic overlap, representation geometry, and task structure; consequently, recent formulations treat distillation not only as logit matching but also as alignment of latent distributions, relational structure, temporal context, modality-general decisive features, or task-specific manifolds (Xue et al., 2022, Tran et al., 9 Jun 2026).

1. Formal problem settings

A canonical starting point is the standard distillation objective

$$\mathcal L = \rho \mathcal{L}_{task} + (1-\rho)\mathcal{L}_{kd},$$

with $\mathcal{L}_{kd}$ typically defined by a KL divergence between teacher and student predictions. In CMKD, however, the distillation term compares $\mathbf{f}_{\boldsymbol{\theta}_s}(\mathbf{x}^b)$ and $\mathbf{f}_{\boldsymbol{\theta}_t}(\mathbf{x}^a)$, where teacher and student consume different modalities; in the multimodal-teacher variant, the teacher may use $(\mathbf{x}^a,\mathbf{x}^b)$. This modality mismatch is precisely why a better teacher for its own modality is not necessarily a better teacher for another modality (Xue et al., 2022).

The earliest CMKD settings in this literature commonly assume paired multimodal data. In action recognition, for example, a teacher trained on labeled RGB videos can supervise a student trained on unlabeled sequences of $3$D skeleton poses, using synchronized RGB–pose pairs as supervision rather than annotations on the target modality. In that formulation, the teacher produces $\mathrm{P_T}$, the student predicts $\mathrm{P_S}$, and transfer can be driven either by softened distributions or by the teacher’s hard decision $\hat c_T = \arg\max_c \mathrm{P_T}(c)$ (Thoker et al., 2019).

More recent work generalizes CMKD to substantially weaker correspondence assumptions. UCMKD formulates teacher and student as $M_T = (\phi, p_T(y\mid z=\phi(x^T)))$ and $\mathcal{L}_{kd}$0, and derives a cross-modal distributional relationship

$\mathcal{L}_{kd}$1

where feature alignment (FA) measures representation-level semantic discrepancy and label alignment (LA) measures prediction-level semantic discrepancy. This replaces pairwise alignment with distributional alignment and permits distillation when paired samples are unavailable (Tran et al., 9 Jun 2026).

CMKD has also been reformulated beyond the conventional teacher/student pipeline. DisCoM-KD addresses overlapping and non-overlapping modality mismatch by jointly learning modality-specific single-modal classifiers from multimodal data and explicitly decomposing each modality into modality-invariant, modality-informative, and modality-irrelevant features, thereby eliminating the need to learn each student separately as well as the teacher classifier (Ienco et al., 2024).

2. Distillation objectives and architectural motifs

Because modality shift invalidates naïve imitation, CMKD methods differ primarily in what they transfer and how they align it.

Work	Setting	Distinctive mechanism
(Thoker et al., 2019)	RGB $\mathcal{L}_{kd}$2 $\mathcal{L}_{kd}$3D pose action recognition	hard-label cross-entropy and mutual learning
(Dai et al., 2021)	action detection in untrimmed videos	atomic, global-context, and boundary distillation
(Sarkar et al., 2022)	self-supervised audio–video learning	masked reconstruction, MMD alignment, bidirectional KD
(Wang et al., 2023)	visual place recognition	self-agent and cross-agent multi-manifold relations
(Yan et al., 21 Aug 2025)	online HD map construction	Teacher-Coach-Student, TGPD, MSRD

One influential design line uses response-based distillation but modifies the supervision signal. In RGB-to-pose action recognition, cross-entropy on the teacher’s hard prediction,

$\mathcal{L}_{kd}$4

outperformed KL-based distillation, and mutual learning among a small ensemble of student networks further improved robustness. On NTU RGB+D, the method nearly matched fully supervised target-modality training: ST-GCN reached $\mathcal{L}_{kd}$5 versus $\mathcal{L}_{kd}$6, and HCN reached $\mathcal{L}_{kd}$7 versus $\mathcal{L}_{kd}$8 (Thoker et al., 2019).

A second line transfers structured intermediate information rather than logits alone. For action detection in untrimmed videos, “Augmented-RGB” combines atomic-level contrastive distillation with sequence-level transfer of Global Contextual Relation and Boundary Saliency. The overall objective is

$\mathcal{L}_{kd}$9

where the sequence terms are designed specifically for temporal localization rather than trimmed-video classification. This formulation reflects the claim that action detection requires temporal structure transfer, not only holistic representation transfer (Dai et al., 2021).

A third line emphasizes relation-level transfer. XKD augments masked audio–visual reconstruction with domain alignment via Maximum Mean Discrepancy and symmetric cross-modal KD between audio and video teachers and students. DistilVPR replaces pointwise feature imitation by preservation of feature relationships, using self-agent and cross-agent relations over Euclidean, spherical, and hyperbolic manifolds. MapKD inserts an intermediate coach into a Teacher-Coach-Student pipeline, then distills both BEV feature structure through Token-Guided 2D Patch Distillation and masked semantic outputs through Masked Semantic Response Distillation (Sarkar et al., 2022, Wang et al., 2023, Yan et al., 21 Aug 2025).

3. Alignment, disentanglement, and what makes transfer effective

A central theoretical and empirical question in CMKD is why some teachers transfer and others do not. The Modality Focusing Hypothesis argues that distillation performance depends on the proportion of modality-general decisive features preserved in the teacher network: with larger $\mathbf{f}_{\boldsymbol{\theta}_s}(\mathbf{x}^b)$0, the student is expected to perform better. The accompanying modality Venn diagram decomposes decisive information into modality-general and modality-specific parts, and the paper’s AV-MNIST and NYU Depth V2 case studies show that higher teacher accuracy can coincide with worse student performance. In this view, transferable competence rather than predictive competence is the relevant criterion for teacher quality (Xue et al., 2022).

Several methods operationalize this idea by separating transferable from non-transferable structure. The dimensional-structure CMKD method ties discriminability to low channel-wise correlation and more uniform intermediate distributions. It implements these principles through the Semantic Enhancement Module,

$\mathbf{f}_{\boldsymbol{\theta}_s}(\mathbf{x}^b)$1

and the Distribution Calibration Module, which encourages uniformly spread intermediate features and cross-modal distribution alignment. The stated purpose is to transfer how knowledge is organized rather than merely matching final predictions (Si et al., 2023).

Frequency-decoupled CMKD makes a closely related claim in the frequency domain: low-frequency features exhibit high cross-modal consistency, whereas high-frequency features have extremely low cross-modal similarity. It therefore applies strong MSE alignment to low-frequency features, relaxed LogMSE alignment to high-frequency features, and adds feature standardization plus a shared classifier to reduce scale and decision-space mismatch. The resulting objective

$\mathbf{f}_{\boldsymbol{\theta}_s}(\mathbf{x}^b)$2

is explicitly asymmetric across feature components (Liu et al., 25 Nov 2025).

DisCoM-KD extends disentanglement more directly. For each modality, it learns $\mathbf{f}_{\boldsymbol{\theta}_s}(\mathbf{x}^b)$3, $\mathbf{f}_{\boldsymbol{\theta}_s}(\mathbf{x}^b)$4, and $\mathbf{f}_{\boldsymbol{\theta}_s}(\mathbf{x}^b)$5, combines adversarial modality confusion with auxiliary modality classifiers and orthogonality penalties, and feeds only the invariant and informative representations to the downstream classifier. This treats CMKD as a joint disentanglement-and-adaptation problem rather than pure teacher imitation (Ienco et al., 2024).

A complementary diagnosis appears in work on widely differing modalities. There, hard constrained loss such as plain $\mathbf{f}_{\boldsymbol{\theta}_s}(\mathbf{x}^b)$6 matching is said to overfit because it forces the student to become exactly the same as the teacher even when the modalities have unique information. The proposed remedy is a margin-based soft feature constraint, a shared classifier space, and quality-based adaptive weighting based on feature norms (Zhao et al., 22 Jul 2025).

4. Supervision regimes, noise, and robustness

CMKD performance is strongly affected by correspondence quality, modality noise, and missing information. In unconstrained audio–visual videos, temporal synchronization alone is not considered a sufficient guarantee of semantic consistency. The Modality Noise Filter uses cross-modal context to erase irrelevant teacher noise, and the Contrastive Semantic Calibration module then distills purified knowledge adaptively according to sample-wise semantic correlation. The distillation loss

$\mathbf{f}_{\boldsymbol{\theta}_s}(\mathbf{x}^b)$7

combines purification, guided supervision, and calibrated contrastive learning (Xia et al., 2023).

Missing-modality settings introduce another robustness problem: not all modalities contribute equally to every task. LCKD therefore uses a teacher election procedure that selects the best modality for each task based on validation Dice score. On BraTS2018, T1c is especially informative for enhancing tumor and tumor core, while Flair is especially informative for whole tumor. Distillation is then applied from the elected teachers to the other modalities, allowing a single non-dedicated model to handle all missing-modality combinations (Wang et al., 2023).

Text-augmented teachers reveal a different failure mode: label leakage. In supervised image classification, exact class-name prompts can make the multimodal teacher artificially strong while degrading transfer to the image-only student. WordNet-relaxed text embeddings are introduced precisely to avoid direct use of exact class names, and interpretability analyses indicate that these prompts encourage heavier reliance on visual features over textual shortcuts while still incorporating the textual cues (Guo et al., 31 Mar 2025).

UCMKD pushes robustness further by removing the paired-data assumption entirely. Its bi-level optimization uses a Wasserstein feature-alignment inner step solved by Sinkhorn iterations and a label-alignment term gated by teacher confidence. This is a distributional rather than samplewise response to missing correspondence (Tran et al., 9 Jun 2026).

5. Representative application domains

Video understanding has been a major testbed. In action recognition, RGB-to-pose CMKD nearly achieved target-modality full supervision without target labels (Thoker et al., 2019). In untrimmed action detection, Augmented-RGB outperformed other popular cross-modal distillation methods and approached or exceeded two-stream performance while using only RGB at inference; on Charades it reached $\mathbf{f}_{\boldsymbol{\theta}_s}(\mathbf{x}^b)$8 mAP versus $\mathbf{f}_{\boldsymbol{\theta}_s}(\mathbf{x}^b)$9 for the two-stream baseline, and on PKU-MMD it reached $\mathbf{f}_{\boldsymbol{\theta}_t}(\mathbf{x}^a)$0 versus $\mathbf{f}_{\boldsymbol{\theta}_t}(\mathbf{x}^a)$1 in the reported setting (Dai et al., 2021). In visual place recognition, DistilVPR transferred multi-modal image–point-cloud relational structure into single-modal students and reported state-of-the-art performance against distillation baselines (Wang et al., 2023).

Speech and language applications demonstrate that CMKD is not confined to vision. In automatic cued speech recognition, audio-to-visual distillation combines frame-level KL/CE transfer, uncertainty-based multi-task weighting, and sequence-level cosine/CTC alignment; the method reported superior CS recognition performance to the state of the art by a large margin on French and British English datasets (Wang et al., 2021). In speech LLMs, text-to-text and speech-to-text distillation are used jointly to address catastrophic forgetting and modality inequivalence, improving VoiceBench in both textual and spoken modes (Wang et al., 18 Sep 2025).

Autonomous systems and remote sensing use CMKD to replace expensive sensors or priors at deployment. The dimensional-structure CMKD method transfers optical knowledge to radar and reported strong gains on a real-world optical–radar dataset and multiple benchmarks (Si et al., 2023). MapKD transfers knowledge from camera–LiDAR fusion with SD/HD priors into a camera-only student and improves the student by $\mathbf{f}_{\boldsymbol{\theta}_t}(\mathbf{x}^a)$2 mIoU and $\mathbf{f}_{\boldsymbol{\theta}_t}(\mathbf{x}^a)$3 mAP on nuScenes while accelerating inference speed (Yan et al., 21 Aug 2025).

Biomedical and scientific modeling provide additional evidence that CMKD can serve as a modality-compensation mechanism. In BraTS2018 brain-tumor segmentation, LCKD improved the state of the art by $\mathbf{f}_{\boldsymbol{\theta}_t}(\mathbf{x}^a)$4 for enhancing tumor, $\mathbf{f}_{\boldsymbol{\theta}_t}(\mathbf{x}^a)$5 for tumor core, and $\mathbf{f}_{\boldsymbol{\theta}_t}(\mathbf{x}^a)$6 for whole tumor in Dice score (Wang et al., 2023). In intracortical neural modeling, pretrained multi-session spike transformers are used as teachers for LFP transformers; distilled LFP models achieved an average $\mathbf{f}_{\boldsymbol{\theta}_t}(\mathbf{x}^a)$7 of $\mathbf{f}_{\boldsymbol{\theta}_t}(\mathbf{x}^a)$8 versus $\mathbf{f}_{\boldsymbol{\theta}_t}(\mathbf{x}^a)$9 for MS-LFP in the reported unsupervised setting and generalized to other sessions without additional distillation (Erturk et al., 13 Dec 2025).

6. Limitations, controversies, and emerging directions

A recurring conclusion is that CMKD is not a universal cure. Teacher accuracy and student accuracy are not positively correlated in general, multimodal teachers are not always optimal teachers for single-modal students, and some tasks are harmed by direct logit transfer when the teacher relies on inaccessible modality-specific cues or is weaker than the student on the target dataset (Xue et al., 2022, Dai et al., 2021, Ienco et al., 2024).

Another open issue is how to choose and adapt teachers. MST-Distill explicitly identifies distillation path selection and knowledge drift as the two critical problems. Its solution is a mixture of specialized teachers, an instance-level routing network, and a MaskNet module that suppresses modality-specific discrepancies and reconstructs teacher representations; the paper also acknowledges higher training time and memory use, and states that the benefit is strongest when modalities are sufficiently alignable (Li et al., 9 Jul 2025).

Recent work also suggests that CMKD is expanding beyond white-box, paired, fixed-teacher assumptions. ARMADA transfers knowledge from vision-language teachers, including black-box teachers, to language-only students via a task-specific TS Aligner, output alignment, and manifold alignment. It reports up to $(\mathbf{x}^a,\mathbf{x}^b)$0 improvement on language understanding tasks and a $(\mathbf{x}^a,\mathbf{x}^b)$1 boost in generative reasoning, while also explicitly noting that cross-modal transfer can import biases from vision models into LLMs, leaving fairness and interpretability as open issues (Sengupta et al., 11 Mar 2026).

Taken together, these results suggest several stable research directions: training teachers to rely more on modality-general decisive features, explicitly separating modality-general and modality-specific information before transfer, replacing pairwise matching with transport-based distribution alignment when pairs are unavailable, and learning task-specific bridges such as coaches, aligners, or adaptive frequency partitions rather than imposing a single universal imitation loss (Xue et al., 2022, Tran et al., 9 Jun 2026, Liu et al., 25 Nov 2025, Yan et al., 21 Aug 2025).