Modality-Adaptive Knowledge Distillation

• The paper introduces modality-adaptive KD as a strategy that conditions the transfer signal on modality-specific cues, enabling efficient learning across differing modality inputs.
• It leverages diverse distillation targets—such as modality-specific predictions, cross-modality relations, and class prototypes—combined with adaptive weighting and calibration to address modality mismatches.
• Empirical results demonstrate that adaptive methods improve performance especially when modality gaps are large, enhancing robustness in multimodal and cross-modal applications.

Modality-adaptive knowledge distillation denotes a family of distillation strategies for multimodal, cross-modal, and missing-modality learning in which the transfer signal is conditioned on modality structure rather than treated as a uniform teacher-to-student imitation problem. In this literature, adaptation may occur at the level of modality-specific predictions, cross-modality relations, class prototypes, teacher selection, sample quality, module contribution, or optimization balance. The common premise is that standard output-level KD, typically expressed as $$\mathcal L=\rho\mathcal L_{task}+(1-\rho)\mathcal L_{kd}$$, is often too coarse when teacher and student observe different modalities, observe different subsets of modalities, or rely on modality contributions that vary by sample and task (Xue et al., 2022, Jin et al., 2021, Liu, 2021, Wang et al., 2024, Li et al., 9 Jul 2025, Liang et al., 2023).

1. Problem formulation and modality mismatch

The central problem is a mismatch between the information available to the teacher during training and the information available to the student during training or deployment. Several recurrent settings appear in the literature. In multimodal classification, both teacher and student may be multimodal, but the student is expected to reproduce not only the teacher’s final prediction on joint inputs but also its behavior on text-only, image-only, and image+text conditions (Liu, 2021, Jin et al., 2021). In cross-modal adaptation, the teacher and student can operate on fundamentally different modalities, such as RGB videos to 3D skeleton poses for action recognition, face images to speech for speaker recognition, or text to images for image classification (Thoker et al., 2019, Zhao et al., 22 Jul 2025). In missing-modality segmentation, the teacher is trained with full multi-modal MRI while the student must segment using only a single modality or an incomplete subset at inference time (Wang et al., 2023, Wang et al., 2024, Zhu et al., 18 Sep 2025). A privileged-information variant arises when a modality is available in training but absent at test time, as in Visual Question Answer-Difference prediction where answers are available only during training (Cho et al., 2021).

This mismatch is not only architectural. It is also semantic. In unconstrained videos, audio and vision are not uniformly aligned: teacher audio may contain background music or other irrelevant signals, and the semantic correlation between audio and video varies sample by sample (Xia et al., 2023). In brain-eye-computer learning, EEG, image, and fusion branches learn at different rates and may dominate gradients asymmetrically (Li et al., 2024). In large multimodal foundation models such as CoCa, image, text, and multimodal modules do not contribute equally at all stages of student training (Liang et al., 2023).

A major conceptual correction to earlier intuition is that crossmodal KD is not determined primarily by teacher accuracy. The “modality Venn diagram” and “modality focusing hypothesis” formalize the decisive issue as the proportion of modality-general decisive features preserved in the teacher. The paper states: “For crossmodal KD, distillation performance depends on the proportion of modality-general decisive features preserved in the teacher network: with larger $\gamma$, the student network is expected to perform better” (Xue et al., 2022). This directly explains failure cases in which a stronger multimodal teacher yields a worse student, including AV-MNIST and NYU Depth V2 (Xue et al., 2022).

2. Distillation targets beyond logits

A defining property of modality-adaptive KD is the enlargement of the distillation target beyond teacher logits. One line of work distills modality-specific behavior. “MSD: Saliency-aware Knowledge Distillation for Multimodal Understanding” adds auxiliary KD losses for text-only and image-only inputs on top of conventional multimodal KD, and then weights those losses according to saliency or learned importance (Jin et al., 2021). This preserves the teacher’s behavior under modality ablation rather than only on the joint input.

A second line distills modality relations. “Multi-Modality Distillation via Learning the teacher’s modality-level Gram Matrix” constructs a modality representation matrix $A$ from image, text, and image+text outputs and defines the modality relation matrix as

$G = A \cdot A^{T}.$

The student is trained to reproduce this modality-level Gram matrix, so that it learns the teacher’s modality interaction structure rather than only its answer distribution (Liu, 2021).

A third line distills class geometry. “Prototype Knowledge Distillation for Medical Segmentation with Missing Modality” computes a class prototype

$c_k = \frac{\sum_{i}z_i \mathbbm{1}[y_i=k] }{\sum_{i}\mathbbm{1}[y_i=k]},$

then forms class-wise cosine-similarity maps

$$M_k(i) = \frac{z_i^T c_k}{\Vert z_i \Vert \Vert c_k \Vert}.$$

The resulting $I^2FV$ maps encode both intra-class compactness and inter-class separation, and the student matches the teacher’s maps through $\mathcal L_{proto}$ in addition to pixel-wise KL distillation (Wang et al., 2023).

Other methods target feature structure or decision regions rather than explicit modality relations. Meta-learned Cross-modal Knowledge Distillation uses pairwise feature alignment weighted by learned modality importance ratios (Wang et al., 2024). AdaMM adds bottleneck distillation through global style matching and adversarial feature alignment, together with a lesion-presence reliability prior (Zhu et al., 18 Sep 2025). DisCoM-KD abandons direct teacher-output imitation and instead decomposes each modality into modality-invariant, modality-informative, and modality-irrelevant features, using $z^{inv}$ and $z^{inf}$ for classification and discarding $\gamma$0 (Ienco et al., 2024). In sensor-to-vision distillation, SAKDN transfers graph-guided semantically discriminative maps rather than raw intermediate activations, after converting 1D wearable signals into GAF-based virtual images (Liu et al., 2020).

The choice of target is closely tied to modality gap. When modalities are widely differing, direct $\gamma$1 equality between teacher and student features can be over-constraining. “Cross-Modal Distillation For Widely Differing Modalities” therefore replaces rigid feature imitation with a margin-based soft constraint,

$\gamma$2

and complements it with classifier-level sharing in a common decision space (Zhao et al., 22 Jul 2025).

3. Adaptive weighting, routing, and calibration mechanisms

The “adaptive” qualifier in this literature usually refers to the mechanism that decides how much, from whom, and at what granularity distillation should occur. In MSD, modality saliency is computed by erasing a modality and measuring the change in the teacher prediction, $\gamma$3. The resulting text-only and image-only KD losses can be weighted either by hand-designed saliency rules or by a meta-learned MLP trained on validation data (Jin et al., 2021).

Meta-learned Cross-modal Knowledge Distillation generalizes this logic to missing-modality learning by learning a modality importance vector $\gamma$4 with bi-level optimization. The normalized weights are obtained by softmax, and the distillation term is

$\gamma$5

so more informative modalities act more strongly as teachers for less informative ones (Wang et al., 2024).

Sample adaptivity appears in several forms. In unconstrained video distillation, the Modality Noise Filter purifies the teacher modality with cross-modal context, and Contrastive Semantic Calibration uses a pair-specific temperature,

$\gamma$6

so the contrastive transfer strength reflects sample-wise semantic correlation after purification (Xia et al., 2023). In widely differing modalities, a quality-based adaptive weight module redistributes sample weights using an $\gamma$7 feature norm quality proxy, downweighting unreliable teacher or student samples (Zhao et al., 22 Jul 2025).

Teacher adaptivity can be elevated to dynamic routing. MST-Distill constructs a mixture of specialized teachers from multimodal and cross-modal sources, inserts MaskNet into selected layers to suppress modality-specific discrepancies and reconstruct teacher representations, and then uses GateNet with Top-$\gamma$8 selection to choose the most transferable teachers per instance (Li et al., 9 Jul 2025). This addresses what the paper identifies as “distillation path selection” and “knowledge drift” (Li et al., 9 Jul 2025).

Adaptation may also operate at the level of optimization dynamics. AMBOKD treats visual, EEG, and fusion branches as rotating teachers and students in online KD, but further modulates the interaction using adaptive $\gamma$9 weights and a dynamic gradient ratio $A$0 to balance modalities that learn at different speeds (Li et al., 2024). In multimodality foundation models, OPTIMA formulates module subset selection as a nonstationary multi-armed bandit and chooses which of the image, text, and multimodal modules should be distilled for the next $A$1 steps based on recent loss decrements (Liang et al., 2023). This is module-wise rather than sample-wise adaptation, but it serves the same function: avoiding uniform distillation pressure when contributions are unequal.

4. Architectural patterns

The earliest and most common architecture is the frozen full-modality teacher with a restricted student. ProtoKD trains a VNet teacher on four MRI modalities and then freezes it while a single-modality VNet student is optimized with segmentation loss, KL distillation, and prototype KD (Wang et al., 2023). MetaKD follows the same full-teacher/incomplete-student logic, with missing features approximated by the mean of available modality features at inference (Wang et al., 2024). AdaMM also uses a full-modality teacher, but inserts an Adapter Bank, a Graph-guided Adaptive Refinement Module, a Bi-Bottleneck Distillation Module, and a Lesion-Presence-Guided Reliability Module into the student path (Zhu et al., 18 Sep 2025).

A distinct pattern is online mutual distillation. In cross-modality medical segmentation, the framework integrates two segmentors, one on synthetic target-style images and one on real target images, with two-way online KD losses and an Image Alignment Module based on adversarial translation (Li et al., 2020). In cross-modal action recognition, the student side is a small ensemble trained without labels on paired RGB–skeleton sequences; each student receives teacher cross-entropy supervision plus mutual KL learning from peer students (Thoker et al., 2019). AMBOKD extends online mutual learning to three modalities—visual, EEG, and fusion—trained end to end (Li et al., 2024).

Multi-teacher systems recur when transfer is asymmetric or modality-specific expertise is valuable. In Visual Question Answer-Difference prediction, the best student result is obtained not by a single “Big” privileged teacher alone, but by combining the Big, Visual, and Question teachers with intermediate feature distillation (Cho et al., 2021). SAKDN similarly uses multiple wearable-sensor teachers and fuses them through the Similarity-Preserving Adaptive Multi-modal Fusion Module before distilling to the RGB student (Liu et al., 2020). MST-Distill formalizes teacher plurality as an explicit ensemble and routing problem (Li et al., 9 Jul 2025).

Finally, some methods depart from the teacher-student paradigm entirely. DisCoM-KD jointly learns all single-modal classifiers from multi-modal training data and explicitly separates transferable invariant content from modality-informative and modality-irrelevant content, eliminating the need to train a teacher classifier and then train each student separately (Ienco et al., 2024). This suggests a broader interpretation of modality-adaptive KD: knowledge transfer can be implemented through representation organization rather than explicit imitation.

5. Empirical patterns across tasks

Across multimodal classification benchmarks, relational and modality-specific objectives generally outperform plain KD. Modality-level Gram-matrix distillation improves over conventional KD on visual entailment, NLVR, and Hateful Memes: VE rises from 71.22 test / 71.43 val to 72.45 / 72.66; NLVR from 73.62 / 73.45 to 75.33 / 75.06; and Hateful Memes from 68.22 / 67.89 to 69.54 / 69.85 (Liu, 2021). MSD with weight learning improves Hateful-Memes from $A$2 ACC / $A$3 AUC under KD to $A$4 / $A$5, MM-IMDB from $A$6 Macro F1 / $A$7 Micro F1 to $A$8 / $A$9, and SNLI-VE from $G = A \cdot A^{T}.$0 to $G = A \cdot A^{T}.$1 (Jin et al., 2021).

In cross-modal action recognition, replacing KL with teacher-argmax cross-entropy and adding mutual learning nearly closes the gap to full supervision. For ST-GCN, the paper reports 71.17% for KL, one student; 74.91% for cross-entropy, one student; and 77.83% for cross-entropy plus mutual learning with two students, versus 78.50% under full supervision (Thoker et al., 2019). This result is especially important because the supervision comes from paired but unlabeled RGB–skeleton sequences rather than target-side annotations.

In missing-modality medical segmentation, structural targets consistently help. ProtoKD achieves the best average Dice Score for all single-modality settings on BraTS 2018, with 63.07 on T1, 65.82 on T2, 77.39 on T1ce, and 64.17 on Flair, while the T1 ablation shows a rise from 58.77 with only $G = A \cdot A^{T}.$2 to 63.07 with both pixel-wise KD and prototype KD (Wang et al., 2023). MetaKD is best on all 15 combinations for WT, 12/15 ET settings, and 13/15 TC settings on BraTS2018, with average Dice improvements over the best competing method of +3.51\% for ET, +2.19\% for TC, and +1.14\% for WT (Wang et al., 2024). AdaMM reports that, compared with MouseGAN++ and Hyper-GAE, average single-modality Dice on BraTS 2024 improves by about +23.55\% and +23.74\%, and single-modality IoU improves by +14.71\% and +17.37\% (Zhu et al., 18 Sep 2025).

Robust cross-modal adaptation in unconstrained or heterogeneous settings shows the same pattern. On UCF51, the MNF+CSC framework reports 72.9 Acc, 72.3 R@1, 91.4 mAP@100, and 82.3 mAP@500; on ActivityNet it reports 52.9 Acc, 48.4 R@1, 50.5 mAP@100, and 41.0 mAP@500 (Xia et al., 2023). For widely differing modalities, soft-constrained distillation improves speaker recognition from 1.71% EER to 1.51% EER and improves CIFAR-100 image classification from 77.07% to 79.14% with classifier-level distillation (Zhao et al., 22 Jul 2025). In multimodality foundation models, OPTIMA improves CoCa-Tiny$G = A \cdot A^{T}.$3 from 71.2 to 71.6 on VQA, 83.9 to 84.3 on SNLI-VE, 80.4 to 81.3 on NLVR2, and 116.8 to 121.2 CIDEr on COCO Caption (Liang et al., 2023).

These empirical results support a regularity visible across the literature: gains are typically largest when the modality gap is large, when the dominant modality is missing, or when modality contributions are strongly asymmetric. This suggests that the main benefit of modality adaptation is not merely extra supervision, but better control over transferability.

6. Misconceptions, limits, and research directions

A persistent misconception is that a more accurate multimodal teacher is automatically a better teacher. The counterexamples in AV-MNIST and NYU Depth V2 show otherwise, and the modality focusing hypothesis attributes the difference to whether the teacher relies on shared decisive features or on modality-specific cues unavailable to the student (Xue et al., 2022). Related evidence appears elsewhere: in Visual Question Answer-Difference prediction, distilling only from the Big Teacher gives 43.74 overall and even Big with intermediate features gives 44.61, whereas the best student result, 45.75, comes from combining Big, Visual, and Question Teachers with intermediate feature distillation (Cho et al., 2021). MST-Distill generalizes this observation by treating teacher-path choice as an instance-level routing problem rather than a fixed design decision (Li et al., 9 Jul 2025).

Another misconception is that direct feature equality is always desirable. Several papers argue the opposite for cross-modal transfer. Widely differing modalities can cause hard $G = A \cdot A^{T}.$4 alignment to overfit modality-specific content (Zhao et al., 22 Jul 2025). Unconstrained videos require teacher-side purification before distillation because the teacher itself may be noisy (Xia et al., 2023). DisCoM-KD argues that transferable cross-modal knowledge should be decomposed into invariant, informative, and irrelevant factors rather than compressed into a single imitation objective (Ienco et al., 2024).

Limitations are also recurrent. ProtoKD still depends on a well-trained multi-modal teacher and on labeled pixel-level information to construct prototypes, and its evaluation is mainly on BraTS (Wang et al., 2023). MetaKD explicitly notes that its missing-modality feature generation is simple mean pooling and suggests future work with conditional generative models or GAN-based feature synthesis (Wang et al., 2024). OPTIMA uses a coarse action space of module subsets and does not model dependencies between arms/modules (Liang et al., 2023). AdaMM’s comparative study concludes that knowledge distillation is stronger overall than data generation, feature generation, robustness enhancement, and multi-task learning for missing-modality brain tumor segmentation, but its own design still assumes a full-modality teacher during training (Zhu et al., 18 Sep 2025).

The field therefore converges on a relatively clear principle. Modality-adaptive KD is most effective when it explicitly distinguishes transferable from non-transferable content, assigns distillation strength according to modality relevance or sample reliability, and avoids assuming that one static teacher or one uniform objective is sufficient. A plausible implication is that future progress will depend less on stronger teachers per se than on better estimators of transferability—whether through saliency, meta-learning, graph structure, routing, disentanglement, or reliability priors.