Adaptive Multi-Modal Multi-Teacher Distillation

• Adaptive Multi-Modal Multi-Teacher Knowledge Distillation is a framework that leverages multiple teacher models and varied modalities with dynamic weighting to drive effective knowledge transfer.
• It employs adaptive, instance-level weighting strategies to flexibly integrate insights from modality-conditioned teachers, avoiding fixed uniform influence.
• Empirical findings in vision-language tasks demonstrate improved performance and efficiency, highlighting its potential for robust, lightweight model deployment.

Adaptive Multi-Modal Multi-Teacher Knowledge Distillation (AMMKD) refers to distillation settings that combine three design commitments: supervision from more than one teacher, the use of multiple modalities or modality-conditioned teachers, and an adaptive mechanism that changes how teacher knowledge is transferred rather than assigning fixed, uniform importance. The term is also the title of a 2025 framework for lightweight vision-LLMs that combines multi-modal feature fusion, multi-teacher distillation, and adaptive optimization for CLIP-style image–text alignment (Li et al., 23 Aug 2025). In the surrounding literature, AMMKD sits at the intersection of generalized distillation from multimodal teachers to modality-limited students (Hu et al., 2021), adaptive multi-teacher weighting (Liu et al., 2021), task-aware teacher election under missing modalities (Wang et al., 2023), structural privileged distillation with multiple teachers (Aslam et al., 2024), and instance-level routing over heterogeneous cross-modal teachers (Li et al., 9 Jul 2025).

1. Conceptual scope and defining axes

The phrase “adaptive multi-modal multi-teacher knowledge distillation” is best understood as a compound descriptor rather than a single canonical algorithm. “Multi-modal” may mean that teachers consume different sensory streams, that a teacher is itself multimodal while the student is modality-limited, or that teacher supervision explicitly preserves modality relations. “Multi-teacher” may denote several independently trained teachers, a fused teacher plus modality-specific teachers, or a set of specialized teacher variants derived from a smaller number of source models. “Adaptive” may range from hard task-wise teacher election to instance-level soft weighting, top-$k$ routing, entropy-based mixture coefficients, or confidence-driven path balancing (Wang et al., 2023).

A recurring source of confusion in the literature is that many influential precursors satisfy only two of the three properties. KD-Net distills from a multi-modal MRI teacher to a mono-modal student, but it is single-teacher and uses fixed hyperparameters rather than adaptive routing (Hu et al., 2021). “Adaptive Multi-Teacher Multi-level Knowledge Distillation” introduces instance-specific teacher weights and multi-level transfer, but all experiments are unimodal image classification (Liu et al., 2021). “Enriching Knowledge Distillation with Cross-Modal Teacher Fusion” fuses a conventional visual teacher with CLIP, making it both multi-modal and multi-teacher, yet the fusion uses fixed global weights $\alpha$ and $\lambda$ and is explicitly not adaptive (Mansourian et al., 12 Nov 2025). LCKD is adaptive in a narrower sense: it elects one best teacher per task and unions those task-wise winners into a teacher set, but it does not learn per-sample or per-batch teacher weights (Wang et al., 2023).

This boundary-setting matters because AMMKD is not exhausted by any single mechanism. Some methods prioritize teacher diversity, others prioritize modality-gap reduction, and others prioritize adaptive aggregation. The most mature formulations combine all three.

2. Genealogy and representative formulations

The literature is better organized as overlapping lines of work than as a single linear progression. Early multimodal-to-monomodal distillation established that privileged modalities can improve a student even when those modalities are absent at inference; later work added teacher diversity, adaptive routing, and structural transfer. The table summarizes representative positions in this space.

Method	Status relative to AMMKD	Characteristic mechanism
KD-Net (Hu et al., 2021)	Multimodal, single-teacher, non-adaptive	Soft output KD plus bottleneck KL from multi-modal MRI teacher to T1ce student
LCKD (Wang et al., 2023)	Multimodal, task-adaptive, hard multi-teacher set	Per-task teacher election by validation Dice, bottleneck feature alignment
PromptMM (Wei et al., 2024)	Multimodal, single-teacher, adaptive quality distillation	Pairwise, listwise, and embedding KD with prompt-tuning
MT-PKDOT (Aslam et al., 2024)	Multimodal, multi-teacher, batch-wise hard selection	Aligned teacher pool and regularized OT on similarity matrices
MST-Distill (Li et al., 9 Jul 2025)	Cross-modal, multi-teacher, instance-routed	GateNet top-$k$ routing and MaskNet teacher specialization
AMMKD (Li et al., 23 Aug 2025)	Multimodal, two-teacher, adaptive weighting	KL + CLIP + MSE with dynamic weighting and MGDA-style optimization

This genealogy also includes methods that contribute a single major ingredient. The modality-level Gram-matrix approach distills relations among text-only, image-only, and joint image+text outputs, making modality interaction itself a distillation target rather than only the final prediction (Liu, 2021). AGSSL, যদিও graph-specific, provides a theoretically explicit instance-level integrated-teacher distribution,

$$\mathbf{p}^{\mathrm{t}(\mathbf{x}_i)}=\sum_{k=1}^K \lambda_\gamma(k,i)\widetilde{\mathbf{h}_i^{(k)}},$$

and argues that adaptive teacher integration should approximate the true Bayesian class-probability (Wu et al., 2022). MT-BKD then reinterprets multi-teacher KD as Bayesian inference under a teacher-informed mixture prior with sample-wise entropy-based teacher weights (Fang et al., 27 May 2026).

3. Canonical objectives and supervision paths

AMMKD does not have a single universal loss. Instead, the literature repeatedly combines supervised task loss with one or more teacher-guided terms operating at the output, feature, structural, or probabilistic-prior level.

In the 2025 AMMKD framework for lightweight vision-LLMs, the student is trained with image-to-text and text-to-image KL matching, a CLIP-style task loss, and feature MSE: $$L_{kl}=\lambda_1 L_{kl}(P \mid Q)+\lambda_2 L_{kl}(Q \mid P),$$

$L_{mse}=MSE(u_i^T,u_i^S)+MSE(w_i^T,w_i^S),$

$L=L_{kl}+L_{clip}+L_{mse}.$

The corresponding teacher and student distributions are defined over normalized image and text embeddings in a shared space, and the paper frames teacher balancing as a multi-objective optimization problem over competing teacher gradients (Li et al., 23 Aug 2025).

Output-distribution aggregation appears in several other forms. AGSSL distills from an integrated teacher posterior obtained by sample-wise weighting of teacher probabilities and supervises the student with KL divergence (Wu et al., 2022). MT-BKD replaces a weighted sum of KD losses with a teacher-informed mixture prior,

$$\pi\!\left(\theta \mid \{\mathbf{p}_i^{(g)}\}\right)\propto \prod_{i=1}^N \sum_{g=1}^G w_i^{(g)} f(h(\mathbf{x}_i;\theta)\mid \mathbf{p}_i^{(g)}),$$

thereby treating teacher aggregation as part of posterior construction rather than as a post hoc average of losses (Fang et al., 27 May 2026).

Feature and latent alignment remain equally central. KD-Net aligns a mono-modal student with a multimodal teacher both through segmentation outputs and through KL divergence between teacher and student bottleneck representations (Hu et al., 2021). LCKD applies cross-modal knowledge distillation at the bottom stage of a 3D U-Net and uses pairwise bottleneck feature matching between available teacher and student modalities, with the total loss

$$\ell_{tot}(\mathcal{D},\mathbf{\Theta})=\ell_{task}(\mathcal{D},\theta^{enc},\theta^{dec})+\alpha\,\ell_{ckd}(\mathcal{D};\theta^{enc})$$

for missing-modality segmentation (Wang et al., 2023). In the setting of widely differing modalities, hard $\alpha$0 alignment is explicitly rejected in favor of soft constraints such as

$\alpha$1

together with shared-classifier supervision and quality-based sample weighting (Zhao et al., 22 Jul 2025).

Structural objectives broaden the notion of what is distilled. The modality-level Gram-matrix method defines

$\alpha$2

from outputs under text-only, image-only, and joint image+text inputs, then matches teacher and student modality relation matrices with MSE (Liu, 2021). MT-PKDOT builds cosine similarity matrices from teacher and student batch features and minimizes an entropy-regularized OT loss over those structural representations, supplemented by a centroid-alignment term (Aslam et al., 2024). These methods treat geometry and cross-sample relations as transferable knowledge in their own right.

4. Adaptive teacher selection, weighting, and routing

The defining feature of AMMKD is not merely the presence of multiple teachers, but the refusal to assign them fixed, uniform influence. The literature offers several distinct adaptive regimes.

Hard teacher election is the simplest form. LCKD selects, for each task $\alpha$3, the modality with the highest single-modality validation Dice,

$\alpha$4

and then forms a deduplicated teacher set

$\alpha$5

This is adaptive because teacher identity depends on task competence, but it remains a hard argmax over validation statistics rather than a learned, sample-wise weighting mechanism (Wang et al., 2023). MT-PKDOT similarly performs batch-wise hard selection by choosing the teacher representation with minimum task loss for the current batch, using the joint multimodal teacher as fallback when aligned modality-specific teachers are worse (Aslam et al., 2024).

Soft instance-level weighting is more expressive. AMTML-KD assigns each teacher a latent vector $\alpha$6 and computes sample-specific weights

$\alpha$7

where $\alpha$8 is derived from the student’s last convolutional representation. Those weights define an integrated soft target for high-level KD and also support multi-level transfer (Liu et al., 2021). AGSSL provides two related sample-wise weight parameterizations, one latent-factor based and one teacher–student similarity based, both normalized so that $\alpha$9 (Wu et al., 2022).

Confidence- and uncertainty-based weighting is another major branch. MMT-ARD defines teacher confidence as

$\lambda$0

then computes a confidence ratio and adaptive sigmoid weight

$\lambda$1

to balance clean and adversarial teachers in robust vision-language distillation (Li et al., 21 Nov 2025). MT-BKD uses sample-wise entropy-based weights,

$\lambda$2

so that lower-entropy teachers receive more prior mass in the posterior (Fang et al., 27 May 2026).

Routing mechanisms make adaptivity sparse. MST-Distill feeds student logits into GateNet,

$\lambda$3

then applies top-$\lambda$4 teacher selection and distills only from the selected specialized teachers, while a load-balancing KL term prevents collapse onto a single teacher (Li et al., 9 Jul 2025). This is closer to mixture-of-experts routing than to convex loss averaging.

The contrast case is equally instructive. RichKD fuses a supervised teacher and CLIP through fixed global weights,

$\lambda$5

with $\lambda$6 and $\lambda$7, and explicitly notes that adaptive sample-wise exploitation of CLIP is future work rather than part of the method (Mansourian et al., 12 Nov 2025). Such methods are multi-modal and multi-teacher, but not AMMKD in the strict adaptive sense.

5. Multimodal transfer regimes and deployment patterns

A central AMMKD use case is privileged multimodal training with restricted unimodal deployment. KD-Net exemplifies this pattern by training a four-modality BraTS teacher and a T1ce-only student, showing that a modality-limited deployment model can absorb information from unavailable modalities through output imitation and bottleneck KL (Hu et al., 2021). MT-PKDOT studies the same general regime for multimodal expression recognition: visual-only students learn from visual+audio, visual+EDA, or visual+EMG teachers, with structural OT used to transfer relations rather than only point-wise features (Aslam et al., 2024).

Missing-modality robustness introduces a more dynamic version of the problem. LCKD assumes that any subset of modalities may be missing during training and test time, randomly drops 0 to 3 modalities during training, elects task-specific teacher modalities, and fills absent features by averaging available encoded features rather than reconstructing them with a separate generator (Wang et al., 2023). This suggests a particular AMMKD stance: if feature spaces are sufficiently aligned by distillation, crude imputation in feature space can become viable.

Several methods transfer not only predictions but also modality relations. The Gram-matrix approach computes teacher outputs under text-only, image-only, and joint image+text conditions, forms $\lambda$8, and matches teacher and student modality relation matrices, thereby making the structure of inter-modality dependence an explicit target (Liu, 2021). PromptMM, although single-teacher, likewise separates collaborative edge knowledge, modality-aware listwise ranking knowledge, and embedding-level content knowledge, and uses a disentangled listwise KD decomposition to modulate how much uncertain negative ordering should influence the student (Wei et al., 2024).

Cross-modal discrepancy itself can be the dominant challenge. In “Cross-Modal Distillation For Widely Differing Modalities,” face-to-speech and text-to-image transfer are improved by a trainable projection head,

$\lambda$9

soft margin-based feature KD, and a quality-based adaptive weight

$k$0

where quality is proxied by feature norm (Zhao et al., 22 Jul 2025). MST-Distill addresses the same problem from the teacher side by masking and reconstructing teacher features with

$k$1

thereby suppressing modality-specific drift before teacher selection (Li et al., 9 Jul 2025).

Deployment-oriented compression is a recurring consequence of these regimes. PromptMM’s student has 1.95M parameters on Netflix, 7.83% of teacher size, and 2.67M parameters on Electronics, 2.70% of teacher size, while removing modality-reduction layers from inference altogether (Wei et al., 2024). The 2025 AMMKD framework similarly precomputes and stores teacher text features as class vectors so that the deployed student can operate with reduced online text-side cost (Li et al., 23 Aug 2025).

6. Empirical findings, misconceptions, and open directions

The empirical record supports the central AMMKD premise, but it also shows that the gain depends strongly on how teacher diversity and adaptivity are realized. In medical segmentation, LCKD reports average Dice gains over the prior state of the art of 3.61% for ET, 5.99% for TC, and 3.76% for WT on BraTS2018, with teacher identity varying by tumor subregion: T1c is best for ET and TC, while Flair is best for WT (Wang et al., 2023). KD-Net shows that a T1ce-only student can improve from $k$2 to $k$3 Dice on ET and from $k$4 to $k$5 on TC through multimodal teacher supervision, although WT is essentially unchanged, underscoring that transferability can be region-dependent rather than uniform (Hu et al., 2021). MT-PKDOT improves the visual-only baseline on BioVid by 5.5% and the Aff-Wild2 visual-only baseline by 3% for valence and 5% for arousal, indicating that multiple privileged teachers can help even when inference is unimodal (Aslam et al., 2024).

Beyond segmentation, the framework family spans recommendation and vision-language modeling. PromptMM reaches $k$6 and $k$7 on Netflix while compressing to a student with zero modality-reduction cost at inference (Wei et al., 2024). The AMMKD vision-LLM reports strong prompt-based classification accuracy across seven datasets, including $k$8 on Caltech101 and $k$9 on UCF-101 in its reported settings, and its ablations show that two teachers outperform one, three, or four teachers, while dynamic weighting outperforms averaging (Li et al., 23 Aug 2025). In robust VLM distillation, MMT-ARD reports $$\mathbf{p}^{\mathrm{t}(\mathbf{x}_i)}=\sum_{k=1}^K \lambda_\gamma(k,i)\widetilde{\mathbf{h}_i^{(k)}},$$0 robust accuracy, $$\mathbf{p}^{\mathrm{t}(\mathbf{x}_i)}=\sum_{k=1}^K \lambda_\gamma(k,i)\widetilde{\mathbf{h}_i^{(k)}},$$1 zero-shot accuracy on ViT-B-32, and a $$\mathbf{p}^{\mathrm{t}(\mathbf{x}_i)}=\sum_{k=1}^K \lambda_\gamma(k,i)\widetilde{\mathbf{h}_i^{(k)}},$$2 training-efficiency increase over traditional single-teacher methods (Li et al., 21 Nov 2025). These results collectively support the proposition that teacher diversity is useful when the aggregation rule is itself adaptive.

A common misconception is that “multi-modal” or “multi-teacher” alone suffices. The literature repeatedly argues otherwise. RichKD shows that cross-modal teacher diversity can improve accuracy and robustness, but it remains a static weighted fusion method and explicitly identifies sample-wise adaptive CLIP exploitation as future work (Mansourian et al., 12 Nov 2025). LCKD is often relevant to AMMKD discussions, yet its own authors characterize teacher election as “greedy teacher selection per task” and suggest future meta-learning of modality weights per task, which would move it closer to full adaptive multi-teacher distillation (Wang et al., 2023). PromptMM contains adaptive quality distillation and modality-aware reweighting, but it is still fundamentally a single-teacher framework (Wei et al., 2024). MT-BKD contributes a principled uncertainty-aware mixture prior, but it is not itself multimodal (Fang et al., 27 May 2026).

Open problems follow directly from these limitations. One unresolved issue is teacher conflict under strong heterogeneity: MST-Distill addresses it with teacher specialization and routing, but at substantial training and memory cost (Li et al., 9 Jul 2025). Another is modality-aware uncertainty calibration: several methods weight teachers by confidence or entropy, yet few calibrate those quantities across modalities. A further issue is representation granularity. Some methods distill logits, some bottleneck features, some relation matrices, and some priors over student predictions, but a unified account of when each granularity is preferable remains absent. The literature also suggests two near-term directions. First, stronger adaptive weighting—meta-learned, sample-wise, and modality-aware—appears repeatedly as an explicit future improvement (Wang et al., 2023). Second, the move from fixed cross-modal fusion to context-aware, input-dependent fusion is already identified as a promising next step in cross-modal teacher fusion (Mansourian et al., 12 Nov 2025).

AMMKD therefore designates less a single settled method than a converging research program. Its mature forms combine heterogeneous teachers, modality-aware or cross-modal supervision, and adaptive aggregation that is strong enough to prevent negative transfer but lightweight enough to preserve the deployment advantage of distillation.