Heterogeneous Multi-Teacher Distillation

• Heterogeneous Multi-Teacher Distillation is a regime in which a single student learns from multiple, diverse teachers differing in architecture, modality, and training data.
• It aggregates various supervisory signals—from logits and intermediate features to task-specific outputs—to mitigate capacity gaps and enhance domain-specific performance.
• Practical implementations span federated learning, vision, audio, and LLM post-training, demonstrating improved robustness and efficiency over single-teacher methods.

Heterogeneous multi-teacher distillation is a many-to-one knowledge distillation regime in which a single student learns from multiple teachers that may differ in architecture, size, depth, tokenizer, training data, task objective, modality, or deployment domain. In contemporary formulations, the student may absorb teacher information through probability distributions, logits, intermediate representations, sequence-level pseudo-labels, or task-specific decoders, and the central problem is how to aggregate nonidentical supervisory signals without sacrificing robustness, safety, or domain-specific capability (Meng et al., 21 Jul 2025, Sariyildiz et al., 18 Mar 2025, Flouro et al., 25 Jan 2026).

1. Problem formulation and scope

A common formulation writes multi-teacher distillation as a supervised-plus-distillation objective,

$$L(\theta) = \mathbb{E}_{(x,y)} \left[ \alpha L_{\mathrm{sup}}(S(x), y) + \sum_{i=1}^m w_i(x) D\!\left( T_i^\tau(x) \,\|\, S^\tau(x) \right) \right],$$

with a sequence-level extension

$$L_{\mathrm{token}}(\theta) = \mathbb{E}_{x} \left[ \sum_{t=1}^{|x|} \sum_{i=1}^m w_{i}^{(\mathrm{tok})}(x,t) D\!\left( T_{i,t}^\tau(x) \,\|\, S_{t}^\tau(x) \right) \right].$$

The same framework has been generalized to token, task, and context scales, where teachers provide per-position distributions, per-task heads or losses, and context-level features tied to deployment domains, subpopulations, safety-critical scenarios, or uncertainty signals (Flouro et al., 25 Jan 2026).

A second formulation operates directly in the probability domain. In collaborative language-model distillation, the unified teacher target is

$$P_T(y \mid x) = \sum_{k=1}^K a_k \, P_{T_k}(y \mid x), \qquad \sum_{k=1}^K a_k = 1,$$

and the student is trained against the fused teacher distribution with KL divergence, optionally combined with supervised cross-entropy and feature alignment. In the operator-theoretic view, multi-teacher aggregation is written as

$$p^{\mathrm{agg}}(\cdot|x) = A(\{p^{(k)}_{T_k}(\cdot|x)\}_{k=1}^K; w, \tau),$$

where the aggregation operator receives teacher distributions, teacher temperatures, simplex-constrained weights, and an optional aggregation temperature (Meng et al., 21 Jul 2025, Flouro et al., 14 Jan 2026).

This breadth of formulation is important because heterogeneous multi-teacher distillation is not confined to one task family. The same abstraction appears in federated learning, speech–music representation unification, compact SSL vision encoders, low-resource summarization, model merging on scarce unlabeled data, and post-training capability integration in frontier LLMs (Shadin et al., 1 Jun 2026, Wei et al., 8 Jun 2025, Mandalika et al., 6 Aug 2025, Sumit et al., 3 Apr 2026, Dalili et al., 24 Dec 2025, Ma et al., 29 Jun 2026).

2. Axioms, guarantees, and theoretical structure

One line of work develops an operator-agnostic axiomatic framework for adaptive weighting across token, task, and context scales. Its standing assumptions are bounded weights, Lipschitz continuity, realizability, and smoothness; its scale-specific axioms impose normalization, positivity, bounded influence, and continuity, with ordinal safety monotonicity at token level, Pareto compatibility at task level, and Safety Context Prioritization at context level. The framework proves existence and non-uniqueness of conforming operators, composes scale-specific weights through product-structure normalization,

$$\tilde w_{\mathrm{unified},k}(x,i,t,c)=w_{\mathrm{tok},k}(x,i,c)\,w_{\mathrm{task},k}(t)\,w_{\mathrm{ctx},k}(c),$$

followed by normalization across teachers, and establishes operator-agnostic convergence of SGD with $$\mathbb{E}[L_{\mathrm{KD}}(\theta_t;G)]-L^\ast \le C/t$$, vanishing KL at accumulation points, bounded gradient variance, contraction-based fixed-point stability for weight updates, perturbation robustness, and a formal treatment of safety-constrained distillation (Flouro et al., 25 Jan 2026).

A complementary probability-domain framework specifies five axioms for valid aggregation operators: Convexity Preservation, Positivity Inheritance, Weight Monotonicity, Continuity, and Temperature Coherence. It proves that weighted arithmetic means, logit-space averaging with softmax, log opinion pools, and power-mean families can all satisfy these axioms, while remaining non-equivalent implementations. For linear-in-weight operators, aggregation reduces stochastic variance as $w^\top \Sigma w$ under standard assumptions, attenuates supervisory bias through convex combination of teacher biases, satisfies a Jensen-type inequality

$$\mathrm{KL}\!\left(\sum_k w_k p^{(k)}_{T_k} \,\middle\|\, p^{(S)}_{T_S}\right) \le \sum_k w_k \,\mathrm{KL}\!\left(p^{(k)}_{T_k} \,\middle\|\, p^{(S)}_{T_S}\right),$$

and yields a log-loss bound and a safety attenuation property for unsafe tokens (Flouro et al., 14 Jan 2026).

A third theoretical strand treats post-hoc merging as multi-teacher distillation. It introduces a flatness-aware PAC-Bayes bound with a cross-task heterogeneity term

$$H_Q(\alpha,\beta)=\sum_{i=1}^T\sum_{j=1}^T \alpha_i \beta_j \big[L_{D_i}(Q_j)-L_{D_j}(Q_j)\big],$$

which formalizes mismatch between diverse fine-tuned model priors and target multi-task distributions. Within that framework, minimizing the student–teacher KL on scarce, unlabeled data tightens an upper bound on the merged model’s excess risk, and SAM is used to target flatter minima because the bound contains explicit flatness terms through squared-gradient proxies and kernel-weighted dispersion penalties (Dalili et al., 24 Dec 2025).

3. Teacher construction and supervision channels

Heterogeneous teacher sets need not consist of independently trained full models. “Distillation via Intermediate Heads” creates a cohort of teachers by attaching lightweight classifier heads to a frozen pretrained backbone at multiple depths. For ResNet teachers, heads are mounted after each group of same-dimensional residual blocks; for VGG11, after each max-pooling layer. The cohort includes the original final head and the intermediate heads, and the student is trained with

$$\mathcal{L}_{\mathrm{KD}}^{\mathrm{DIH}}(x)=\frac{1}{k+1}\sum_{k=1}^{k+1}\tau^2 H\!\big(p_{T_k}^\tau(x),p_s^\tau(x)\big),$$

combined with hard-label cross-entropy. The stated rationale is capacity-gap mitigation: shallow heads provide simpler, often higher-entropy supervision, while deeper heads preserve sharper decision boundaries (Asadian et al., 2021).

A different construction inserts explicit teacher assistants. Densely Guided Knowledge Distillation builds a sequence $$L_{\mathrm{token}}(\theta) = \mathbb{E}_{x} \left[ \sum_{t=1}^{|x|} \sum_{i=1}^m w_{i}^{(\mathrm{tok})}(x,t) D\!\left( T_{i,t}^\tau(x) \,\|\, S_{t}^\tau(x) \right) \right].$$0 and departs from sequential teacher-assistant KD by densely guiding each smaller model with every larger model seen so far. The student receives all KD terms simultaneously, and stochastic teaching randomly drops some teachers or assistants at mini-batch level. Because DGKD is logit-level, it is architecture-agnostic and does not require feature-space alignment (Son et al., 2020).

Another direction models heterogeneity through information flow rather than logits. “Heterogeneous Knowledge Distillation using Information Flow Modeling” defines a network’s information flow as the vector of layerwise mutual informations with the targets, matches teacher and student via kernel-based conditional probability tables derived from Quadratic Mutual Information, and stabilizes heterogeneous transfer with an auxiliary teacher whose topology matches the student but has doubled channels. The student is then trained with a phase-aware loss

$$L_{\mathrm{token}}(\theta) = \mathbb{E}_{x} \left[ \sum_{t=1}^{|x|} \sum_{i=1}^m w_{i}^{(\mathrm{tok})}(x,t) D\!\left( T_{i,t}^\tau(x) \,\|\, S_{t}^\tau(x) \right) \right].$$1

where intermediate-layer weights decay over epochs while the final layer retains unit weight, reflecting the paper’s “critical learning periods” perspective (Passalis et al., 2020).

These constructions show that “multi-teacher” can denote multiple full teachers, multiple heads cut from one teacher, multiple assistants arranged by capacity, or proxy teachers that mediate between incompatible architectures. A plausible implication is that heterogeneity is as much about supervisory granularity as about model diversity.

4. Weighting, routing, and normalization mechanisms

The literature spans fixed averaging, confidence-driven weighting, agreement-aware routing, distribution balancing, and explicit normalization of teacher targets. Representative mechanisms are summarized below.

Mechanism	Core rule	Representative setting
Distillation via Intermediate Heads (Asadian et al., 2021)	Uniform $$L_{\mathrm{token}}(\theta) = \mathbb{E}_{x} \left[ \sum_{t=1}^{|x|} \sum_{i=1}^m w_{i}^{(\mathrm{tok})}(x,t) D\!\left( T_{i,t}^\tau(x) \,\|\, S_{t}^\tau(x) \right) \right].$$2 across heads, including the “Main” head	Capacity-gap mitigation in CNN classification
FedMTFI (Shadin et al., 1 Jun 2026)	Uniform teacher averaging, plus scalar SHAP weight $$L_{\mathrm{token}}(\theta) = \mathbb{E}_{x} \left[ \sum_{t=1}^{|x|} \sum_{i=1}^m w_{i}^{(\mathrm{tok})}(x,t) D\!\left( T_{i,t}^\tau(x) \,\|\, S_{t}^\tau(x) \right) \right].$$3 multiplying $$L_{\mathrm{token}}(\theta) = \mathbb{E}_{x} \left[ \sum_{t=1}^{|x|} \sum_{i=1}^m w_{i}^{(\mathrm{tok})}(x,t) D\!\left( T_{i,t}^\tau(x) \,\|\, S_{t}^\tau(x) \right) \right].$$4	Heterogeneous federated learning
Collaborative multi-teacher LM distillation (Meng et al., 21 Jul 2025)	Entropy-driven dynamic weights $$L_{\mathrm{token}}(\theta) = \mathbb{E}_{x} \left[ \sum_{t=1}^{|x|} \sum_{i=1}^m w_{i}^{(\mathrm{tok})}(x,t) D\!\left( T_{i,t}^\tau(x) \,\|\, S_{t}^\tau(x) \right) \right].$$5	Parameter-efficient language-model deployment
EWAD / CPDP (Sumit et al., 3 Apr 2026)	Entropy-weighted confidence, JSD agreement gate $$L_{\mathrm{token}}(\theta) = \mathbb{E}_{x} \left[ \sum_{t=1}^{|x|} \sum_{i=1}^m w_{i}^{(\mathrm{tok})}(x,t) D\!\left( T_{i,t}^\tau(x) \,\|\, S_{t}^\tau(x) \right) \right].$$6, and capacity-proportional divergence preservation	Low-resource abstractive summarization
CoMAD (Mandalika et al., 6 Aug 2025)	Per-token consensus gate from student–teacher cosine affinity and inter-teacher agreement	SSL vision-transformer distillation
SFedKD (Xu et al., 11 Jul 2025)	Decoupled target/non-target weights $$L_{\mathrm{token}}(\theta) = \mathbb{E}_{x} \left[ \sum_{t=1}^{|x|} \sum_{i=1}^m w_{i}^{(\mathrm{tok})}(x,t) D\!\left( T_{i,t}^\tau(x) \,\|\, S_{t}^\tau(x) \right) \right].$$7 and $$L_{\mathrm{token}}(\theta) = \mathbb{E}_{x} \left[ \sum_{t=1}^{|x|} \sum_{i=1}^m w_{i}^{(\mathrm{tok})}(x,t) D\!\left( T_{i,t}^\tau(x) \,\|\, S_{t}^\tau(x) \right) \right].$$8 from class-distribution discrepancy	Sequential federated learning
PHI-S (Ranzinger et al., 2024)	Isotropic standardization $$L_{\mathrm{token}}(\theta) = \mathbb{E}_{x} \left[ \sum_{t=1}^{|x|} \sum_{i=1}^m w_{i}^{(\mathrm{tok})}(x,t) D\!\left( T_{i,t}^\tau(x) \,\|\, S_{t}^\tau(x) \right) \right].$$9	Label-free agglomerative models

Within the multi-scale axiomatic framework, token, task, and context operators are composed by product-then-normalize, and the same paper gives two admissible families: confidence/entropy-based weights and disagreement/variance-aware weights. This formulation explicitly separates what any weighting scheme must satisfy from how weights are instantiated, and it allows only a subset of scales to be used while retaining consistent normalization through a log-additive representation (Flouro et al., 25 Jan 2026).

In collaborative LLM distillation, teacher weights are recomputed per input by inverse entropy normalization,

$$P_T(y \mid x) = \sum_{k=1}^K a_k \, P_{T_k}(y \mid x), \qquad \sum_{k=1}^K a_k = 1,$$0

and the student also matches intermediate representations through

$$P_T(y \mid x) = \sum_{k=1}^K a_k \, P_{T_k}(y \mid x), \qquad \sum_{k=1}^K a_k = 1,$$1

By contrast, FedMTFI keeps teacher averaging uniform and places the adaptive emphasis on feature importance via the scalar SHAP multiplier $$P_T(y \mid x) = \sum_{k=1}^K a_k \, P_{T_k}(y \mid x), \qquad \sum_{k=1}^K a_k = 1,$$2, while PHI-S does not weight teachers differently but instead balances the geometry of each teacher’s activation distribution before matching (Meng et al., 21 Jul 2025, Shadin et al., 1 Jun 2026, Ranzinger et al., 2024).

A recurring misconception is that multi-teacher distillation always implies learned teacher weights. Several systems instead report strong performance with uniform teacher averaging, while reserving adaptation for routing, confidence gates, token masking, teacher selection, or target-space normalization (Asadian et al., 2021, Shadin et al., 1 Jun 2026, Sariyildiz et al., 18 Mar 2025).

5. Heterogeneity across domains and deployment settings

In federated learning, heterogeneity is explicitly device-level, model-level, and statistical. FedMTFI clusters clients by hardware capability and model type, trains one cluster-specific model per cluster on non-IID data, aggregates locally trained models within each cluster by FedAvg into prototypes, and distills the resulting teacher set into a compact global student on public or auxiliary data. SHAP is computed on the server and averaged across teachers, so the student focuses on features teachers collectively deem important; no additional client communication occurs during server-side distillation (Shadin et al., 1 Jun 2026). SFedKD addresses a different federated regime—Sequential Federated Learning—by selecting multiple teachers from the previous round, weighting target-class and non-target-class distillation separately according to class-distribution discrepancy, and greedily pruning redundant teachers through a complementary selection mechanism framed as a variant of maximum coverage (Xu et al., 11 Jul 2025).

In audio, heterogeneous multi-teacher distillation has been used to unify speech and music representation models. The reported framework uses HuBERT or WavLM as speech teachers, MERT as the music teacher, and a shared student with teacher-specific feature translators. Data-Domain Separation routes speech samples to the speech teacher and music samples to the music teacher, while the total loss

$$P_T(y \mid x) = \sum_{k=1}^K a_k \, P_{T_k}(y \mid x), \qquad \sum_{k=1}^K a_k = 1,$$3

is tuned to avoid domination by the easier-to-distill music teacher; the best trade-off in the paper occurs at $$P_T(y \mid x) = \sum_{k=1}^K a_k \, P_{T_k}(y \mid x), \qquad \sum_{k=1}^K a_k = 1,$$4 (Wei et al., 8 Jun 2025).

In vision, heterogeneous multi-teacher distillation appears in several distinct forms. CoMAD unifies MAE, MoCo v3, and iBOT by giving the student a heavier mask than the teachers, aligning teacher tokens into the student space with lightweight adapters, and fusing tokenwise advice through a parameter-free consensus gate. DUNE distills a ViT-Base universal encoder from DINO-v2, MASt3R, and Multi-HMR, uses transformer projectors rather than simple MLPs, and studies “full sharing” versus restricted sharing of heterogeneous datasets. Label-free agglomerative models, by contrast, train a single student vision encoder to match multiple heterogeneous foundation models without labels and focus on target conditioning rather than dynamic teacher weights (Mandalika et al., 6 Aug 2025, Sariyildiz et al., 18 Mar 2025, Ranzinger et al., 2024).

In LLM post-training, MOPD defines heterogeneity at the level of RL recipe and capability. It first trains domain-specialized RL teachers, then distills them into a student on the student’s own rollouts with a reverse-KL objective,

$$P_T(y \mid x) = \sum_{k=1}^K a_k \, P_{T_k}(y \mid x), \qquad \sum_{k=1}^K a_k = 1,$$5

Routing is deterministic by domain, and teacher-prefill is served asynchronously as an external service, so the method is on-policy with respect to the student and avoids the exposure bias of off-policy finetuning (Ma et al., 29 Jun 2026).

In low-resource summarization, heterogeneity arises from model family, tokenizer compatibility, teacher capacity, and language. EWAD routes per-token supervision between teacher KD and gold supervision according to inter-teacher agreement and teacher confidence, while CPDP imposes a geometric constraint that the student should lie closer to the smaller teacher than to the larger one in a way proportional to inter-teacher divergence. The same study includes same-family logit KD, cross-architecture pseudo-label KD, and cross-lingual pseudo-label KD across ten languages (Sumit et al., 3 Apr 2026).

6. Empirical record

Empirical results show that heterogeneous multi-teacher distillation often improves over single-teacher or heuristic baselines, but the gains are setting-dependent. DIH reports consistent gains over canonical KD across several teacher–student pairs. For a Res8 student on CIFAR-100, DIH reaches 63.79 with a VGG11 teacher versus 61.10 for KD and 60.47 for CE; with WRN-28-2, DIH reaches 63.32 versus 60.78 for KD. On Tiny-ImageNet, a Res8 student with a WR28-2 teacher reaches 43.89 versus 40.70 for KD, and with VGG11 reaches 43.78 versus 40.76 (Asadian et al., 2021).

In heterogeneous federated learning, FedMTFI reports 64.48% on CIFAR-10 and 87.28% on FMNIST for the student model, compared with 56.15% and 80.65% for FedAvg, 58.39% and 82.31% for FedProx, and 59.53% and 84.43% for FedKDShap. Its SHAP ablation on the 3-cluster, 10-client, 30-round configuration shows 67.94% versus 70.61% on CIFAR-10 and 91.55% versus 93.43% on FMNIST without and with SHAP, respectively (Shadin et al., 1 Jun 2026).

Cross-domain audio distillation reports that the best multi-distill configuration, using DDS, a hybrid translator, and $$P_T(y \mid x) = \sum_{k=1}^K a_k \, P_{T_k}(y \mid x), \qquad \sum_{k=1}^K a_k = 1,$$6, yields 80.40% average for HB+MR and 78.17% average for WL+MR, both close to larger ensembles of two distilled students. In few-shot evaluation with 20 shots per class, HB+MR reaches 55.39 average accuracy and WL+MR reaches 56.34, outperforming the corresponding single-teacher distilled baselines (Wei et al., 8 Jun 2025).

For compact SSL vision models, CoMAD’s ViT-Tiny reaches 75.4% Top-1 on ImageNet-1K, 47.3% mIoU on ADE20K, and 44.5% box average precision with 40.5% mask average precision on MS-COCO. The same paper reports that the combined gate outperforms uniform averaging and one-sided gating, and that using all three teachers performs better than any teacher pair or single teacher (Mandalika et al., 6 Aug 2025).

In label-free agglomerative modeling, PHI-S achieves the best average benchmark rank among the methods studied: 2.92 for ViT-B/16, versus 3.81 for global standardization, 4.21 for channelwise standardization, and 7.83 for baseline MSE. The same study reports raw teacher-space MSEs for PHI-S of 4.7200e−4 on DFN-CLIP, 1.9010 on SigLIP, 0.8865 on DINOv2, and 8.3330 on SAM (Ranzinger et al., 2024).

In LLM post-training, MOPD reports a normalized score of 0.937 on Qwen3-30B-A3B, above Mix-RL at 0.882, Cascade RL at 0.775, Off-Policy Finetune at 0.824, and linear-averaging Param-Merge at 0.328. On MiMo-V2-Flash, MOPD matches or exceeds teachers on several benchmarks, including AIME25 at 94.1 versus 93.9 for the teacher, HMMT25 at 84.4 versus 82.6, LCB at 83.2 versus 82.6, and $$P_T(y \mid x) = \sum_{k=1}^K a_k \, P_{T_k}(y \mid x), \qquad \sum_{k=1}^K a_k = 1,$$7-Bench at 80.3 versus 79.6 (Ma et al., 29 Jun 2026).

Not all results are uniformly positive. In low-resource summarization, logit-level KD yields the most reliable gains; on BTS, A2 reaches ROUGE-L 0.3797 versus the teacher’s 0.4058, retaining 93.6% of teacher ROUGE-L, while on BanSum A2 reaches ROUGE-L 0.2877 versus 0.2998, retaining 95.9%. However, the Qwen2.5 EWAD/CPDP ablation on the 20K BanSum subset shows that baseline fine-tuning remains best, with ROUGE-L 0.2160 and BERTScore F1 0.7389, while EWAD reaches 0.1767 ROUGE-L and EWAD+CPDP reaches 0.1740 (Sumit et al., 3 Apr 2026).

7. Limitations, controversies, and open directions

A central limitation is that formal validity rarely determines a unique implementation. The operator-agnostic framework explicitly proves non-uniqueness: multiple conforming operators satisfy the same axioms but yield different constants and behaviors, and violating boundedness or continuity can destabilize optimization. The same work identifies open problems in axiom minimality and independence, complexity bounds, interactions with gradient surgery and Pareto MTL, non-stationary or continual learning, scale-specific temperature operators, and tighter lower bounds matching the reported $$P_T(y \mid x) = \sum_{k=1}^K a_k \, P_{T_k}(y \mid x), \qquad \sum_{k=1}^K a_k = 1,$$8 rate (Flouro et al., 25 Jan 2026).

Several empirical studies identify setting-specific failure modes. In MOPD, stability depends strongly on “same-origin” teachers: replacing a teacher with a stronger but distributionally different external model raised the initial per-token KL by about fivefold and destabilized training, with catastrophic collapse for the top-k variant around step 18. In summarization, confidence-only weighting “fails severely,” and more complex distillation improves semantic similarity for short summaries but degrades longer outputs; the same paper concludes that “data scaling outweighs loss engineering” in some regimes (Ma et al., 29 Jun 2026, Sumit et al., 3 Apr 2026).

Other limitations are structural. FedMTFI assumes access to a public or auxiliary dataset for distillation and SHAP analysis and depends on reasonable clustering quality; poor clustering can reduce teacher diversity or produce weak prototypes. PHI-S assumes normality and requires a Hadamard matrix of size $$P_T(y \mid x) = \sum_{k=1}^K a_k \, P_{T_k}(y \mid x), \qquad \sum_{k=1}^K a_k = 1,$$9, while whitening-based alternatives underperform because many teacher feature distributions are not full rank and $$p^{\mathrm{agg}}(\cdot|x) = A(\{p^{(k)}_{T_k}(\cdot|x)\}_{k=1}^K; w, \tau),$$0 becomes ill-conditioned. DUNE, despite strong results, does not impose explicit geometric consistency at distillation time, and the authors identify adaptive teacher weighting, explicit geometric consistency, dynamic data-sharing policies, and conflict-aware co-training as future directions (Shadin et al., 1 Jun 2026, Ranzinger et al., 2024, Sariyildiz et al., 18 Mar 2025).

A broader controversy concerns whether heterogeneity should be resolved in weight space, probability space, representation space, or routing space. The current record suggests that no single answer dominates across all domains. Probability-domain aggregation offers strong axiomatic guarantees; feature translators and proxy teachers help when latent spaces differ sharply; reliability gates and discrepancy-aware weighting matter under disagreement or class imbalance; and distribution balancing can outperform teacher reweighting when the primary issue is activation-scale mismatch. This suggests that “heterogeneous multi-teacher distillation” is better understood as a family of design principles—aggregation, alignment, routing, and normalization—than as a single algorithmic template.