Multi-Level Knowledge Distillation

• Multi-Level Knowledge Distillation (MLKD) is a family of methods where student models are supervised using hierarchical teacher knowledge—from final predictions to intermediate and relational features.
• MLKD integrates diverse strategies such as output-intermediate supervision, alignment plus correlation, semantic granularity, and multi-teacher hierarchies to transfer rich information.
• Empirical studies show MLKD’s effectiveness with noticeable gains (e.g., up to 18.23% improvement in noisy speech emotion recognition) despite increased training complexity.

Multi-Level Knowledge Distillation (MLKD) denotes a family of distillation methods in which the student is supervised by more than one level, perspective, or semantic scope of teacher knowledge during training. In the literature surveyed here, the relevant “levels” include final predictions and intermediate features, alignment of individual samples and correlation across samples, token-, word-, sentence-, and structure-level representations, and word-, utterance-, and conversation-level semantics in multi-turn dialogue. The term is therefore broader than a single architectural recipe: in different papers it can refer to multi-layer supervision, multi-granularity teaching, relation-level transfer, or multi-stage teacher hierarchies (Tang et al., 2020, Ding et al., 2020, Li et al., 2022, Li et al., 2024).

1. Conceptual scope and definitions

A central conceptual foundation for MLKD is the claim that teacher knowledge is hierarchically structured. “Understanding and Improving Knowledge Distillation” separates teacher knowledge into three hierarchical levels: “knowledge of the `universe’,” “domain knowledge,” and “instance specific knowledge,” and argues that these three factors “play a major role in knowledge distillation” (Tang et al., 2020). In that account, universe-level knowledge behaves as regularization through label smoothing, domain-level knowledge injects class relationships into the student’s logit geometry, and instance-specific knowledge rescales per-example gradients.

A second foundation is the distinction between knowledge alignment and knowledge correlation. “Multi-level Knowledge Distillation via Knowledge Alignment and Correlation” argues that traditional KD mainly transfers knowledge alignment of an individual sample, whereas contrastive-learning-based KD methods mainly transfer relational knowledge between different samples, namely knowledge correlation. MLKD is introduced there precisely to combine both, with the stated aim of transferring “the full knowledge from teacher to student” and improving “the reliability and transferability of learned representations” (Ding et al., 2020).

The surveyed literature also shows that the phrase multi-level is not used uniformly. In incremental learning, M$^2$KD uses “multi-level” for supervision at the main final-output level and an auxiliary intermediate-output branch derived from intermediate features (Zhou et al., 2019). In multilingual pre-training, MMKD uses token-, word-, sentence-, and structure-level alignment objectives (Li et al., 2022). In multi-turn NLU, MIDAS uses “multi-level” in a semantic sense: word-level slot filling, sentence-level intent detection, and conversation-level domain classification (Li et al., 2024). In reduced-lead ECG classification, LRH-Net uses “multi-level” for a multi-step, multi-teacher distillation hierarchy rather than feature-level matching (Chauhan et al., 2022).

2. Principal formulations of “level”

The most common formulation of MLKD is output-level plus intermediate-level distillation. AMTML-KD explicitly divides teacher information into “high-level knowledge,” namely integrated soft targets, and “intermediate-level knowledge,” namely hidden-layer hints from multiple teachers (Liu et al., 2021). CA-MKD similarly combines prediction-level transfer and one intermediate feature level, using confidence-aware teacher weights computed separately for each level (Zhang et al., 2021). M$^2$KD applies output-level distillation on the main branch and auxiliary-output distillation on features extracted from the second residual block (Zhou et al., 2019).

A second formulation is alignment plus relation or structure distillation. In MLKD via knowledge alignment and correlation, the two levels are per-sample alignment and cross-sample correlation (Ding et al., 2020). In graph-based feature distillation, the levels are vertex, edge, and spectral structure within a channels relational graph, with losses

$$\mathcal{L}_{M}=\alpha \mathcal{L}_{V}+\beta \mathcal{L}_{E}+\gamma \mathcal{L}_{S},$$

followed by an overall task loss (Wang et al., 2024). In MLKD-BERT, the levels are embedding layer, Transformer layer, and prediction layer, and each of those levels mixes feature-level and relation-level objectives (Zhang et al., 2024).

A third formulation is semantic granularity. MMKD aligns token, word, sentence, and structure representations in multilingual learning (Li et al., 2022). MIDAS distills slot-, intent-, and domain-specialized teachers into a single student for multi-turn dialogue understanding (Li et al., 2024). In self-distillation, LFMA builds auxiliary predictors on shallow, middle, and deep backbone blocks and ensembles their distributions into a self-teacher (Lin et al., 2021).

A fourth formulation is multi-stage or multi-teacher hierarchy. LRH-Net defines sequential MLKD and parallel MLKD by combining a 12-lead teacher, a reduced-lead teacher, and a low-resource reduced-lead student with weighted KL terms plus BCE (Chauhan et al., 2022). Continual-learning MLKD uses multiple previous models and distills both feature-level and logit-level knowledge on external unlabeled data (Kim et al., 18 Aug 2025).

Formulation of “level”	Representative papers	Distilled knowledge
Output + intermediate features	(Zhou et al., 2019, Liu et al., 2021, Zhang et al., 2021)	Logits, auxiliary outputs, hints
Alignment + correlation	(Ding et al., 2020, Wang et al., 2024)	Per-sample alignment, sample relations, graph structure
Semantic granularity	(Li et al., 2022, Li et al., 2024)	Token/word/sentence/structure or slot/intent/domain
Multi-stage teacher hierarchy	(Chauhan et al., 2022, Kim et al., 18 Aug 2025)	Multi-teacher outputs, features, logits

This variation suggests that MLKD is best treated as a research family rather than a single canonical loss.

3. Objectives and mechanisms

At the alignment level, a standard MLKD formulation is direct feature matching after a learned transformation:

$$\mathcal{L}_{\mathrm{Align}}=\mathbb{E}_{x}\left[\left\|h_\varphi(f_\theta^S(x))-f_\eta^T(x)\right\|_2^2\right].$$

This is the explicit alignment loss used in “Multi-level Knowledge Distillation via Knowledge Alignment and Correlation” (Ding et al., 2020). That same paper adds a correlation objective so that the student preserves how samples are arranged relative to one another, not only the teacher representation of each sample in isolation.

A different but related design appears in MMKD, where the total multilingual objective is

$$\mathcal{L}=\mathcal{L}_{\mathrm{TLM}}+\mathcal{L}_{\mathrm{XWCL}}+\mathcal{L}_{\mathrm{SentA}}+\alpha \mathcal{L}_{\mathrm{StrucA}}.$$

Here the levels are explicitly token, word, sentence, and structure. The structure term uses batchwise similarity matrices and KL divergence, so relation-level knowledge is treated as a first-class distillation target rather than an auxiliary regularizer (Li et al., 2022).

MLKD-BERT provides a particularly explicit decomposition by model stage. Its first-stage objective is

$$\mathcal{L}_{\mathrm{Stage\ 1}}=\mathcal{L}_{\mathrm{EMB}}+\mathcal{L}_{\mathrm{MHA}}+\mathcal{L}_{\mathrm{FFN}},$$

and its second-stage objective is

$$\mathcal{L}_{\mathrm{Stage\ 2}}=\mathcal{L}_{\mathrm{SS}}+\mathcal{L}_{\mathrm{SC}}+\mathcal{L}_{\mathrm{KD}}.$$

The paper’s distinctive step is to distill self-attention relation from MHA outputs rather than raw attention distributions, which allows the student to use fewer attention heads than the teacher (Zhang et al., 2024).

Graph-based MLKD shifts the unit of transfer from tensor values to graph structure. In “Exploring Graph-based Knowledge,” the teacher and student are aligned at vertex, edge, and spectral levels of a channels relational graph, with teacher-derived spatial, channel, and relational attention masks weighting the losses (Wang et al., 2024). This broadens the meaning of “level” from network depth to graph-structural abstraction.

Multi-teacher MLKD often adds adaptive weighting. AMTML-KD computes instance-specific teacher weights

$$w_{t,i}=\frac{\exp(\gamma_{t,i})}{\sum_{t'}\exp(\gamma_{t',i})},$$

builds an integrated soft target

$$\tilde y_i^T=\sum_{t=1}^{m} w_{t,i}\tilde y_{t,i}^T,$$

and optimizes

$\mathcal{L}=L_{KD}+\alpha L_{Angle}+\beta L_{HT}.$

This combines high-level soft targets, angle-based structural knowledge, and multi-group hints from multiple teachers (Liu et al., 2021). CA-MKD uses a related but label-aware strategy: teacher weights are derived from cross-entropy to the ground-truth one-hot label, and separate weights are used for logit-level and feature-level transfer (Zhang et al., 2021).

4. Architectural patterns and training regimes

Most MLKD methods follow an offline teacher–student pipeline, but they vary sharply in where “levels” enter the system. A straightforward vision example is “Multi-granularity for knowledge distillation,” which introduces a frozen self-analyzed teacher, granularity-wise distillation, and a stable excitation distillation mode, while evaluating the final student without AKE and DKE at inference time (2108.06681). LFMA moves further toward self-distillation: extra heads are attached to multiple backbone blocks, their class distributions are ensembled, and the auxiliary heads are removed after training (Lin et al., 2021).

In language modeling and NLU, multiple semantic or representational levels are often distilled into a single encoder. MMKD starts from English BERT as teacher and mBERT as student, and distills token-, word-, sentence-, and structure-level knowledge from parallel corpora (Li et al., 2022). MIDAS uses three teachers specialized for sentence-level intent, word-level slot filling, and conversation-level domain classification; the student is trained with KD, similarity, and relation losses that integrate these semantic levels (Li et al., 2024). MLKD-BERT stays within the teacher–student BERT compression setting but makes the attention-head count of the student flexible by distilling MHA-output relations rather than head-aligned attention maps (Zhang et al., 2024).

Speech and audio papers frequently treat noisy or temporally varying inputs as an additional axis of level structure. In noisy SER, the teacher is wav2vec 2.0 trained on clean speech, the student is distil wav2vec 2.0 trained on noisy speech, and the student is supervised by six teacher hidden layers plus output-level KL and cross-entropy:

$^2$0

with $^2$1 and $^2$2 (Liu et al., 2023). IML-KD for speaker verification derives Integrated Inputs from Integrated Gradients over speech segments of different durations and combines instance-, class-, and batch-level alignment in

$^2$3

Its “multi-level” refers simultaneously to multiple temporal contexts and multiple alignment levels (Yang et al., 2024).

Biomedical and continual-learning settings often reinterpret “levels” as input hierarchy or teacher chronology. LRH-Net uses a 12-lead teacher, a reduced-lead teacher, and a reduced-lead student with BCE plus weighted KL distillation terms, in sequential and parallel variants (Chauhan et al., 2022). M$^2$4KD retains all previous task-end models in class-incremental learning, uses them as teachers for their own historical class blocks, and adds auxiliary intermediate-output distillation (Zhou et al., 2019). The 2025 CIR method extends this further by distilling features and correlation-based logits from multiple previous models on external unlabeled data (Kim et al., 18 Aug 2025).

5. Empirical behavior across tasks

The empirical literature shows that MLKD is not confined to a single domain. In image classification, “Multi-granularity for knowledge distillation” reports that its mechanism improves accuracy by 0.58% on average and by 1.08% in the best over the baselines (2108.06681). LFMA reports substantial self-distillation gains on ResNet18, including CIFAR100: Baseline 73.08%, LFMA 79.71%, CUB200: Baseline 51.72%, LFMA 59.65%, Stanford40: Baseline 42.97%, LFMA 49.21%, and ImageNet: Baseline 69.75%, LFMA 70.84% (Lin et al., 2021).

Graph-based MLKD shows that relation and topology transfer can be competitive at both classification and detection scales. On CIFAR-100, examples include WRN-40-2 $^2$5 WRN-40-1: student 71.92, SimKD 75.56, ours 75.99 and ResNet-32x4 $^2$6 ResNet-8x4: student 73.09, SimKD 78.08, ours 78.89. On MS-COCO, the main Faster R-CNN-FPN result with ResNet101 $^2$7 ResNet50 is teacher 42.0 AP, student 37.4, ours 41.9 (Wang et al., 2024). Camera–radar IMKD extends the idea to multimodal 3D detection; on nuScenes test it reports 67.0% NDS and 61.0% mAP (Mishra et al., 17 Dec 2025).

In language tasks, MMKD reports 75.4 average accuracy on XNLI and 70.1 average F1 on XQuAD, with pronounced gains on Bulgarian and Hindi relative to mBERT (Li et al., 2022). MLKD-BERT reports 75.6 average on GLUE for its 4-layer student and 79.1 for its 6-layer student, while also showing that reducing student attention heads from 12 to 6 or 3 decreases MNLI runtime by 9.13% or 14.11% with relatively small accuracy loss (Zhang et al., 2024).

In noisy speech emotion recognition, the proposed MLKD reports an average absolute improvement of 18.23% UA over the baseline across five noise types, with especially large gains at low SNRs (Liu et al., 2023). In speaker verification, IML-KD reports 3.02 EER for ECAPA-C128 and 3.51 EER for ResNet18, improving over the corresponding students and prior KD baselines on VoxCeleb1 (Yang et al., 2024). In low-resource ECG classification, LRH-Net reports that 3-lead sequential MLKD reaches 60.39 challenge metric score versus 58.49 for the non-distilled LRH-Net, and 2-lead parallel MLKD reaches 58.43 versus 56.66 (Chauhan et al., 2022).

Continual-learning results likewise point to multi-level transfer as a retention mechanism. In the CIR setting, adding MLKD to the baseline raises final accuracy from 19.54 to 39.82, and the full system with dynamic self-supervised learning reaches 42.00 (Kim et al., 18 Aug 2025). In M$^2$8KD, the benefit is described as growing as the number of incremental steps increases, because conventional sequential KD distills only from the last model whereas M$^2$9KD directly leverages all previous model snapshots (Zhou et al., 2019).

6. Misconceptions, limitations, and open issues

A common misconception is that MLKD always means many-layer feature matching. The surveyed papers do not support such a narrow definition. Some methods are explicitly two-level, such as prediction-level plus one intermediate feature level in CA-MKD (Zhang et al., 2021). Some are semantic rather than architectural, such as slot/intent/domain distillation in MIDAS (Li et al., 2024). Some are chronological, such as sequential and parallel multi-teacher transfer in LRH-Net (Chauhan et al., 2022). One paper even uses the acronym MLKD for Mutual-Learning Knowledge Distillation in nighttime UAV tracking rather than for multi-level knowledge distillation, while still combining teacher-to-student and student-to-student transfer paths (Liu, 2023).

A second misconception is that any “MKD” paper is automatically an MLKD paper. “MKD: a Multi-Task Knowledge Distillation Approach for Pretrained LLMs” is explicitly adjacent rather than direct: it is a multi-task, output-level, architecture-agnostic framework and “not a multi-level distillation method” in the usual sense (Liu et al., 2019).

The literature also exposes several recurrent limitations. Many MLKD systems add training-time complexity through extra heads, projection layers, graph construction, multiple teachers, or staged optimization, even when they keep inference-time cost unchanged by removing auxiliary modules after training (Lin et al., 2021, Wang et al., 2024). Some formulations remain underspecified in important implementation details: the available content for “Multi-granularity for knowledge distillation” lacks exact equations and architectural specifics (2108.06681); LFMA does not fully specify the architecture of each extra layer or the loss weights $$\mathcal{L}_{M}=\alpha \mathcal{L}_{V}+\beta \mathcal{L}_{E}+\gamma \mathcal{L}_{S},$$0 (Lin et al., 2021); the graph-based paper has notation inconsistencies in its edge-loss and degree-matrix formulas (Wang et al., 2024); IML-KD does not report the actual Integrated Gradients step number $$\mathcal{L}_{M}=\alpha \mathcal{L}_{V}+\beta \mathcal{L}_{E}+\gamma \mathcal{L}_{S},$$1 (Yang et al., 2024); and the CIR paper does not print explicit formulas for feature KD or EMA updates (Kim et al., 18 Aug 2025).

Across these papers, a plausible synthesis is that MLKD is best understood not as a single algorithm but as a design principle: the student should inherit teacher information at more than one level of abstraction, relation, or semantic scope. What changes from paper to paper is the ontology of those levels—layers, branches, graphs, samples, teachers, tasks, or dialogue scopes—and the choice of which combinations most effectively preserve teacher behavior without erasing the student’s own architectural constraints.