---
title: Multi-Level Knowledge Distillation
url: https://www.emergentmind.com/topics/multi-level-knowledge-distillation-mlkd
type: topic
---

# Multi-Level Knowledge Distillation

Multi-Level Knowledge Distillation (MLKD) denotes a family of distillation methods in which the student is supervised by more than one level, perspective, or semantic scope of teacher knowledge during training. In the literature surveyed here, the relevant “levels” include final predictions and intermediate features, alignment of individual samples and correlation across samples, token-, word-, sentence-, and structure-level representations, and word-, utterance-, and conversation-level semantics in multi-turn dialogue. The term is therefore broader than a single architectural recipe: in different papers it can refer to multi-layer supervision, multi-granularity teaching, relation-level transfer, or multi-stage teacher hierarchies [2002.03532][2012.00573][2211.01200][2408.08144].

## 1. Conceptual scope and definitions

A central conceptual foundation for MLKD is the claim that teacher knowledge is hierarchically structured. “Understanding and Improving Knowledge Distillation” separates teacher knowledge into three hierarchical levels: “knowledge of the `universe’,” “domain knowledge,” and “instance specific knowledge,” and argues that these three factors “play a major role in knowledge distillation” [2002.03532]. In that account, universe-level knowledge behaves as regularization through label smoothing, domain-level knowledge injects class relationships into the student’s logit geometry, and instance-specific knowledge rescales per-example gradients.

A second foundation is the distinction between **knowledge alignment** and **knowledge correlation**. “Multi-level Knowledge Distillation via Knowledge Alignment and Correlation” argues that traditional KD mainly transfers knowledge alignment of an individual sample, whereas contrastive-learning-based KD methods mainly transfer relational knowledge between different samples, namely knowledge correlation. MLKD is introduced there precisely to combine both, with the stated aim of transferring “the full knowledge from teacher to student” and improving “the reliability and transferability of learned representations” [2012.00573].

The surveyed literature also shows that the phrase *multi-level* is not used uniformly. In incremental learning, M\(^2\)KD uses “multi-level” for supervision at the main final-output level and an auxiliary intermediate-output branch derived from intermediate features [1904.01769]. In multilingual pre-training, MMKD uses token-, word-, sentence-, and structure-level alignment objectives [2211.01200]. In multi-turn NLU, MIDAS uses “multi-level” in a semantic sense: word-level slot filling, sentence-level intent detection, and conversation-level domain classification [2408.08144]. In reduced-lead ECG classification, LRH-Net uses “multi-level” for a multi-step, multi-teacher distillation hierarchy rather than feature-level matching [2204.08000].

## 2. Principal formulations of “level”

The most common formulation of MLKD is **output-level plus intermediate-level distillation**. AMTML-KD explicitly divides teacher information into “high-level knowledge,” namely integrated soft targets, and “intermediate-level knowledge,” namely hidden-layer hints from multiple teachers [2103.04062]. CA-MKD similarly combines prediction-level transfer and one intermediate feature level, using confidence-aware teacher weights computed separately for each level [2201.00007]. M\(^2\)KD applies output-level distillation on the main branch and auxiliary-output distillation on features extracted from the second residual block [1904.01769].

A second formulation is **alignment plus relation or structure distillation**. In MLKD via knowledge alignment and correlation, the two levels are per-sample alignment and cross-sample correlation [2012.00573]. In graph-based feature distillation, the levels are vertex, edge, and spectral structure within a channels relational graph, with losses
$$
\mathcal{L}_{M}=\alpha \mathcal{L}_{V}+\beta \mathcal{L}_{E}+\gamma \mathcal{L}_{S},
$$
followed by an overall task loss [2405.08547]. In MLKD-BERT, the levels are embedding layer, Transformer layer, and prediction layer, and each of those levels mixes feature-level and relation-level objectives [2407.02775].

A third formulation is **semantic granularity**. MMKD aligns token, word, sentence, and structure representations in multilingual learning [2211.01200]. MIDAS distills slot-, intent-, and domain-specialized teachers into a single student for multi-turn dialogue understanding [2408.08144]. In self-distillation, LFMA builds auxiliary predictors on shallow, middle, and deep backbone blocks and ensembles their distributions into a self-teacher [2112.13642].

A fourth formulation is **multi-stage or multi-teacher hierarchy**. LRH-Net defines sequential MLKD and parallel MLKD by combining a 12-lead teacher, a reduced-lead teacher, and a low-resource reduced-lead student with weighted KL terms plus BCE [2204.08000]. Continual-learning MLKD uses multiple previous models and distills both feature-level and logit-level knowledge on external unlabeled data [2508.12692].

| Formulation of “level” | Representative papers | Distilled knowledge |
|---|---|---|
| Output + intermediate features | [1904.01769], [2103.04062], [2201.00007] | Logits, auxiliary outputs, hints |
| Alignment + correlation | [2012.00573], [2405.08547] | Per-sample alignment, sample relations, graph structure |
| Semantic granularity | [2211.01200], [2408.08144] | Token/word/sentence/structure or slot/intent/domain |
| Multi-stage teacher hierarchy | [2204.08000], [2508.12692] | Multi-teacher outputs, features, logits |

This variation suggests that MLKD is best treated as a research family rather than a single canonical loss.

## 3. Objectives and mechanisms

At the alignment level, a standard MLKD formulation is direct feature matching after a learned transformation:
$$
\mathcal{L}_{\mathrm{Align}}=\mathbb{E}_{x}\left[\left\|h_\varphi(f_\theta^S(x))-f_\eta^T(x)\right\|_2^2\right].
$$
This is the explicit alignment loss used in “Multi-level Knowledge Distillation via Knowledge Alignment and Correlation” [2012.00573]. That same paper adds a correlation objective so that the student preserves how samples are arranged relative to one another, not only the teacher representation of each sample in isolation.

A different but related design appears in MMKD, where the total multilingual objective is
$$
\mathcal{L}=\mathcal{L}_{\mathrm{TLM}}+\mathcal{L}_{\mathrm{XWCL}}+\mathcal{L}_{\mathrm{SentA}}+\alpha \mathcal{L}_{\mathrm{StrucA}}.
$$
Here the levels are explicitly token, word, sentence, and structure. The structure term uses batchwise similarity matrices and KL divergence, so relation-level knowledge is treated as a first-class distillation target rather than an auxiliary regularizer [2211.01200].

MLKD-BERT provides a particularly explicit decomposition by model stage. Its first-stage objective is
$$
\mathcal{L}_{\mathrm{Stage\ 1}}=\mathcal{L}_{\mathrm{EMB}}+\mathcal{L}_{\mathrm{MHA}}+\mathcal{L}_{\mathrm{FFN}},
$$
and its second-stage objective is
$$
\mathcal{L}_{\mathrm{Stage\ 2}}=\mathcal{L}_{\mathrm{SS}}+\mathcal{L}_{\mathrm{SC}}+\mathcal{L}_{\mathrm{KD}}.
$$
The paper’s distinctive step is to distill **self-attention relation** from MHA outputs rather than raw attention distributions, which allows the student to use fewer attention heads than the teacher [2407.02775].

Graph-based MLKD shifts the unit of transfer from tensor values to graph structure. In “Exploring Graph-based Knowledge,” the teacher and student are aligned at vertex, edge, and spectral levels of a channels relational graph, with teacher-derived spatial, channel, and relational attention masks weighting the losses [2405.08547]. This broadens the meaning of “level” from network depth to graph-structural abstraction.

Multi-teacher MLKD often adds adaptive weighting. AMTML-KD computes instance-specific teacher weights
$$
w_{t,i}=\frac{\exp(\gamma_{t,i})}{\sum_{t'}\exp(\gamma_{t',i})},
$$
builds an integrated soft target
$$
\tilde y_i^T=\sum_{t=1}^{m} w_{t,i}\tilde y_{t,i}^T,
$$
and optimizes
$$
\mathcal{L}=L_{KD}+\alpha L_{Angle}+\beta L_{HT}.
$$
This combines high-level soft targets, angle-based structural knowledge, and multi-group hints from multiple teachers [2103.04062]. CA-MKD uses a related but label-aware strategy: teacher weights are derived from cross-entropy to the ground-truth one-hot label, and separate weights are used for logit-level and feature-level transfer [2201.00007].

## 4. Architectural patterns and training regimes

Most MLKD methods follow an offline teacher–student pipeline, but they vary sharply in where “levels” enter the system. A straightforward vision example is “Multi-granularity for knowledge distillation,” which introduces a frozen self-analyzed teacher, granularity-wise distillation, and a stable excitation distillation mode, while evaluating the final student without AKE and DKE at inference time [2108.06681]. LFMA moves further toward self-distillation: extra heads are attached to multiple backbone blocks, their class distributions are ensembled, and the auxiliary heads are removed after training [2112.13642].

In language modeling and NLU, multiple semantic or representational levels are often distilled into a single encoder. MMKD starts from English BERT as teacher and mBERT as student, and distills token-, word-, sentence-, and structure-level knowledge from parallel corpora [2211.01200]. MIDAS uses three teachers specialized for sentence-level intent, word-level slot filling, and conversation-level domain classification; the student is trained with KD, similarity, and relation losses that integrate these semantic levels [2408.08144]. MLKD-BERT stays within the teacher–student BERT compression setting but makes the attention-head count of the student flexible by distilling MHA-output relations rather than head-aligned attention maps [2407.02775].

Speech and audio papers frequently treat noisy or temporally varying inputs as an additional axis of level structure. In noisy SER, the teacher is wav2vec 2.0 trained on clean speech, the student is distil wav2vec 2.0 trained on noisy speech, and the student is supervised by six teacher hidden layers plus output-level KL and cross-entropy:
$$
L=\alpha L_{\text{KL}}+(1-\alpha)L_C+L_{\text{MSE}},
$$
with \(T=7\) and \(\alpha=0.7\) [2312.13556]. IML-KD for speaker verification derives Integrated Inputs from Integrated Gradients over speech segments of different durations and combines instance-, class-, and batch-level alignment in
$$
L_{ML}=L_{batch}+L_{class}+L_{inst}.
$$
Its “multi-level” refers simultaneously to multiple temporal contexts and multiple alignment levels [2409.09389].

Biomedical and continual-learning settings often reinterpret “levels” as input hierarchy or teacher chronology. LRH-Net uses a 12-lead teacher, a reduced-lead teacher, and a reduced-lead student with BCE plus weighted KL distillation terms, in sequential and parallel variants [2204.08000]. M\(^2\)KD retains all previous task-end models in class-incremental learning, uses them as teachers for their own historical class blocks, and adds auxiliary intermediate-output distillation [1904.01769]. The 2025 CIR method extends this further by distilling features and correlation-based logits from multiple previous models on external unlabeled data [2508.12692].

## 5. Empirical behavior across tasks

The empirical literature shows that MLKD is not confined to a single domain. In image classification, “Multi-granularity for knowledge distillation” reports that its mechanism improves accuracy by **0.58% on average** and by **1.08% in the best** over the baselines [2108.06681]. LFMA reports substantial self-distillation gains on ResNet18, including **CIFAR100: Baseline 73.08%, LFMA 79.71%**, **CUB200: Baseline 51.72%, LFMA 59.65%**, **Stanford40: Baseline 42.97%, LFMA 49.21%**, and **ImageNet: Baseline 69.75%, LFMA 70.84%** [2112.13642].

Graph-based MLKD shows that relation and topology transfer can be competitive at both classification and detection scales. On CIFAR-100, examples include **WRN-40-2 \(\rightarrow\) WRN-40-1: student 71.92, SimKD 75.56, ours 75.99** and **ResNet-32x4 \(\rightarrow\) ResNet-8x4: student 73.09, SimKD 78.08, ours 78.89**. On MS-COCO, the main **Faster R-CNN-FPN** result with **ResNet101 \(\rightarrow\) ResNet50** is **teacher 42.0 AP, student 37.4, ours 41.9** [2405.08547]. Camera–radar IMKD extends the idea to multimodal 3D detection; on nuScenes test it reports **67.0% NDS and 61.0% mAP** [2512.15581].

In language tasks, MMKD reports **75.4** average accuracy on XNLI and **70.1** average F1 on XQuAD, with pronounced gains on Bulgarian and Hindi relative to mBERT [2211.01200]. MLKD-BERT reports **75.6** average on GLUE for its 4-layer student and **79.1** for its 6-layer student, while also showing that reducing student attention heads from 12 to 6 or 3 decreases MNLI runtime by **9.13%** or **14.11%** with relatively small accuracy loss [2407.02775].

In noisy speech emotion recognition, the proposed MLKD reports an average absolute improvement of **18.23% UA** over the baseline across five noise types, with especially large gains at low SNRs [2312.13556]. In speaker verification, IML-KD reports **3.02 EER** for ECAPA-C128 and **3.51 EER** for ResNet18, improving over the corresponding students and prior KD baselines on VoxCeleb1 [2409.09389]. In low-resource ECG classification, LRH-Net reports that 3-lead sequential MLKD reaches **60.39** challenge metric score versus **58.49** for the non-distilled LRH-Net, and 2-lead parallel MLKD reaches **58.43** versus **56.66** [2204.08000].

Continual-learning results likewise point to multi-level transfer as a retention mechanism. In the CIR setting, adding MLKD to the baseline raises final accuracy from **19.54** to **39.82**, and the full system with dynamic self-supervised learning reaches **42.00** [2508.12692]. In M\(^2\)KD, the benefit is described as growing as the number of incremental steps increases, because conventional sequential KD distills only from the last model whereas M\(^2\)KD directly leverages all previous model snapshots [1904.01769].

## 6. Misconceptions, limitations, and open issues

A common misconception is that MLKD always means **many-layer feature matching**. The surveyed papers do not support such a narrow definition. Some methods are explicitly two-level, such as prediction-level plus one intermediate feature level in CA-MKD [2201.00007]. Some are semantic rather than architectural, such as slot/intent/domain distillation in MIDAS [2408.08144]. Some are chronological, such as sequential and parallel multi-teacher transfer in LRH-Net [2204.08000]. One paper even uses the acronym **MLKD** for **Mutual-Learning Knowledge Distillation** in nighttime UAV tracking rather than for multi-level knowledge distillation, while still combining teacher-to-student and student-to-student transfer paths [2312.07884].

A second misconception is that any “MKD” paper is automatically an MLKD paper. “MKD: a Multi-Task Knowledge Distillation Approach for Pretrained Language Models” is explicitly adjacent rather than direct: it is a multi-task, output-level, architecture-agnostic framework and “not a multi-level distillation method” in the usual sense [1911.03588].

The literature also exposes several recurrent limitations. Many MLKD systems add training-time complexity through extra heads, projection layers, graph construction, multiple teachers, or staged optimization, even when they keep inference-time cost unchanged by removing auxiliary modules after training [2112.13642][2405.08547]. Some formulations remain underspecified in important implementation details: the available content for “Multi-granularity for knowledge distillation” lacks exact equations and architectural specifics [2108.06681]; LFMA does not fully specify the architecture of each extra layer or the loss weights \(\alpha,\beta,\gamma,\delta\) [2112.13642]; the graph-based paper has notation inconsistencies in its edge-loss and degree-matrix formulas [2405.08547]; IML-KD does not report the actual Integrated Gradients step number \(m\) [2409.09389]; and the CIR paper does not print explicit formulas for feature KD or EMA updates [2508.12692].

Across these papers, a plausible synthesis is that MLKD is best understood not as a single algorithm but as a design principle: the student should inherit teacher information at more than one level of abstraction, relation, or semantic scope. What changes from paper to paper is the ontology of those levels—layers, branches, graphs, samples, teachers, tasks, or dialogue scopes—and the choice of which combinations most effectively preserve teacher behavior without erasing the student’s own architectural constraints.

Source: https://www.emergentmind.com/topics/multi-level-knowledge-distillation-mlkd