Multi-level Consistent Knowledge Distillation

• Multi-level consistent knowledge distillation is a framework where the student network aligns with teacher outputs across architectural, temporal, and relational levels rather than only final predictions.
• It employs consistency constraints such as KL divergence, cross-entropy, and feature regression to fuse knowledge from intermediate, historical, and multi-teacher sources.
• Empirical studies in image classification, language modeling, and object detection show improved student performance and robustness with efficient resource usage.

Searching arXiv for directly relevant papers on multi-level consistent knowledge distillation and closely related formulations. Multi-level consistent knowledge distillation denotes a family of distillation procedures in which teacher and student are required to agree across more than one representational, structural, or temporal level, rather than only at the final output distribution. In the literature, the relevant “levels” vary by setting: they may be shallow, intermediate, and deep network features; historical model snapshots in incremental learning; multiple teachers with sample-wise reliability; multiple chains of thought for the same question; query hierarchies in sparse 3D perception; or task-semantic strata such as intent, slots, and domain in dialogue NLU. A central commonality is the use of explicit consistency constraints that bind these levels into a shared supervisory signal, typically through KL divergence, cross-entropy, feature regression, relation matching, or combinations thereof (Lin et al., 2021, Zhou et al., 2019, Zhang et al., 2021, Li et al., 2 Feb 2026).

1. Scope and meanings of “multi-level”

Across the literature, the term does not refer to a single canonical algorithm. In self-distillation for image classification, it often means consistency across network depth: early, middle, and deep blocks all contribute predictive knowledge, and the final classifier is regularized toward a consensus teacher formed from those levels (Lin et al., 2021). In incremental learning, it can mean consistency both across time and across depth, so that the current model is aligned not only with an immediately previous teacher but with all historical models and with an auxiliary intermediate classifier (Zhou et al., 2019). In multi-teacher classification, it can denote simultaneous distillation at output and intermediate-feature levels, with teacher importance changing per sample (Zhang et al., 2021, Liu et al., 2021).

Other works redefine the levels more structurally. “Multi-level Knowledge Distillation via Knowledge Alignment and Correlation” separates instance-level alignment from relation-level correlation and treats both as complementary forms of transferred knowledge (Ding et al., 2020). “Knowledge consistent distillation” focuses on channel-level consistency, arguing that teacher and student often encode similar concepts in different channel bases and that teacher features should therefore be transformed into a student-consistent basis before feature matching (Han et al., 2021). In pre-trained LLMs, levels include embeddings, MHA outputs, FFN hidden states, sample-sample relations, and logits (Zhang et al., 2024). In dialogue NLU, levels are word-level slots, sentence-level intents, and conversation-level domains (Li et al., 2024).

Axis of “level”	Representative formulation	Core consistency target
Network depth	LFMA, MLKD-BERT	Blocks, layers, or splits agree with mapped teacher states
Teacher multiplicity	CA-MKD, AMTML-KD	Multiple teachers are fused or weighted per instance
Time or history	M2KD, IML-KD	Historical snapshots or temporal input trajectories stay aligned
Relation structure	MLKD, MLKD-BERT	Teacher-student sample or token geometry is preserved
Task-semantic hierarchy	MIDAS	Intent, slot, and domain knowledge are distilled jointly

This variety suggests that multi-level consistent KD is best understood as a design principle: supervision is distributed across several coherent levels, and the student is trained to remain compatible with all of them.

2. Core objective patterns and consistency operators

A recurrent formulation is a supervised objective plus one or more consistency regularizers. In LFMA, the overall loss is

$$Loss = \alpha Loss_{1} + \beta Loss_{2} + \gamma Loss_{3} + \delta Loss_{4},$$

where $Loss_1$ is KLD between the backbone output and an ensemble of multilevel block predictors, $Loss_2$ is summed KLD between that ensemble and the extra-layer outputs, $Loss_3$ is cross-entropy at the final output, and $Loss_4$ is deep supervision at all extra layers (Lin et al., 2021). In M$^2$KD, the total objective is

$L_{total} = L_{MMD} + \lambda L_{AD},$

with $L_{MMD}$ distilling from all historical model snapshots and $L_{AD}$ distilling an auxiliary classifier attached to an intermediate residual block (Zhou et al., 2019). In CA-MKD, the full objective is

$$\mathcal L = \mathcal L_{CE} + \alpha \mathcal L_{KD} + \beta \mathcal L_{inter},$$

where both logit-level and feature-level terms are weighted by sample-wise teacher reliability computed from ground-truth consistency (Zhang et al., 2021). In representation-centric MLKD, the basic form is

$Loss_1$0

optionally augmented by supervised contrastive and cross-entropy terms (Ding et al., 2020). In reasoning KD, MCC-KD uses

$Loss_1$1

where $Loss_1$2 is a bidirectional KL divergence between answer distributions induced by different chains of thought for the same question (Chen et al., 2023).

The operators themselves vary with the level being aligned. KL divergence remains the dominant choice for softened class distributions and rationale-conditioned answer distributions (Lin et al., 2021, Zhang et al., 2021, Chen et al., 2023). $Loss_1$3, $Loss_1$4, and MSE are common for feature or coordinate alignment, as in CA-MKD’s last-block feature matching, DiScene’s encoder/query/prior/anchor supervision, and speech-emotion MLKD’s layerwise matching between teacher and student wav2vec states (Zhang et al., 2021, Li et al., 2 Feb 2026, Liu et al., 2023). Cosine similarity and triplet-style relation losses appear when the target is not an absolute representation but a geometry over samples or channels (Han et al., 2021, Li et al., 2024). This suggests a stable division of labor: distributional operators are usually reserved for semantic outputs, while regression or relational operators are used for internal structure.

3. Vision and continual-learning formulations

LFMA is a prototypical within-network formulation. A backbone is decomposed into blocks $Loss_1$5; each block feature $Loss_1$6 is passed through an Extra Layer $Loss_1$7 to produce a class distribution $Loss_1$8, and the distributions are averaged into an ensemble teacher $Loss_1$9, with uniform $Loss_2$0 in experiments (Lin et al., 2021). Consistency is enforced between the final classifier and $Loss_2$1, between each block head and $Loss_2$2, and between all outputs and the ground truth. The method is trained end-to-end, but the Extra Layers are removed at inference, so there is no additional test-time parameter or latency cost (Lin et al., 2021).

M$Loss_2$3KD generalizes consistency along a temporal axis. In single-head class-incremental learning, the paper argues that distilling only from the last model propagates accumulated forgetting. Its remedy is multi-model distillation from all previous snapshots, with each old class segment supervised by the historical model that first learned it, plus auxiliary distillation from an intermediate classifier attached to the second residual block (Zhou et al., 2019). Historical models are reconstructed on-the-fly from a shared backbone using mask-based pruning, which yields 9.80 MB additional memory on iILSVRC-small and 0.84 MB on CIFAR-100, compared with 68.0 MB and 9.4 MB for iCaRL (Zhou et al., 2019). Here “consistency” means both temporal anchoring and depth-wise stabilization.

Confidence-aware and adaptive multi-teacher methods target a different source of inconsistency: conflict among teachers. CA-MKD computes per-sample teacher weights from the cross-entropy between each teacher’s prediction and the one-hot label, then uses separate reliability weights for logits and last-block feature matching (Zhang et al., 2021). AMTML-KD instead learns instance-level teacher importance through latent teacher representations and a student-dependent scoring function, combines the resulting integrated soft targets with angle-based relational KD, and assigns different teachers to different student groups through a multi-group hint strategy (Liu et al., 2021). In both cases, equal-weight teacher averaging is treated as inadequate.

Several papers focus on representation-space compatibility itself. KCD introduces a channel-wise consistency matrix and reports that teacher and student often activate different channels even for the same class; it therefore transforms teacher features through bipartite channel matching before applying feature-based KD, and reports that correlation-based consistency with bipartite matching performs best (Han et al., 2021). MLKD via knowledge alignment and correlation argues that standard KD transfers only per-sample alignment, whereas relation-based methods transfer sample correlation; its objective combines both and is explicitly task-agnostic and model-agnostic (Ding et al., 2020). MetaMixer, in online KD, combines CutMix-like local mixing with feature-space global mixing so that low-level localizable knowledge and high-level semantic knowledge regularize each other in peer distillation (Wang et al., 2023).

Task-specific structural variants follow the same pattern. CDFKD-MFS introduces a multi-header student, feature sharing across multiple backbone depths, and per-teacher header, ensemble, and feature losses in a data-free multi-teacher setting (Hao et al., 2022). LRH-Net uses two-step or parallel teacher hierarchies to distill 12-lead ECG knowledge into 3-lead or 2-lead low-resource students, treating lead configuration itself as a level in the distillation hierarchy (Chauhan et al., 2022).

4. Language, reasoning, and dialogue

In large-language-model reasoning, MCC-KD reinterprets levels as multiple chains of thought and answer-token distributions. For each question, multiple rationales are generated and filtered for diversity; the student is trained with cross-entropy on rationale-plus-answer sequences and a bidirectional KL divergence between answer distributions induced by two different rationales for the same question (Chen et al., 2023). The method uses $Loss_2$4 rationales in the main experiments, and the paper reports that performance improves substantially from $Loss_2$5 to $Loss_2$6, with smaller gains thereafter (Chen et al., 2023). Here consistency is view-invariance at the answer level despite rationale diversity.

MLKD-BERT is more explicitly layered. Stage 1 distills embedding-layer token similarities, MHA self-attention relations, and FFN hidden states; Stage 2 distills sample-sample similarity, supervised contrastive relations, and softened logits (Zhang et al., 2024). A notable architectural contribution is the use of MHA-splits, which allow the student to use fewer attention heads than the teacher while still matching teacher-defined relation structures. The paper reports that $Loss_2$7 reaches an average GLUE score of 75.6 with 14.5M parameters and 9.4× speedup, retaining about 95.1% of teacher performance, while $Loss_2$8 reaches 79.1 and about 99.5% of teacher performance (Zhang et al., 2024).

MIDAS applies multi-teacher KD to multi-turn NLU with three specialist teachers: intent detection, slot filling, and domain classification (Li et al., 2024). Its loss combines averaged multi-teacher KL supervision, standard student cross-entropy, cosine-similarity-based teacher-student logit alignment, relation KD through triplet constraints on hidden states, and optional teacher-prediction supervision (Li et al., 2024). Unlike single-teacher joint NLU KD, the levels here are semantic strata rather than network layers. A plausible implication is that dialogue-level consistency can be enforced without forcing one teacher to encode all granularities equally well.

5. Audio, detection, medical signals, and 3D perception

In noisy speech emotion recognition, multi-level KD is instantiated as clean-teacher/noisy-student supervision between wav2vec 2.0 and distil wav2vec 2.0. The teacher’s layers $Loss_2$9 supervise the student’s six layers through an MSE loss on hidden states, combined with KL on softened logits and cross-entropy on noisy labels (Liu et al., 2023). On factory noise, the student with MLKD improves over the student without supervision by +6.10 at 0 dB, +4.42 at 5 dB, +3.37 at 10 dB, +1.32 at 15 dB, +1.57 at 20 dB, and +1.68 on clean speech, and the student model is 197.91 MB versus 360.17 MB for the teacher (Liu et al., 2023).

Integrated Multi-level KD for speaker verification defines levels differently again: instance-level, class-level, and batch-level relations are combined with Integrated Gradient-based input-sensitive representations built from speech segments of various durations (Yang et al., 2024). The total objective is

$Loss_3$0

and the paper reports that, for ECAPA teacher and ECAPA-C128 student, EER drops from 3.59 to 3.02, while for ResNet34 teacher and ResNet18 student it drops from 3.76 to 3.51 (Yang et al., 2024).

In detection transformers, KD-DETR identifies the core problem as the absence of consistent distillation points. It therefore introduces a separate set of shared distillation queries, split into random general queries and teacher-derived specific queries, and applies weighted KL, $Loss_3$1, and GIoU distillation to teacher and student outputs evaluated at exactly those same queries (Wang et al., 2022). On DAB-DETR with ResNet-50 teacher and ResNet-18 student, AP increases from 36.2 to 41.4; in compressed decoder settings, gains are even larger, such as 31.8 to 38.9 for a 6/2 configuration (Wang et al., 2022).

DiScene extends the same concern to sparse 3D occupancy prediction. Its Multi-level Consistent Knowledge Distillation comprises encoder-level feature alignment, query-level feature matching, prior-level spatial guidance, and anchor-level high-confidence transfer, combined as

$Loss_3$2

with $Loss_3$3, $Loss_3$4, $Loss_3$5, and $Loss_3$6 (Li et al., 2 Feb 2026). Teacher-Guided Initialization complements these losses by initializing the student decoder from the teacher. On Occ-ScanNet, DiScene achieves 23.2 FPS without depth priors and improves OPUS by 36.1%; with depth, DiScene† raises mIoU from 38.96 to 47.17 and exceeds EmbodiedOcc by 3.7 with 1.62× faster inference speed (Li et al., 2 Feb 2026).

6. Empirical patterns, misconceptions, and open issues

A first recurring empirical pattern is that multi-level consistency usually outperforms output-only KD. LFMA raises ResNet-18 on CIFAR-100 from 73.08% to 79.71% and on ImageNet from 69.75% to 70.84% (Lin et al., 2021). CA-MKD improves ShuffleNetV1 from 71.70 to 77.94 under WRN40-2 teachers and reports an average improvement of 0.81% over EBKD across seven teacher-student pairs (Zhang et al., 2021). MCC-KD raises LLaMA-7B on GSM8K from 38.01 to 41.58 and improves several out-of-distribution benchmarks as well (Chen et al., 2023). These results indicate that consistency terms become most useful when they constrain behavior unavailable from hard labels alone.

A second pattern is that “multi-level” should not be reduced to “multi-layer feature matching.” In the surveyed papers, levels may correspond to class and batch relations (Ding et al., 2020), intent/slot/domain semantics (Li et al., 2024), historical checkpoints (Zhou et al., 2019), rationale-conditioned answer distributions (Chen et al., 2023), or query hierarchies (Li et al., 2 Feb 2026). A common misconception is therefore that multi-level consistent KD is synonymous with attaching auxiliary heads at several CNN stages; that is only one specialization.

A third pattern is that more detail is not automatically better. DiScene reports that coarse feature-based distillation outperforms fine logit-based distillation at the query, prior, and anchor levels (Li et al., 2 Feb 2026). MLKD-BERT shows that relation-level MHA supervision and feature-level FFN supervision are complementary rather than interchangeable, and that a one-stage optimization of all losses underperforms its two-stage schedule by 2.3 average points on the reported ablation (Zhang et al., 2024). KCD similarly argues that naïve same-index feature matching is suboptimal when channel semantics are misaligned (Han et al., 2021).

A fourth pattern is that multi-teacher consistency requires aggregation, not mere accumulation. CA-MKD uses label-aware reliability weights; AMTML-KD learns instance-level teacher importance; MIDAS averages teacher probabilities and also uses similarity and relation losses to reconcile specialist teachers (Zhang et al., 2021, Liu et al., 2021, Li et al., 2024). This suggests that multi-teacher multi-level KD is fundamentally an aggregation problem: the supervisory signal must itself be internally coherent.

The main limitations are also recurrent. Many papers explicitly report higher training-time cost from auxiliary heads, multiple teacher forward passes, model reconstruction, or Integrated Inputs (Lin et al., 2021, Zhou et al., 2019, Yang et al., 2024). Hyperparameters such as $Loss_3$7, temperatures, pruning ratios, split counts, or segment durations require tuning (Zhou et al., 2019, Zhang et al., 2021, Zhang et al., 2024). Architecture dependence remains significant: some methods are natural for staged CNNs or transformer blocks, but require nontrivial choices for arbitrary backbones (Lin et al., 2021, Liu et al., 2023). Future directions stated or implied in the papers include adaptive level weighting, richer teacher-teacher consistency, more explicit feature or attention alignment, broader use in detection and segmentation, and stronger handling of robustness and out-of-distribution transfer (Lin et al., 2021, Zhang et al., 2021, Li et al., 2 Feb 2026).

Taken together, the literature presents multi-level consistent knowledge distillation not as a single recipe, but as a general strategy for constraining a student with several mutually informative views of teacher knowledge. The levels may be architectural, temporal, relational, or semantic; the essential requirement is that they are coupled by explicit consistency objectives so that coarse and fine knowledge, local and global structure, or old and new information do not drift apart during training.