Layer-wise Distillation Methods

• Layer-wise distillation is a technique that applies supervision at various internal layers to capture hierarchical representations for better knowledge transfer.
• It uses methods like hidden state matching, attention-based teacher aggregation, and dynamic routing to overcome alignment challenges between teacher and student models.
• This approach improves learning stability, minimizes overfitting, and enables efficient compression by preserving critical internal features beyond mere output signals.

Layer-wise distillation is a family of distillation procedures in which supervision is applied through the internal depth of a network rather than only through final logits or task outputs. Across the literature, the term encompasses several related but non-identical practices: direct matching of intermediate hidden states, attention maps, or subspaces; learned aggregation of multiple teacher layers for a single student layer; self-distillation from internally generated layer targets; progressive layer-by-layer optimization for model merging; and prune-and-recover schemes that explicitly manipulate which layers remain in the student. The common premise is that useful knowledge is often distributed across lower, middle, and upper layers, so output-only distillation can omit information that matters for transfer, generalization, compression, or efficient inference (Chang et al., 2021, Passban et al., 2020, Xu et al., 18 Feb 2025).

1. From output-level knowledge distillation to intermediate supervision

Classical knowledge distillation, as described in ALP-KD, couples teacher and student mainly at the prediction layer: the student matches the teacher’s final predictive distribution, typically together with the ordinary supervised loss. Layer-wise distillation extends that regime by supervising internal components, motivated by the observation that deep models encode hierarchical computations across depth and that a student can benefit not only from the teacher’s final predictions but also from how the teacher processes information layer by layer (Passban et al., 2020).

This rationale is explicit in speech self-supervised learning. DistilHuBERT argues that the last layer is not necessarily the richest distillation target for speech SSL, because lower layers contain more speaker-related or acoustic information, middle layers more phonetic information, and higher layers more semantic or content information. The method therefore distills multiple intermediate HuBERT layers rather than only the final layer, treating the problem as multi-task learning over teacher hidden states (Chang et al., 2021).

A more theoretical account appears in the information-theoretic analysis of knowledge points. There, a DNN is viewed as a layer-wise information-discarding system, and distillation is interpreted as helping the student preserve more task-relevant input units in intermediate layers, learn different knowledge points simultaneously, and optimize more stably than training from scratch (Zhang et al., 2022).

Taken together, these formulations establish that layer-wise distillation is not a single objective but a shift in where supervision is injected: from the model output toward the internal hierarchy of representations.

2. Layer correspondence, alignment, and teacher aggregation

A central difficulty is that teacher and student rarely have identical depth or width. Early layer-wise methods often assumed a hard alignment function that maps student layer $j$ to a single teacher layer. ALP-KD identifies two problems with that design when the teacher has $n$ layers and the student $m$ layers with $n \gg m$: the skip problem, in which many teacher layers are ignored, and the search problem, in which deciding which teacher layers to keep becomes heuristic. Its remedy is attention-based combinatorial projection, where each student layer receives a learned weighted combination of teacher layers rather than a one-to-one match (Passban et al., 2020).

Task-aware layEr-wise Distillation (TED) preserves explicit layer mapping, but it modifies what is aligned. Instead of matching raw teacher and student hidden states, TED trains task-aware filters that extract the task-relevant part of each layer’s hidden state and distills those filtered features. This is meant to reduce the knowledge gap between teacher and student and to alleviate underfitting caused by capacity mismatch or teacher redundancy (Liang et al., 2022).

Several later methods replace fixed alignment with proportional, sample-dependent, or module-dependent correspondence. In reasoning transfer from text to audio, the aligned teacher layer for each student layer is determined by proportional mapping rather than naïve indexing, and a skip-layer “1-in-$k$” regime is introduced as an intermediate option between top-layer-only KD and full layer-wise KD (Yang et al., 23 Sep 2025). In DLink for EEG foundation models, a dynamic Router computes sample-specific weights over all teacher layers and aggregates them according to structural and spectral cues, because the best teacher layer may vary by sample and task (Wang et al., 16 Apr 2026). In multimodal foundation models, OPTIMA treats subsets of the image encoder, text encoder, and multimodal decoder as bandit arms and adaptively chooses which module subset to distill more frequently, reflecting the observation that module importance is task-dependent and nonstationary (Liang et al., 2023).

Heterogeneous hidden dimensions create a second alignment problem. “Logit Distillation on Manifolds” introduces per-layer projection modules $W_i$, maps student states into teacher space, orthogonalizes both sides, and compares subspaces on the Grassmann manifold; BRL instead learns compression coefficients in a Bregman basis induced by the teacher layer’s transfer function and trains the student to predict those coefficients (Yang et al., 30 May 2026, Amid et al., 2022).

3. Supervisory signals and objective design

The losses used in layer-wise distillation are highly varied because the supervised object itself varies. DistilHuBERT distills the 4th, 8th, and 12th hidden layers of HuBERT with a sum of $\ell_1$ and cosine terms,

$$\mathcal{L} = \mathcal{L}^{(4)} + \mathcal{L}^{(8)} + \mathcal{L}^{(12)},$$

where each $\mathcal{L}^{(l)}$ combines element-wise closeness with directional alignment. The student uses three prediction heads during distillation, and the losses are summed as a multi-task objective (Chang et al., 2021).

Layer-wise objectives can also be embedded in broader task losses. DistilMOS fine-tunes a self-supervised MOS predictor with utterance-level MOS regression and, simultaneously, per-layer token classification targets obtained by clustering frame-level hidden representations from each SSL layer. The final loss combines MOS regression with the average token prediction loss over all SSL layers and uses $\alpha = 0.1$; the token prediction branch is used only during training and adds no test-time cost (Yang et al., 20 Jan 2026).

Self-distillation formulations can be fully internal. LR-Drop regularizes two dropout-sampled siamese subnetworks by matching hidden states, multi-head attention matrices, and final output distributions,

$n$0

so that each stochastic subnetwork acts as the other’s teacher and student (Ni et al., 2024).

Other objectives are explicitly geometry-aware or spectrum-aware. LEAP augments standard distillation with an exit-aware loss that pushes intermediate layers toward the final representation so that convergence-based early exit remains usable (Kapadia et al., 1 May 2026). DLink aligns teacher and student in the frequency domain, using magnitude and encoded phase spectra in a Router-weighted layer-wise spectral loss and a PSD-based routing loss (Wang et al., 16 Apr 2026). RC combines KL distillation, cross-entropy, and a geometric subspace loss, $n$1, reflecting a view of distillation as manifold alignment rather than Euclidean point matching (Yang et al., 30 May 2026).

Layer-wise signals can also govern optimization rather than appear only as additive penalties. The layer-wise learning-rate scheme for transfer learning computes per-layer discrepancies between student and teacher attention, Jacobian, or Hessian maps and uses those discrepancies to update per-layer learning rates every 25 epochs, instead of relying on a single cumulative distillation loss for the entire network (Kokane et al., 2024).

4. Self-distillation, regularization, and preservation of internal knowledge

A prominent use of layer-wise distillation is as a regularizer against catastrophic forgetting or representation collapse during fine-tuning. DistilMOS is explicit on this point: MOS-only fine-tuning of pretrained SSL encoders can forget pretrained knowledge and overfit the MOS training set. Its solution is to extract frame-level outputs from all Transformer layers, cluster each layer independently with $n$2-means, use $n$3 clusters with MiniBatchKMeans and batch size 64, and train $n$4 token predictors—each a 3-layer MLP with GELU activations—to predict the resulting layer-specific token IDs (Yang et al., 20 Jan 2026).

The same paper also reports that an MSE distillation variant, which regresses continuous SSL embeddings directly, underperforms token prediction, and that CCA-based analysis shows DistilMOS and MSE distillation maintain high correlation with SSL layer outputs while MOS-only models and the w/o token prediction variant have correlations close to random initialization. This supports the paper’s claim that discrete layer-wise targets can preserve hierarchical pretrained structure more effectively than unrestricted task-only fine-tuning (Yang et al., 20 Jan 2026).

DistilHuBERT adopts a related principle in pre-training compression. A 2-layer Transformer student is initialized from the teacher’s CNN extractor and first two Transformer layers, then trained to predict selected teacher layers through three heads that are removed after distillation. The heads are useful during pre-training but not needed afterward, and removing them reduces model size by about 13% with only a slight performance drop (Chang et al., 2021).

LR-Drop shows that the teacher need not be external at all. The two dropout-sampled subnetworks are reciprocal teacher and student models, and the consistency constraints are imposed inside every Transformer layer rather than only at the output. This reframes layer-wise distillation as mutual self-distillation under stochastic parameter masking (Ni et al., 2024).

5. Compression, pruning, and adaptive inference

In model compression, layer-wise distillation often means controlling depth explicitly rather than merely supervising every layer of a fixed student. “Iterative Layer-wise Distillation for Efficient Compression of LLMs” repeatedly estimates layer importance by removing each layer in turn, measuring performance degradation on seven representative datasets, pruning the least important layer, fine-tuning the reduced model with a joint KLDivLoss and MSELoss objective, and then recomputing importance. On Qwen2.5-3B, this reduces 36 layers to 28 layers, yielding a 2.47 billion parameter model with only a 9.7% quality loss, and to 24 layers with an 18% loss; layers 17–24 are most often removed, while removing final layers is much more harmful (Kovalev et al., 7 Nov 2025).

PPCL applies a related depth-aware logic to Diffusion Transformers, but at the level of contiguous intervals. It fits linear probes to teacher layers, uses first-order differential trend analysis of CKA similarity to detect redundant depth-wise intervals, and then distills each interval non-sequentially by aligning the student’s interval entrance with the teacher’s interval boundary. The resulting framework combines depth-wise and width-wise pruning in a single training phase and reports a 50\% reduction in parameter count with less than 3\% degradation in key objective metrics (Ma et al., 20 Nov 2025).

Efficiency objectives can also expose incompatibilities with standard layer-wise supervision. LEAP argues that ordinary layer-aligned distillation suppresses the representational convergence required for convergence-based early exit, producing zero effective speedup under standard deployment conditions. LEAP-MiniLM instead adds an exit-aware objective so that intermediate layers approximate the final representation, and at $n$5 reports 1.61$n$6 measured wall-clock speedup, 91.9% of samples exiting by layer 7, and 1.80$n$7 theoretical layer reduction on an NVIDIA L4 at batch size 1 (Kapadia et al., 1 May 2026).

Several methods extend layer-wise distillation to nonstandard compression targets. LaDiMo converts selected FFN layers of LLaMA2-7B into MoE blocks by training each MoE layer independently to reproduce its original FFN output with MSE plus a load-balancing auxiliary loss; replacing the last 12 layers yields about 10% throughput improvement, about 97% of the original MMLU accuracy, a model of about 6.7B parameters, and activated parameters of 5.5B (Kim et al., 2024). InDistill first prunes teacher widths so that teacher and student layers become directly compatible, then distills intermediate layers sequentially under a curriculum that respects critical learning periods and increasing layer difficulty (Sarridis et al., 2022). LaSNN uses layer-wise ANN-to-SNN attention distillation to fine-tune a converted SNN and reports competitive accuracy with about 20x faster inference than converted SNNs with similar performance (Hong et al., 2023). ProDistill brings the same layer-by-layer principle to model merging, optimizing one layer at a time with dual activations from the merged and teacher models and scaling to models with over 10B parameters (Xu et al., 18 Feb 2025).

6. Empirical patterns, trade-offs, and recurring misconceptions

Across modalities, the reported gains are substantial but uneven. DistilHuBERT reduces HuBERT’s size by 75%, is 73% faster, and retains most performance in ten different downstream tasks (Chang et al., 2021). DistilMOS consistently improves in-domain BVCC and zero-shot SOMOS; for WavLM Large on SOMOS utterance-level SRCC, the score improves from 0.188 for SSL-MOS to 0.387 for DistilMOS (Yang et al., 20 Jan 2026). DLink enables compact EEG students to outperform lightweight baselines while approaching fully fine-tuned FM performance; on FACED, MiC-M reaches 0.5221 ACC-B, and on Mumtaz2016, MiC-M with DLink achieves 0.9024 ACC-B (Wang et al., 16 Apr 2026). ProDistill reports up to 6.14% and 6.61% improvements in vision and NLU model merging tasks, respectively (Xu et al., 18 Feb 2025).

Several misconceptions are corrected repeatedly in the literature. One is that the final layer is always the best distillation target; DistilHuBERT and DLink both argue that intermediate representations can be more transferable than the top layer, especially in speech SSL and EEG FMs (Chang et al., 2021, Wang et al., 16 Apr 2026). A second is that more layer-wise supervision is automatically better. TED shows that matching all teacher hidden states can underfit the student because the teacher’s representations contain redundant information, while the audio reasoning framework finds that full layer-wise text KD plus SFT improves AQA but degrades SER, and that skip-layer distillation sits between top-layer-only KD and full layer-wise KD (Liang et al., 2022, Yang et al., 23 Sep 2025). A third is that layer-wise distillation is uniformly compatible with adaptive inference; LEAP argues the opposite for standard layer-aligned distillation under convergence-based early exit (Kapadia et al., 1 May 2026).

A broader pattern is the movement away from rigid one-to-one layer matching. Attention-based teacher fusion, task-aware filters, dynamic routing, geometric projection, spectral alignment, module-wise bandit selection, and progressive layer-by-layer optimization all replace the assumption that a student should simply copy one corresponding teacher layer. This suggests that the most durable interpretation of layer-wise distillation is not “match every layer,” but “treat depth as a structured supervision space whose useful targets depend on modality, architecture, task, and deployment objective.”