Two-Tiered Knowledge Distillation

• The paper introduces two-tiered knowledge distillation that integrates high-level soft targets with intermediate feature representations to enhance the student model’s performance.
• The method employs dual loss functions—soft target loss and intermediate correlation loss—to balance global decision boundaries with fine-grained structure matching.
• Empirical results across vision and speech tasks reveal improvements in accuracy, robustness, and transferability while mitigating gradient conflicts in complex teacher setups.

Two-tiered knowledge distillation refers to a class of techniques in model compression and transfer learning wherein the student model is concurrently or sequentially supervised by two qualitatively distinct types of knowledge from one or more teacher models. The two tiers represent hierarchical levels of information: high-level semantic content (e.g., classification logits or soft targets) and lower-level, often intermediate, representations (e.g., feature or attention maps, intra-activation relationships) or inter-instance relations. This framework enhances the information transferred during distillation, allowing the student to capture both global decision boundaries and finer-grained, structural, or relational knowledge. Its expressiveness enables robust knowledge transfer under model capacity constraints, cross-task transfer, or under adverse input conditions.

1. Foundational Principles and Types of Two-Tier Knowledge

Two-tiered distillation architectures are distinguished by their explicit transfer of complementary information streams:

• Tier 1: High-Level (Semantic/Logit/Soft Target) Knowledge This tier uses the teacher’s output logits or softmax distributions (soft targets) as an additional supervisory signal. These soft targets encode inter-class similarity and dark knowledge, guiding the student’s final predictions toward the teacher’s class assignments and confidence structure.
• Tier 2: Intermediate/Structural/Relational Knowledge This tier covers mechanisms that transfer more granular or structural information, such as intermediate feature maps, attention patterns, intra-layer similarities, or inter-instance relationships (as in relational/contrastive KD). This encompasses feature matching, attention transfer, hint-based learning (e.g., FitNet), and sample correlation alignment.

Distinct instantiations of two-tiered KD include adaptive multi-level distillation with multiple teachers (Liu et al., 2021), multi-level alignment and correlation (Ding et al., 2020), dual-head distillation decoupling logit and probability supervision (Yang et al., 2024), two-step schedules with intra-activation pattern pre-alignment (Nathoo et al., 2023), and collaborative teaching from both expert and dynamic scratch teachers (Zhao et al., 2019).

2. Mathematical Formulations and Loss Construction

Formally, two-tiered knowledge distillation frameworks combine loss functions from both tiers in the overall training objective, often with learnable or fixed balancing weights:

• Soft Target Loss (High-Level):

$$L_{\text{soft}} = \sum_{i=1}^N \left[ H(y_i, p_S(x_i)) + \lambda\, D_{KL}\left(\tilde{p}_i^T \, \|\, \tilde{p}_i^S \right) \right]$$

where $p_S(x_i)$ is the student’s hard prediction, $\tilde{p}_i^S$ and $\tilde{p}_i^T$ are (possibly weighted) softened outputs from student and (multiple) teachers with temperature $T$, and $\lambda > 0$ weighs ground-truth versus distillation loss (Liu et al., 2021).

• Intermediate/Correlation Loss (Tier 2):
  • Multi-group hint loss:

  $$L_{\text{hint}} = \sum_{i=1}^N \sum_{g=1}^G \left\| v_g(x_i) - \sum_{t=1}^m w_{t,i}\, F_t^g(u_t(x_i)) \right\|_2^2$$

  where $v_g(x_i)$ and $u_t(x_i)$ are feature maps from the student’s and teachers’ grouped layers, with adapter $F_t^g$ (Liu et al., 2021). - Feature alignment or logit KL:

  $p_S(x_i)$0

  (Ding et al., 2020) - Correlation alignment (relational KD):

  $p_S(x_i)$1

  where $p_S(x_i)$2 and $p_S(x_i)$3 are softmax-normalized cosine similarity distributions (Ding et al., 2020).

The total objective is a linear or weighted sum, e.g.,

$p_S(x_i)$4

Recent variants (e.g., dual-head KD (Yang et al., 2024)) avoid conflicting gradients by isolating losses to distinct classifier heads to eliminate destructive interference.

3. Representative Two-Tier Frameworks and Variants

Multiple research directions manifest two-tiered knowledge transfer as follows:

Framework	Tier 1: High-Level KD	Tier 2: Structural/Relational KD
AMTML-KD (Liu et al., 2021)	Teacher-soft-targets (adaptive, multi)	Multi-group hint transfer (weighted teacher feature fusion)
MLKD (Ding et al., 2020)	Per-sample feature/logit alignment	Pairwise sample correlation alignment (contrastive/relational)
DHKD (Yang et al., 2024)	Probability-level (CE/KL on softmax)	Logit-level (BinaryKL) via auxiliary classifier head
2-step KD (Nathoo et al., 2023)	Output response or Gram-matrix matching (pretrain stage)	Supervised or mixed fine-tuning with continued KD
CTKD (Zhao et al., 2019)	Dynamic scratch teacher (step-by-step logit matching)	Pre-trained expert teacher attention map matching

Each approach exploits distinct aspects of teacher supervision, employing either parallel, staged, or hybrid control over the student’s learning trajectory.

4. Algorithmic Schemes and Training Schedules

Algorithmic instantiations of two-tiered KD vary in the timing and integration of the two supervision tiers:

• Parallel (Simultaneous) Supervision:

Both tiers’ losses are computed every iteration/batch. Instances include adaptive multi-teacher distillation (Liu et al., 2021) and dual-head KD (Yang et al., 2024).

• Sequential (Two-Step) Protocols:

The student is first aligned with teacher representations via unsupervised KD, then fine-tuned under supervised loss alone or in a mixed scheme. Two-step KD for speech enhancement is a prime example (Nathoo et al., 2023), which demonstrates that pure KD pretraining followed by supervised optimization outperforms one-step mixtures, especially for highly compressed students or noisy settings.

• Collaborative/Hybrid:

CTKD employs both a jointly trained scratch teacher (providing path-level guidance) and a fixed expert teacher (providing focus-level attention guidance) (Zhao et al., 2019).

Common training elements include adaptive weight calculation (e.g., instance-level softmax attention over multiple teachers (Liu et al., 2021)), use of projections/adapters to align intermediate dimensions, and ablation or gradual annealing of loss weights to probe the impact of each tier.

5. Empirical Performance Across Domains

Empirical studies demonstrate consistent gains from two-tiered distillation across diverse domains and architectures:

• Computer Vision Benchmarks:

On CIFAR-10/100 and Tiny-ImageNet, AMTML-KD improves student accuracy by up to 0.8% over strong single-tier multi-teacher and mutual learning baselines; multi-level KD (MLKD) surpasses prior contrastive and attention-based KD on CIFAR-100, ImageNet, and Cityscapes (Liu et al., 2021, Ding et al., 2020).

• Robustness to Compression and Adverse Conditions:

The two-step pretrain-then-supervise protocol for tiny speech enhancement yields SDR gains of up to 0.9 dB at –5 dB SNR and 63x compression, outperforming any weighted-loss mixture of KD and supervision (Nathoo et al., 2023).

• Generalization and Transferability:

Multi-level KD (MLKD) consistently provides more transferable representations for linear evaluation on STL-10 and Tiny-ImageNet, and shows superior performance for self-supervised distillation from contrastive (e.g., MoCo-v2) teachers (Ding et al., 2020).

• Ablation Studies:

Removal of either supervision tier produces a measurable drop in performance, underscoring their complementary value. For example, in MLKD, “Align+Corr” yields higher accuracy than either term alone; in AMTML-KD, eliminating adaptive weights or hints reduces accuracy by 0.3–0.7% (Liu et al., 2021, Ding et al., 2020).

• Avoidance of Gradient Pathologies:

Dual-head KD resolves catastrophic collapse seen when naive summation of probability and logit-level losses is used in a single head (Yang et al., 2024).

6. Extensions, Implications, and Practical Guidance

• Adaptive Weighting and Teacher Assignment:

Instance-adaptive teacher weighting via learnable latent factors enhances student performance over naive averaging of teacher outputs (Liu et al., 2021). Tier-2 “group” assignment of features, weighted or prioritized by teacher competence, is empirically beneficial.

• Structural Decoupling for Loss Integration:

When gradients from distinct losses conflict (as mathematically characterized in DHKD via neural collapse theory), architectural isolation (dual-head design) avoids destructive interference yet retains both loss benefits for the shared feature backbone (Yang et al., 2024).

• Extensibility:

Two-tiered KD is a flexible metaframework, subsuming prior approaches such as FitNet, attention transfer, relational/contrastive KD, and multi-head supervision. It is model-agnostic and applicable under both supervised and self-supervised regimes (Ding et al., 2020).

• Resource and Cost Considerations:

Additional computational load is generally limited to training (e.g., extra teacher in CTKD, KD pretraining stage), with negligible inference overhead for the student.

7. Limitations and Open Problems

Although two-tiered KD consistently improves performance over single-tier methods, typical gains are modest in well-behaved, high-capacity students and may saturate as the number of teachers grows (Liu et al., 2021). Direct naive aggregation of potentially incompatible losses (without adapter modules or architectural separation) can lead to optimization failure (Yang et al., 2024). Theoretical understanding of the optimal curriculum, inter-tier balance, or adaptation for emerging cross-modal or structural transfer tasks remains incomplete. Further, the cost-effectiveness of multi-teacher or collaborative approaches must be weighed when scaling to very deep or heterogeneous teacher ensembles.

---

Two-tiered knowledge distillation, by orchestrating complementary forms of supervisory signal, enables student models to inherit both high-level semantic authority and deep structural regularization from teachers. This principle underpins state-of-the-art results in model compression, transfer, and robustness across computer vision and speech domains (Liu et al., 2021, Ding et al., 2020, Zhao et al., 2019, Nathoo et al., 2023, Yang et al., 2024).