Dual-Layer Knowledge Distillation

• Dual-Layer Knowledge Distillation is a dual-channel method where both output logits and hidden features guide the student model, enhancing convergence and generalization.
• It has been effectively applied in one-shot federated medical imaging, where matching teacher outputs and intermediate activations improves performance under non-IID data.
• DLKD frameworks incorporate diverse designs—including dual teachers, dual spaces, and dual optimization layers—to provide adaptive supervision and tackle complex learning challenges.

Dual-Layer Knowledge Distillation (DLKD) denotes a class of knowledge-distillation schemes that transfer supervisory information at two complementary levels rather than relying solely on final-logit matching. In the most literal usage, DLKD trains a student to match both a teacher’s output logits and an intermediate hidden representation; in broader usage, the same label is applied to dual-teacher, dual-space, dual-granularity, or dual-branch distillation designs. A particularly explicit instantiation appears in one-shot federated medical imaging, where DLKD is the server-side aggregation mechanism that trains a global student on synthetic feature-level data by matching both the ensemble teacher’s softened outputs and its intermediate-layer features in a single communication round (Ma et al., 25 Jul 2025).

1. Terminology and scope

The term is used in several related, but not identical, senses across the literature. In all cases, the common denominator is that distillation is split into two complementary transfer channels rather than a single output-level supervision.

Formulation	Duality	Setting
DLKD (Ma et al., 25 Jul 2025)	output logits + intermediate feature layer	one-shot federated medical imaging
FairDTD (Li et al., 2024)	feature teacher + structure teacher; node/logit + graph/representation distillation	fair GNNs
DeepKD (Huang et al., 21 May 2025)	gradient/optimization level + logit/class level	image classification and detection
Dual knowledge distillation (Xiao et al., 2024)	temporal/output KD + embedding/representation KD	efficient sound event detection
DWA-KD (Vu et al., 25 Feb 2026)	token-level dual-space KD + sequence-level alignment	cross-tokenizer LLM distillation

In the medical federated-learning formulation, “dual-layer” is literal: the student matches the teacher at the output layer and at a chosen intermediate hidden layer, such as the 3rd block of ResNet-18 (Ma et al., 25 Jul 2025). In FairDTD, the duality is partly in the teachers and partly in the representation depth: a feature teacher and a structure teacher each supervise the student through node-level logits and graph-level representations (Li et al., 2024). In DeepKD, the term is used more abstractly: one layer of the method operates in gradient space through independent momentum buffers for task-oriented, target-class, and non-target-class gradients, while the other operates in logit space through dynamic top-$k$ denoising of dark knowledge (Huang et al., 21 May 2025). This suggests that DLKD is best understood as a family of two-channel distillation strategies rather than a single standardized algorithm.

2. Canonical formulation in one-shot federated medical imaging

The clearest named DLKD method is the server-side aggregation stage of a one-shot federated-learning framework for medical imaging classification (Ma et al., 25 Jul 2025). Each client trains a classifier $c_{\gamma_k}$ on local feature-level data and uploads it to the server together with a Feature-Guided Rectified Flow generative model. The server forms an ensemble teacher by parameter averaging,

$$\bar{c}_\gamma = \frac{1}{K}\sum_{k=1}^K c_{\gamma_k},$$

generates synthetic features $\hat{z}$ from the uploaded FG-RF models, and trains a randomly initialized global student $s_\psi$ only once on this synthetic dataset (Ma et al., 25 Jul 2025).

The DLKD objective combines hard-label cross-entropy, logit-level KL divergence, and feature-level $L_2$ regression: $$\mathcal{L}_{\text{total}} = \mathbb{E}_{\hat{z}, y} \Big[ (1-\alpha)\,\mathcal{L}_{\text{CE}}(s_\psi(\hat{z}), y) + \alpha\,\mathcal{L}_{\text{KL}} + \beta\,\mathcal{L}_{\text{feat}} \Big].$$ Here, $\mathcal{L}_{\text{KL}}$ matches softened teacher and student output distributions with temperature $T$, while $\mathcal{L}_{\text{feat}}$ aligns the teacher and student activations at an intermediate layer $c_{\gamma_k}$0; in the ResNet-18 instantiation, the selected layer is the third layer or block (Ma et al., 25 Jul 2025). The resulting update is performed entirely at the server: $c_{\gamma_k}$1

This formulation is designed for a setting in which the global model must converge in one aggregation round under highly non-IID medical data. The motivation is that logits alone do not preserve sufficiently rich hierarchical information, especially when pathology-specific cues are subtle and synthetic features do not perfectly match real feature distributions. The added feature-alignment term is intended to stabilize training on synthetic data, capture cross-site shared structure, and improve convergence and generalization under non-IID distributions (Ma et al., 25 Jul 2025).

3. Distillation levels, teachers, and representational views

Across the literature, the most recurrent DLKD pattern is the combination of output-layer distillation and hidden-layer or representation-level distillation. This pattern is explicit in the medical one-shot federated-learning method, in FairDTD’s node-level and graph-level losses, in the temporal-averaging and embedding-enhanced SED framework, in DSKD’s supervision of shallow and final layers, and in DWA-KD’s token-level KL plus embedding- and hidden-state Soft-DTW alignment (Ma et al., 25 Jul 2025, Li et al., 2024, Xiao et al., 2024, Luo et al., 2022, Vu et al., 25 Feb 2026).

A second recurring pattern is duality in the source of knowledge rather than only in representational depth. FairDTD employs two fairness-oriented teachers, $c_{\gamma_k}$2 and $c_{\gamma_k}$3, trained on node features and graph structure respectively, and combines them through

$c_{\gamma_k}$4

where each $c_{\gamma_k}$5 itself includes soft-logit and intermediate-representation terms (Li et al., 2024). DRKD similarly separates knowledge into pixel-wise global relations and instance-wise relations for object detection, while DMKD separates spatially important and channel-wise informative clues through dual masking branches (Ni et al., 2023, Yang et al., 2023). These variants broaden DLKD from “two layers in one network” to “two complementary representational subspaces.”

A third pattern is duality in optimization and information filtering. DeepKD decomposes training signals into task-oriented gradient, target-class gradient, and non-target-class gradient, but its own interpretation of dual-layer KD is that optimization-level decoupling and logit-level denoising work together (Huang et al., 21 May 2025). DWA-KD adopts a similarly layered view in language-model distillation: one layer aligns dual token spaces through weighted KL, while another aligns whole sequences through Soft-DTW on embeddings and final hidden states (Vu et al., 25 Feb 2026). A plausible implication is that DLKD has become a generic design principle for separating complementary supervisory channels, even when those channels are no longer literal hidden layers.

Related work predating the explicit DLKD label provides important antecedents. DSKD supervises shallow and final student layers with teacher predictions and feature maps, and can be viewed as a sparse or degenerate two-layer variant when only one intermediate head is retained (Luo et al., 2022). ALP-KD replaces fixed teacher–student layer pairing by attention-weighted combinations of teacher layers, offering a cross-layer alternative to fixed dual-layer matching (Passban et al., 2020). The learnable KD layer of letKD inserts a residual knowledge module into intermediate and penultimate student layers so that the student uses distilled templates during inference as well as training (Gorgun et al., 2023).

4. Optimization, alignment, and theoretical motivation

In the medical federated-learning setting, the formal motivation for DLKD is a domain-adaptation-style error bound. If

$c_{\gamma_k}$6

then the student’s real-data error is bounded by terms involving the divergence between real and synthetic distributions and the discrepancy between student and teacher behavior. The authors argue that feature alignment reduces the distribution-gap term by pulling generated features toward the real-data subspace represented by the teacher’s features, while joint logit and feature matching reduces the distillation-consistency term (Ma et al., 25 Jul 2025). In that formulation, DLKD is not merely an empirical regularizer but an attempt to tighten the upper bound on real-data error when the student is trained only on synthetic features.

FairDTD motivates dual-teacher distillation through a causal graph. Sensitive attributes influence both node features and graph structure through unblocked causal paths, so feature-only and structure-only teachers each weaken a different bias channel. The student, which sees the full graph, is then regularized by both teachers through node-level logits and normalized intermediate representations. The same framework introduces node-specific temperatures,

$c_{\gamma_k}$7

rescaled to $c_{\gamma_k}$8, so that teacher influence is adapted to per-node confidence (Li et al., 2024). This provides an instance-adaptive counterpart to fixed-temperature KL.

DeepKD provides a different theoretical lens by analyzing gradient signal-to-noise ratio (GSNR). It separates task-oriented gradient, target-class gradient, and non-target-class gradient, computes separate momentum buffers, and argues that higher-GSNR components should receive higher momentum. Its Dynamic Top-k Mask starts from $c_{\gamma_k}$9, holds $$\bar{c}_\gamma = \frac{1}{K}\sum_{k=1}^K c_{\gamma_k},$$0 in a middle phase, and then increases to $$\bar{c}_\gamma = \frac{1}{K}\sum_{k=1}^K c_{\gamma_k},$$1 over the final phase, thereby denoising dark knowledge early and gradually reintroducing harder non-target classes later (Huang et al., 21 May 2025). DWA-KD adopts an analogous reliability principle at the token level: weights increase when the student is uncertain and the teacher is confident after projection into the student vocabulary, while sequence-level Soft-DTW aligns lexical and semantic trajectories under tokenizer mismatch (Vu et al., 25 Feb 2026). Taken together, these methods present DLKD as a mechanism for aligning not only targets but also signal quality, representational geometry, and training dynamics.

5. Empirical behavior across application domains

In one-shot federated medical imaging, DLKD is central to the strongest reported results. On Brain Tumor, Chest X-Ray TB, and HAM10000, the full FG-RF + DLKD method reaches 60.15, 86.32, and 70.01 accuracy, compared with 55.84, 84.74, and 68.99 for logit-only KD and 55.08, 83.16, and 64.91 without DLKD. The overall framework achieves up to 21.73% improvement over multi-round federated-learning approaches and exceeds FedISCA by an average of 21.75%; t-SNE visualizations further show tighter cross-client clustering with DLKD than without KD (Ma et al., 25 Jul 2025).

In fair graph learning, FairDTD reports simultaneous gains in fairness and utility. On the Credit dataset with a GCN backbone, vanilla GCN gives ACC 73.37, $$\bar{c}_\gamma = \frac{1}{K}\sum_{k=1}^K c_{\gamma_k},$$2, and $$\bar{c}_\gamma = \frac{1}{K}\sum_{k=1}^K c_{\gamma_k},$$3; FairGKD gives ACC 79.63, $$\bar{c}_\gamma = \frac{1}{K}\sum_{k=1}^K c_{\gamma_k},$$4, and $$\bar{c}_\gamma = \frac{1}{K}\sum_{k=1}^K c_{\gamma_k},$$5; FairDTD gives ACC 80.83, $$\bar{c}_\gamma = \frac{1}{K}\sum_{k=1}^K c_{\gamma_k},$$6, and $$\bar{c}_\gamma = \frac{1}{K}\sum_{k=1}^K c_{\gamma_k},$$7. The paper states that, across Pokec-z, Pokec-n, and Credit, FairDTD achieves the best or tied-best fairness while preserving high model utility (Li et al., 2024).

In efficient sound event detection, the dual scheme of Temporal-Averaging Knowledge Distillation and Embedding-Enhanced Feature Distillation shows that output-level and representation-level KD are both useful. The SE-CRNN-16 student with EEFD + TAKD reaches PSDS1 0.465 and PSDS2 0.690 without additional AFL or data augmentation, compared with 0.456 and 0.693 for the 4.5M-parameter teacher. With AFL + data augmentation, the same 1.3M-parameter student reaches PSDS1 0.474 and PSDS2 0.698, while the 400K-parameter SE-CRNN-8 + EEFD + TAKD exceeds the 1.2M-parameter baseline CRNN on both PSDS1 and PSDS2 (Xiao et al., 2024).

In logit-based image classification and detection, DeepKD reports gains from combining decoupled momentum with dynamic top-k denoising. On ImageNet, KD+DeepKD with top-k raises ResNet50→MobileNet-V1 from 70.50% to 74.65%; on COCO detection, KD+DeepKD with DTM improves AP from 30.13 to 32.16, and DKD+DeepKD reaches 34.20 AP. The same work reports consistent improvements on CIFAR-100 and shows that assigning higher momentum to TOG and NCG and lower momentum to TCG yields the best performance (Huang et al., 21 May 2025).

In object detection, dual-branch distillation has also produced substantial gains. DRKD improves Faster R-CNN based on ResNet50 from 38.4% to 41.6% mAP and RetinaNet based on ResNet50 from 37.4% to 40.3% mAP on COCO 2017, with especially notable gains on small objects through the instance-wise relation branch (Ni et al., 2023). DMKD reports gains of 4.1% and 4.3% when RetinaNet and Cascade Mask R-CNN are respectively used as teacher networks, and its ablations show that dual masking outperforms spatial-only or channel-only masking (Yang et al., 2023).

In cross-tokenizer language-model distillation, DWA-KD reports average ROUGE-L 19.35 for Qwen-1.5–1.8B → GPT-2 340M, compared with 17.25 for DSKD, and 22.85 for Qwen2.5-7B → GPT-2 1.5B, compared with 21.04 for DSKD. Its ablations indicate that entropy-based token weighting and Soft-DTW alignment at the embedding and final hidden-state layers make complementary contributions (Vu et al., 25 Feb 2026).

6. Limitations, misconceptions, and open directions

A common misconception is that DLKD names a single, settled method. The available literature shows otherwise. In some works it denotes output-logit plus intermediate-feature distillation; in others it denotes dual teachers, dual branches, dual spaces, or dual optimization layers (Ma et al., 25 Jul 2025, Li et al., 2024, Huang et al., 21 May 2025, Vu et al., 25 Feb 2026). The unifying idea is architectural or supervisory duality, not a fixed loss.

Several limitations recur across formulations. The medical federated-learning method does not provide the exact numeric values of $$\bar{c}_\gamma = \frac{1}{K}\sum_{k=1}^K c_{\gamma_k},$$8, $$\bar{c}_\gamma = \frac{1}{K}\sum_{k=1}^K c_{\gamma_k},$$9, and $\hat{z}$0 in the main body, even though these hyperparameters govern the balance among hard labels, KL loss, and feature regression; its privacy argument also depends on feature-level generation rather than raw-image generation and on a single one-shot communication regime (Ma et al., 25 Jul 2025). FairDTD assumes one binary sensitive attribute and requires labeled sensitive attributes; it also depends on the quality of partial-data teachers and has only been demonstrated on Pokec-z, Pokec-n, and Credit (Li et al., 2024). DeepKD remains logit-centric, uses a fixed GSNR-to-momentum mapping through a single $\hat{z}$1, and relies on a heuristic 0.3/0.7 curriculum for dynamic top-k masking (Huang et al., 21 May 2025). DWA-KD inherits quadratic Soft-DTW cost in sequence length and depends on projector stability, while DMKD and DRKD assume task-specific structures such as feature pyramids, RoIAlign, or ground-truth boxes for instance extraction (Vu et al., 25 Feb 2026, Yang et al., 2023, Ni et al., 2023).

Open directions in the surveyed work are correspondingly diverse. FairDTD identifies multi-sensitive-attribute fairness, more general teacher designs, and extension from node classification to link prediction and graph classification (Li et al., 2024). DeepKD suggests extending GSNR-guided optimization to feature-level losses and replacing hard masking with smooth weighting (Huang et al., 21 May 2025). DWA-KD points toward multi-teacher distillation, more layers, and adaptive sequence-band learning (Vu et al., 25 Feb 2026). The learnable KD-layer approach suggests a different frontier: stacking multiple inference-time KD layers so that teacher knowledge is not only distilled into parameters but remains an explicit part of the student’s forward transform (Gorgun et al., 2023). This suggests that future DLKD systems may become increasingly heterogeneous, combining multi-depth supervision, adaptive weighting, structured alignment, and task-specific inductive biases rather than converging on a single canonical recipe.