Deep-Supervised Knowledge Distillation (DSKD)

• Deep-Supervised Knowledge Distillation (DSKD) is a paradigm that applies teacher supervision at multiple network depths rather than just the final output.
• It uses auxiliary classifiers and side outputs to align intermediate representations with teacher predictions and features, enhancing optimization and robustness.
• In segmentation, DSKD leverages dual loss functions—deep distribution loss and pixel-wise self-distillation—to capture both global structures and local details.

Searching arXiv for papers on Deep-Supervised Knowledge Distillation and the cited segmentation instantiation. Deep-Supervised Knowledge Distillation (DSKD) denotes a class of knowledge distillation methods in which supervision is injected not only at a student model’s terminal output but also at intermediate depths, typically through auxiliary heads or hierarchical outputs. In the classification formulation introduced as “Knowledge Distillation with Deep Supervision,” DSKD uses teacher class predictions and teacher feature maps to supervise shallow student layers, with adaptive loss-based weighting across auxiliary heads (Luo et al., 2022). In a later segmentation instantiation, “Deep Self-knowledge Distillation: A hierarchical supervised learning for coronary artery segmentation,” the same broad principle is realized through hierarchical side-output supervision in an encoder–decoder, combining a multi-depth distributional loss with pixel-wise self-distillation (Lin, 3 Sep 2025). Across these formulations, the central premise is that distillation should preserve hierarchical knowledge rather than restrict transfer to the final layer alone.

1. Conceptual scope and relation to conventional knowledge distillation

Standard response-based knowledge distillation, as summarized in the segmentation paper, distills teacher knowledge from the final output alone, usually by matching softened logits or probabilities (Lin, 3 Sep 2025). The DSKD formulation in the classification paper identifies a specific limitation of this practice: gradients originating only from the last layer may attenuate as they backpropagate to shallow layers, leaving early student representations under-supervised and thereby reducing the effectiveness of hierarchical knowledge transfer (Luo et al., 2022).

DSKD addresses this limitation by extending supervision “deep” into the network. In the classification setting, this is implemented by attaching auxiliary classifiers to shallow student stages and training them against teacher signals (Luo et al., 2022). In the segmentation setting, the same principle appears as supervision over hierarchical decoder outputs, where multi-scale side outputs receive distillation constraints rather than leaving all supervision to the final segmentation map (Lin, 3 Sep 2025).

This suggests that DSKD is best understood not as a single loss, but as an architectural and training principle: teacher knowledge is exposed to the student at multiple representational depths. The intended effect, stated explicitly across the two papers, is to improve optimization, regularize intermediate learning, and strengthen generalization by constraining the student’s internal learning trajectory rather than only its endpoint (Luo et al., 2022).

2. Canonical classification formulation

The classification formulation of DSKD, proposed in “Knowledge Distillation with Deep Supervision” (Luo et al., 2022), is defined for a student with $L$ stages. Auxiliary classifiers $c_l$ are attached to each shallow stage $l = 1, \ldots, L-1$, while the original final classifier is denoted $c_L$. Each auxiliary head includes multiple lightweight convolutional blocks, followed by global average pooling and a fully connected layer. The heads are constructed with depthwise separable convolutions and are designed to mimic the main branch’s downsampling path so that shallow features are converted into coarser semantics suitable for classification (Luo et al., 2022).

Teacher supervision enters through two complementary channels. First, teacher class-prediction distributions supervise both the final student classifier and all auxiliary classifiers. Second, the teacher’s last-layer feature map $F_T$ supervises the feature map before global average pooling in each auxiliary head, denoted $F_{c_l}$, after a learnable projection $r(\cdot)$ for dimensional alignment (Luo et al., 2022). The framework is explicitly training-only: auxiliary heads and alignment modules are removed at inference, so test-time cost is unchanged (Luo et al., 2022).

The classification paper formalizes temperature-scaled softmax as

$$\sigma_k(z/\tau) = \frac{\exp(z_k/\tau)}{\sum_{j=1}^K \exp(z_j/\tau)},$$

with teacher logits $z_T$, student logits $z_{c_l}$, and $c_l$0 classes. The final-layer KD term is

$c_l$1

Shallow-layer class distillation is

$c_l$2

Feature supervision uses mean squared error:

$c_l$3

and

$c_l$4

The total objective is

$c_l$5

where $c_l$6, and the reported experimental setting uses $c_l$7, $c_l$8, and $c_l$9 (Luo et al., 2022).

A distinctive feature of this formulation is adaptive weight allocation. For shallow prediction losses,

$l = 1, \ldots, L-1$0

and for shallow feature losses,

$l = 1, \ldots, L-1$1

The stated rationale is to assign larger weights to lagging heads with larger current losses, thereby balancing learning speeds across shallow layers (Luo et al., 2022).

3. Segmentation-oriented self-distillation instantiation

The segmentation paper “Deep Self-knowledge Distillation: A hierarchical supervised learning for coronary artery segmentation” instantiates DSKD in a teacher–student framework where the teacher is the model from the previous epoch and the student is the current model, both sharing the same architecture (Lin, 3 Sep 2025). The paper uses U-Net3+ as the experimental baseline, though the method is described as generic for encoder–decoder segmentation models (Lin, 3 Sep 2025).

Let the decoder depth be $l = 1, \ldots, L-1$2. At each decoder stage $l = 1, \ldots, L-1$3, a side-output head produces a one-channel full-resolution map by applying a $l = 1, \ldots, L-1$4 convolution $l = 1, \ldots, L-1$5 for channel reduction, bilinear upsampling $l = 1, \ldots, L-1$6 to the original spatial resolution, and a Sigmoid activation. With feature map $l = 1, \ldots, L-1$7, the side output is written as

$l = 1, \ldots, L-1$8

and denoted $l = 1, \ldots, L-1$9 (Lin, 3 Sep 2025).

Distillation is applied at two levels. First, every decoder stage participates in a distributional side-output distillation process called Deep Distribution Loss (DDL). Second, the shallowest and final side output is trained with a Pixel-wise Self-Knowledge Distillation Loss (PSDL), which uses a soft target formed by blending the previous epoch’s prediction with the ground truth (Lin, 3 Sep 2025). This construction makes the method both self-distillation and deep-supervised distillation.

The paper characterizes the two loss components as a dual regularization mechanism. DDL supplies loosely constrained, multi-depth, structure-level guidance; PSDL supplies tightly constrained, pixel-level guidance at the final prediction. The intended division of labor is that DDL shapes global or coarse structure, while PSDL anchors local detail (Lin, 3 Sep 2025). A plausible implication is that this segmentation variant translates the general DSKD principle into a form compatible with hierarchical decoder outputs rather than stagewise classification heads.

4. Loss design and hierarchical supervision mechanisms

In the segmentation formulation, the Deep Distribution Loss begins by partitioning each side output $c_L$0 into $c_L$1 non-overlapping patches of size $c_L$2, where $c_L$3. For the $c_L$4-th patch $c_L$5, the paper defines a two-dimensional vector

$c_L$6

This vector indicates whether a patch is entirely foreground or entirely background. The entries over all patches are flattened and normalized with a softmax with temperature $c_L$7:

$c_L$8

Student and teacher side outputs thereby induce distributions $c_L$9 and $F_T$0, and DDL is defined as

$F_T$1

The paper explicitly describes this representation as “loosely constrained,” because it ignores per-pixel detail and only encodes whether patches are entirely foreground or background (Lin, 3 Sep 2025).

The Pixel-wise Self-Knowledge Distillation Loss operates on the final student prediction $F_T$2 and final teacher prediction $F_T$3. A soft target is constructed as

$F_T$4

where $F_T$5 is the binary ground-truth mask and $F_T$6 is scheduled over epochs. The pixel-wise distillation objective is

$F_T$7

For binary segmentation, the paper gives cross-entropy in the form

$F_T$8

Teacher reliability is treated as evolving during training, and the paper therefore schedules $F_T$9 linearly:

$F_{c_l}$0

where $F_{c_l}$1 is the total number of epochs and $F_{c_l}$2 is the final weight (Lin, 3 Sep 2025).

To retain direct segmentation supervision, the student also receives a Dice loss:

$F_{c_l}$3

The full objective is

$F_{c_l}$4

with all terms equally weighted in the reported experiments (Lin, 3 Sep 2025).

Compared with the classification formulation, the segmentation variant does not align intermediate feature tensors directly. Instead, it distills compact probabilistic summaries of side outputs plus a soft pixel-wise target at the final layer (Lin, 3 Sep 2025). This suggests that DSKD is adaptable: what remains invariant is multi-depth teacher supervision, while the specific supervisory object may be logits, features, side-output distributions, or pixel-wise maps.

5. Training procedures, implementation choices, and computational properties

The classification DSKD training loop is teacher-frozen. For each minibatch, the teacher computes logits $F_{c_l}$5 and feature map $F_{c_l}$6; the student backbone computes stagewise features; each auxiliary head produces logits $F_{c_l}$7 and pre-global-average-pooling feature maps $F_{c_l}$8; per-layer KD and feature losses are computed; adaptive shallow-layer weights are formed from current losses; the total loss is backpropagated through the student, auxiliary heads, and projection modules (Luo et al., 2022). At inference, all auxiliary heads and $F_{c_l}$9 are discarded (Luo et al., 2022).

The segmentation self-distillation training loop differs in that no separate pre-trained teacher is required. A single model $r(\cdot)$0 is initialized. At epoch $r(\cdot)$1, the student is the current model $r(\cdot)$2, while the teacher is the previous epoch’s snapshot $r(\cdot)$3. If $r(\cdot)$4, only the Dice loss is used. For $r(\cdot)$5, teacher and student both produce final and side outputs; $r(\cdot)$6, $r(\cdot)$7, and $r(\cdot)$8 are then summed; after updating the student, the teacher is replaced by the updated snapshot for the next epoch (Lin, 3 Sep 2025).

The reported implementation choices in the two papers differ by task.

Aspect	Classification DSKD (Luo et al., 2022)	Segmentation self-DSKD (Lin, 3 Sep 2025)
Primary architecture	Various teacher–student pairs including VGG, ResNet, WRN, ShuffleNetV1/V2, MobileNetV2	U-Net3+ baseline in experiments
Optimization	240 epochs, batch size 64, weight decay $r(\cdot)$9	100 epochs, batch size 4, AdamW with weight decay $$\sigma_k(z/\tau) = \frac{\exp(z_k/\tau)}{\sum_{j=1}^K \exp(z_j/\tau)},$$0
Distillation hyperparameters	$$\sigma_k(z/\tau) = \frac{\exp(z_k/\tau)}{\sum_{j=1}^K \exp(z_j/\tau)},$$1, $$\sigma_k(z/\tau) = \frac{\exp(z_k/\tau)}{\sum_{j=1}^K \exp(z_j/\tau)},$$2, $$\sigma_k(z/\tau) = \frac{\exp(z_k/\tau)}{\sum_{j=1}^K \exp(z_j/\tau)},$$3	best reported $$\sigma_k(z/\tau) = \frac{\exp(z_k/\tau)}{\sum_{j=1}^K \exp(z_j/\tau)},$$4, $$\sigma_k(z/\tau) = \frac{\exp(z_k/\tau)}{\sum_{j=1}^K \exp(z_j/\tau)},$$5, best reported $$\sigma_k(z/\tau) = \frac{\exp(z_k/\tau)}{\sum_{j=1}^K \exp(z_j/\tau)},$$6

Both papers state that inference-time cost is unchanged because the extra supervisory structures are used only during training (Luo et al., 2022, Lin, 3 Sep 2025). The classification paper quantifies training-time overhead as small in parameters and FLOPs, with examples including WRN-16-2 from $$\sigma_k(z/\tau) = \frac{\exp(z_k/\tau)}{\sum_{j=1}^K \exp(z_j/\tau)},$$7M to $$\sigma_k(z/\tau) = \frac{\exp(z_k/\tau)}{\sum_{j=1}^K \exp(z_j/\tau)},$$8M parameters and from $$\sigma_k(z/\tau) = \frac{\exp(z_k/\tau)}{\sum_{j=1}^K \exp(z_j/\tau)},$$9M to $z_T$0M FLOPs, ResNet8x4 from $z_T$1M to $z_T$2M parameters and from $z_T$3M to $z_T$4M FLOPs, and MobileNetV2 from $z_T$5M to $z_T$6M parameters and from $z_T$7M to $z_T$8M FLOPs (Luo et al., 2022). The segmentation paper describes the overhead from side outputs, patch partitioning, and probability-vector construction as modest and explicitly notes that it does not affect inference speed or memory (Lin, 3 Sep 2025).

6. Empirical performance and comparative evidence

On CIFAR-100, the classification DSKD paper reports consistent gains over vanilla KD across seven teacher–student pairs. Examples include WRN-40-2 $z_T$9 WRN-40-1, where KD yields $z_{c_l}$0 Top-1 accuracy and DSKD yields $z_{c_l}$1; ResNet32x4 $z_{c_l}$2 ResNet8x4, where KD yields $z_{c_l}$3 and DSKD yields $z_{c_l}$4; and ResNet32x4 $z_{c_l}$5 ShuffleNetV2, where KD yields $z_{c_l}$6 and DSKD yields $z_{c_l}$7 (Luo et al., 2022). On TinyImageNet, reported gains include WRN-40-2 $z_{c_l}$8 WRN-40-1 from $z_{c_l}$9 to $c_l$00, and ResNet32x4 $c_l$01 ShuffleNetV2 from $c_l$02 to $c_l$03 (Luo et al., 2022). The paper further states that averaged across TinyImageNet pairs, DSKD exceeds KD, FitNet, AT, VID, CRD, ICKD, and DIST by $c_l$04, $c_l$05, $c_l$06, $c_l$07, $c_l$08, $c_l$09, and $c_l$10, respectively (Luo et al., 2022).

The classification paper also compares DSKD to deep supervision methods that do not use teacher knowledge in the same way. On CIFAR-100, it reports average improvements over DSN, BYOT, DKS, and DCM of $c_l$11, $c_l$12, $c_l$13, and $c_l$14, respectively; on TinyImageNet, the corresponding averages are $c_l$15, $c_l$16, $c_l$17, and $c_l$18 (Luo et al., 2022). The authors interpret this as evidence that teacher predictions and teacher features provide more effective shallow-layer supervision than ground-truth-only deep supervision, self-distillation, or peer distillation (Luo et al., 2022).

In the coronary artery segmentation setting, the self-DSKD paper evaluates on XCAD and DCA1, with 70% train, 10% validation, and 20% test splits on both datasets (Lin, 3 Sep 2025). XCAD contains 1,621 coronary X-ray angiography images at $c_l$19, while DCA1 contains 134 grayscale images at $c_l$20 (Lin, 3 Sep 2025). Metrics are Dice Similarity Coefficient (DSC), Accuracy (ACC), Sensitivity (SEN), and Intersection-over-Union (IOU), defined as

$c_l$21

$c_l$22

On XCAD, the best competing baseline listed is DconnNet at DSC $c_l$23, while the U-Net3+ baseline obtains DSC $c_l$24. Adding DDL raises DSC to $c_l$25; adding PSDL raises it to $c_l$26; combining both yields DSC $c_l$27, ACC $c_l$28, SEN $c_l$29, and IOU $c_l$30 (Lin, 3 Sep 2025). On DCA1, the best competing baseline listed is DconnNet at DSC $c_l$31, the U-Net3+ baseline obtains DSC $c_l$32, U-Net3+ + DDL yields $c_l$33, U-Net3+ + PSDL yields $c_l$34, and the full method yields DSC $c_l$35, ACC $c_l$36, SEN $c_l$37, and IOU $c_l$38 (Lin, 3 Sep 2025). The paper summarizes the gains as $c_l$39 DSC over the best non-distilled competitor on XCAD and $c_l$40 on DCA1; relative to the U-Net3+ baseline, the gains are $c_l$41 and $c_l$42, respectively (Lin, 3 Sep 2025).

Ablations in both papers emphasize that multi-depth supervision is additive rather than redundant. In classification, adding shallow class-prediction supervision and shallow feature supervision together performs better than either alone, and adding both at multiple shallow layers gives the best result for ResNet32x4 $c_l$43 ShuffleNetV2 on CIFAR-100, reaching $c_l$44 (Luo et al., 2022). In segmentation, DDL and PSDL each improve performance on both datasets, and the combination is best (Lin, 3 Sep 2025).

7. Interpretation, limitations, and relation to adjacent distillation paradigms

Within the broader KD literature represented in the supplied sources, DSKD differs from standard response-based KD by supervising multiple depths rather than only the final output (Luo et al., 2022, Lin, 3 Sep 2025). It also differs from feature-based KD methods such as FitNets, Attention Transfer, or FSP in how intermediate knowledge is represented. The classification DSKD paper uses teacher final features with explicit projection and MSE alignment (Luo et al., 2022), whereas the segmentation paper explicitly avoids direct feature regression at intermediate decoder stages and instead distills a compact patch-level distribution derived from side outputs (Lin, 3 Sep 2025). The segmentation paper notes that this avoids explicit feature alignment or adapter modules and is designed as a looser constraint to reduce inductive bias mismatch across scales (Lin, 3 Sep 2025).

Several practical limitations are identified in the two sources. The classification paper notes modest increases in training-time parameters, FLOPs, and memory due to auxiliary heads and stored intermediate features (Luo et al., 2022). It also reports that simple heads consisting only of global average pooling and a fully connected layer do not help beyond baseline, whereas more complex heads with convolutional blocks are necessary to extract coarse semantics from shallow features (Luo et al., 2022). The segmentation paper highlights sensitivity to $c_l$45, $c_l$46, and patch granularity $c_l$47, reporting best values at $c_l$48, $c_l$49, and $c_l$50, and stating that values that are too large or too small degrade performance (Lin, 3 Sep 2025).

The self-distillation segmentation variant introduces an additional dependency on teacher quality. Because the teacher is the previous epoch’s snapshot, early teachers are weak; the paper therefore linearly increases $c_l$51 so that training gradually trusts the teacher more over time (Lin, 3 Sep 2025). In the classification formulation, the teacher is fixed and pre-trained, so this issue does not arise in the same form (Luo et al., 2022).

The two papers also indicate different extension paths. The classification paper identifies extension to detection, segmentation, and multi-teacher settings as promising directions (Luo et al., 2022). The segmentation paper states that the method is tailored to binary vessel segmentation but can generalize to other medical segmentation tasks with hierarchical decoders, and suggests richer patch statistics, region-aware or topology-aware distributions, semi-supervised or self-training regimes, cross-domain KD, and integration with classic response-based KD at the final layer if logits are available (Lin, 3 Sep 2025). This suggests that DSKD is a methodological family whose concrete design depends on the representational structure of the task.

Taken together, the sources establish DSKD as a hierarchical distillation paradigm with two defining properties: teacher information is supplied at multiple depths, and auxiliary supervision is removed after training so that deployment cost remains unchanged (Luo et al., 2022, Lin, 3 Sep 2025). In classification, the paradigm is instantiated through auxiliary classifiers and feature alignment with adaptive weighting; in coronary artery segmentation, it is instantiated through side-output distribution matching and pixel-wise self-distillation. The common thread is that hierarchical supervision is treated as a mechanism for improving optimization, robustness, and generalization by shaping intermediate representations rather than only final predictions.