Sub-logit Decoupled Distillation (SDD)

• Sub-logit Decoupled Distillation (SDD) is a knowledge distillation method that partitions global logits into multiple sub-vectors for granular and specialized supervision.
• It employs dedicated loss functions, including KL divergence and orthogonality constraints, to ensure robust and diverse knowledge transfer across heterogeneous architectures.
• Empirical evaluations show significant accuracy improvements, such as a +4.1% gain on CIFAR-100 and consistent enhancements on ImageNet and fine-grained datasets.

Sub-logit Decoupled Distillation (SDD) is a paradigm within knowledge distillation (KD) that enhances logit-based transfer by decomposing high-dimensional logits into multiple sub-logit vectors. This strategy enables more granular supervision, facilitates knowledge transfer in both homogeneous and heterogeneous architecture settings, and improves robustness to sample ambiguity. SDD has been realized in various frameworks, most notably as a component of Heterogeneous Complementary Distillation (HCD) (Xu et al., 14 Nov 2025), Scale Decoupled Distillation (Luo, 20 Mar 2024), and through principled analysis in decoupled logit KD literature (Zhao et al., 2022).

1. Motivation and Conceptual Foundations

Traditional KD typically matches the global soft logit vector—from global average pooled features—between a pretrained teacher and a student network. This approach, while effective for homogeneous teacher-student pairs, faces limitations:

• The global logit combines evidence from disparate spatial regions, causing semantic entanglement and ambiguous transfer, especially for fine-grained distinctions or spatially complex tasks.
• For heterogeneous pairs (e.g., vision transformer to CNN), structural disparities in feature representations exacerbate the misalignment.

SDD addresses these issues by partitioning the global (or shared) logits into multiple sub-logits, enabling:

• Specialized supervision per sub-logit, reducing transfer difficulty.
• Richer, more diverse knowledge transfer compared to monolithic logit alignment.
• Enhanced handling of spatial and semantic ambiguity by transferring localized or complementary knowledge separately (Luo, 20 Mar 2024, Xu et al., 14 Nov 2025).

2. Formalization and Algorithmic Design

SDD can be instantiated as follows (abstracting from (Xu et al., 14 Nov 2025, Luo, 20 Mar 2024)):

Let $\mathbf{z} \in \mathbb{R}^C$ be the teacher's global logit. SDD partitions $\mathbf{z}$ into $n$ sub-logits, $\{\mathbf{z}^{(i)}\}_{i=1}^n$, with each $\mathbf{z}^{(i)} \in \mathbb{R}^{C_i}$, $\sum_i C_i = C$. For each sub-logit—possibly corresponding to specific spatial partitions, semantic groupings, or feature blocks—a dedicated loss constrains the student's corresponding sub-logit or fused vector.

A general distillation loss under SDD is:

$$\mathcal{L} = \mathcal{L}_{\mathrm{CE}} + \lambda_{KD}\, \mathrm{KL}(\sigma(\mathbf{t}/\tau)\|\sigma(\mathbf{z}^{s}/\tau)) + \beta \sum_{i=1}^n \mathrm{KL}(\sigma(\widetilde{\mathbf{Z}^{(i)}}/\tau)\| \sigma(\mathbf{z}^{s}/\tau)) + \omega\,\mathcal{L}_{\mathrm{OL}}$$

where $\mathbf{t}$ is the teacher logit, $\mathbf{z}^s$ is the student logit, $\widetilde{\mathbf{Z}^{(i)}}$ is a fusion (additive or weighted) of the $i$-th sub-logit and the teacher logit, $\mathcal{L}_{\mathrm{OL}}$ is an orthogonality loss for sub-logit diversity, and $\lambda_{KD}, \beta, \omega$ are loss weights.

Pseudocode for a typical SDD pipeline (Xu et al., 14 Nov 2025):

1. Generate shared logits by concatenating teacher features and student intermediate features via a Complementary Feature Mapper (CFM).
2. Partition the logits into $n$ sub-logits.
3. Optionally fuse each sub-logit with teacher's logit.
4. Impose KL-divergence on each sub-logit and the overall global logit.
5. Enforce sub-logit diversity using orthogonality loss, typically by normalizing and minimizing off-diagonal dot products between non-ground-truth channels.
6. Aggregate all losses and update the student.

3. Consistent and Complementary Knowledge in Sub-Logits

SDD as instantiated in the scale-decoupled framework (Luo, 20 Mar 2024) further decomposes each sub-logit into:

• Consistent component: The logit dimension(s) aligned with the teacher's global predicted class $c^*$.
• Complementary component: All other class dimensions, capturing sample ambiguity and "dark knowledge."

The loss is given by:

$$L_{SDD} = L_{CE}(y, z_S) + \alpha \left( L_{cons} + \beta \cdot L_{comp} \right)$$

• $L_{cons}$ penalizes the discrepancy in the consistent sub-logit channels.
• $L_{comp}$ upweights the error on complementary channels (typically $\beta>1$), regularizing the student to respect ambiguity and not overfit on ambiguous samples.

This decoupling allows the student to inherit both semantic certainty and calibrated uncertainty from the teacher. Empirically, fusion of both consistent and complementary signals delivers the strongest performance (Luo, 20 Mar 2024).

4. Implementation Considerations and Computational Efficiency

Key SDD instantiations exhibit low architectural overhead and computational cost:

Method	Teacher-Student Scope	Additional Classifiers	Sub-logit Partitioning	Training Overhead
HCD+SDD (Xu et al., 14 Nov 2025)	Penultimate+Intermed.	No	CFM, then reshape/split	+10–15%
Scale SDD (Luo, 20 Mar 2024)	All feature locations	No	Multi-scale pooling; shared W	+0–1 ms/batch

• CFM modules are lightweight fully-connected projections.
• Sub-logit orthogonality and partitioning can be implemented efficiently in parallel.
• Overhead is modest compared to feature-based KD methods (feature contrastive KD is typically ~3x slower) (Luo, 20 Mar 2024).
• Hyperparameters (number of sub-logits $n$, weights $\beta$, $\omega$, temperature $\tau$) require per-task tuning, often via ablation.

5. Empirical Results Across Settings

SDD and variants have demonstrated significant empirical gains on standard benchmarks:

• CIFAR-100 heterogeneous KD (e.g., Swin-Tiny→ResNet-18): SDD achieves 82.8% Top-1 accuracy, outperforming vanilla KD (78.7%) and SDD without orthogonality (82.3%) (Xu et al., 14 Nov 2025).
• ImageNet-1K: SDD yields consistent gains (0.4–0.8% Top-1) over strong logit and feature KD baselines (Xu et al., 14 Nov 2025).
• Fine-grained datasets: CUB-200 and Aircraft see increases of +1–4% Top-1 (Luo, 20 Mar 2024, Xu et al., 14 Nov 2025).
• Ablations: Increasing the number of sub-logits improves accuracy up to an optimal value ($n=4$ for CIFAR-100, $n=2$ or $4$ for ImageNet) (Xu et al., 14 Nov 2025).
• Training cost: Minimal compared to feature-based methods; SD-KD matches global KD in per-batch time and remains $>3\times$ faster than feature contrastive approaches (Luo, 20 Mar 2024).

Dataset/Setting	Vanilla KD	SDD Variant (Best)	Gain
CIFAR-100 (Het.)	78.7%	82.8%	+4.1%
ImageNet-1K	Baseline	Baseline+0.4–0.8%	+0.4–0.8%
CUB-200	56.09%	60.51%	+4.42%

6. Theoretical and Practical Significance

SDD provides several key advantages:

• Granular Supervision: Partitioning logits addresses the bottleneck of global, entangled knowledge transfer, particularly when student-teacher feature spaces are misaligned or structurally heterogeneous (Xu et al., 14 Nov 2025).
• Diversity and Robustness: Orthogonality regularization encourages specialization, preventing redundant knowledge transfer and enhancing generalization (Xu et al., 14 Nov 2025).
• Ambiguity Regularization: By explicitly upweighting ambiguous (complementary) logit channels, SDD enables students to better manage hard or borderline samples (Luo, 20 Mar 2024).
• Modularity: SDD is compatible with decoupled logit KD formulations (e.g., DKD (Zhao et al., 2022)), can be fused with feature-based methods, and requires limited code changes.

7. Limitations and Future Directions

• Hyperparameter Sensitivity: Performance depends on careful tuning of sub-logit count, complementary weight $\beta$, and orthogonality loss weight $\omega$; optimal values may be dataset- and architecture-dependent (Xu et al., 14 Nov 2025, Luo, 20 Mar 2024).
• Uniform Partitioning: Current SDD splits sub-logits uniformly; adaptive or semantic-aware partitioning may further improve efficiency.
• Task Scope: While SDD excels in classification and fine-grained recognition, extending SDD to domains reliant on spatial or localization cues (e.g., detection) may require augmentation or integration with feature-based signals (Luo, 20 Mar 2024).
• Dynamic Weighting: Future research aims to automate or learn balancing weights for sub-logit components and losses on a per-sample or class basis.

SDD represents an efficient and principled enhancement to logit-level distillation, bridging the gap between conventional logit-KD efficiency and feature-based KD expressivity (Zhao et al., 2022, Luo, 20 Mar 2024, Xu et al., 14 Nov 2025).