Unified Knowledge Distillation Framework

• Unified Knowledge Distillation Framework is a teacher–student approach that merges multiple supervision signals using modular pipelines and unified loss formulations.
• It leverages knowledge aggregation, representation homogenization, and relation preservation to align intermediate features and logits effectively.
• The framework addresses heterogeneous architectures and cross-modal tasks by unifying objectives, optimizing key metrics such as classification accuracy and detection mAP.

Searching arXiv for papers on unified knowledge distillation frameworks and closely related variants. Searching arXiv for "unified knowledge distillation framework" and related terms. Unified knowledge distillation frameworks are teacher–student formulations that consolidate multiple supervision signals into a single training program, a shared objective, or an explicitly modularized distillation pipeline. In recent work, this unification has been realized in several technically distinct ways: by transforming intermediate and final representations into a common distributional view, by expressing feature distillation as distribution matching over push-forward measures, by decomposing classical KD into normalized target and non-target components, by factorizing relation-based KD into affinity, normalization, and loss modules, and by extending the same design principle to heterogeneous architectures, cross-modal retrieval, pathology foundation models, medical imaging, industrial ranking, and audio reasoning (Huang et al., 2024, Montesuma, 2 Apr 2025, Yang et al., 2023, Li et al., 2022, Jiang et al., 13 Feb 2026).

1. Meanings of “unified” in the literature

The phrase does not denote a single canonical algorithm. In "Harmonizing knowledge Transfer in Neural Network with Unified Distillation" (Huang et al., 2024), unification means that intermediate features from different stages are first fused by a stacked Adaptive Features Fusion module and then converted into diagonal-Gaussian parameters, so that feature transfer and logits transfer can both be enforced through KL-based constraints. In "KD$^{2}$M: An unifying framework for feature knowledge distillation" (Montesuma, 2 Apr 2025), unification means that feature-based KD is written as standard supervised learning plus a generic distribution discrepancy $\mathbb{D}(g_{S\sharp}P,g_{T\sharp}P)$, with Wasserstein, KL, or MMD choices recovering different existing methods. In "From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels" (Yang et al., 2023), unification means expressing teacher-guided KD and teacher-free self-KD through a shared decomposition of target and non-target supervision.

A second usage emphasizes modular decomposition. "Towards a Unified View of Affinity-Based Knowledge Distillation" (Li et al., 2022) explicitly defines relation-based KD through three modules—affinity, normalization, and loss—and studies $4 \times 5 \times 4 = 80$ combinations. In this sense, unification is not a single loss but a design space with interchangeable components. A third usage is application-specific: DETRDistill unifies logits, object-centric features, and query assignment for set-based detectors (Chang et al., 2022), BEV-LGKD unifies soft-label, depth, and LiDAR-guided BEV feature transfer for camera-only 3D detection (Li et al., 2022), and MLLMEmbed-ReID unifies cloud adaptation and edge distillation for RGB, infrared, sketch, and text re-identification (Jiang et al., 13 Feb 2026).

This variety suggests that “unified” has become a structural descriptor rather than a narrow methodological label. Depending on context, it may refer to a common probabilistic parameterization, a common discrepancy functional, a common multi-loss recipe, a common representation space, or a common deployment pipeline.

2. Core patterns of unification

A recurring pattern is knowledge aggregation before alignment. UniKD aggregates all $L$ intermediate features into a single fused representation $R$ by top-down AFF, predicts Gaussian mean and log-variance from $R$, and aligns teacher and student through a closed-form KL divergence between diagonal Gaussians, while still retaining logits-based KL and hard-label CE (Huang et al., 2024). GPFM combines expert knowledge distillation from UNI, Phikon, and CONCH with self-knowledge distillation via local–global alignment akin to DINO, so that external expert supervision and internal consistency are optimized jointly (Ma et al., 2024). UKDSL combines supervised BCE on labeled impressions, teacher-guided BCE on labeled data, and teacher-guided BCE on much larger unlabeled consideration traffic in one loss for cascade ranking systems (Eghbalzadeh et al., 5 Feb 2025).

A second pattern is representation homogenization. UHKD applies FFT, Gaussian-based frequency-domain filtering, and pooling on teacher features, while the student is mapped by a learnable Feature Alignment Module into the same frequency-domain shape, after which MSE and logit-level KL are jointly optimized (Yu et al., 28 Oct 2025). FOFA addresses heterogeneous CNN–ViT–MLP transfer by combining Region-Aware Attention on the student side with Adaptive Feedback Prompts on the teacher side, so that student features are re-aggregated into the teacher’s view and teacher features become more student-friendly (Lin et al., 15 Jan 2025). MLLMEmbed-ReID uses instructional prompting to force Qwen2-VL embeddings from RGB, IR, sketch, and text into a shared $\mathbb{R}^d$ space before distillation to a CLIP-based edge model (Jiang et al., 13 Feb 2026).

A third pattern is relation preservation. mAKD formalizes relation-based KD through explicit affinity matrices and normalized discrepancies (Li et al., 2022). DETRDistill preserves query-level prediction structure, object-centric feature structure, and query–GT assignment structure within the DETR paradigm (Chang et al., 2022). BEV-LGKD restricts feature transfer to LiDAR-guided foreground regions in BEV, rather than distilling global RGB-heavy features indiscriminately (Li et al., 2022). The face-recognition unified KD framework combines Instance-Level Embedding Distillation and Relation-Based Pairwise Similarity Distillation so that individual embedding alignment and geometric relationship preservation are optimized simultaneously (Mishra et al., 15 Aug 2025).

Framework	What is unified	Main ingredients
UniKD (Huang et al., 2024)	Intermediate and final knowledge into a distribution-level view	AFF, FDP, $\mathcal{L}_{FL}$, $\mathcal{L}_{L}$
KD$^{2}$M (Montesuma, 2 Apr 2025)	Feature KD as classification plus distribution discrepancy	$\mathbb{D}(g_{S\sharp}P,g_{T\sharp}P)$0
mAKD (Li et al., 2022)	Relation KD as modular composition	affinity, normalization, loss
UHKD (Yu et al., 28 Oct 2025)	Heterogeneous features in frequency space	FTM, FAM, MSE, KL, CE
MLLMEmbed-ReID (Jiang et al., 13 Feb 2026)	Cloud teacher adaptation and edge distillation across four modalities	prompting, LoRA-SFT, PCM, FR

3. Objective functions and optimization structure

The mathematical core of unified KD frameworks is usually an additive objective in which different knowledge sources are balanced by explicit coefficients. In UniKD, the student is trained by

$\mathbb{D}(g_{S\sharp}P,g_{T\sharp}P)$1

where $\mathbb{D}(g_{S\sharp}P,g_{T\sharp}P)$2 is the KL divergence between teacher and student diagonal-Gaussian feature distributions predicted from fused intermediate representations, and $\mathbb{D}(g_{S\sharp}P,g_{T\sharp}P)$3 is the logits-based KD loss (Huang et al., 2024). KD$\mathbb{D}(g_{S\sharp}P,g_{T\sharp}P)$4M generalizes the feature-based case to

$\mathbb{D}(g_{S\sharp}P,g_{T\sharp}P)$5

thereby subsuming empirical Wasserstein, class-conditional Wasserstein, joint Wasserstein, Gaussian-Wasserstein, Gaussian KL, and MMD instantiations (Montesuma, 2 Apr 2025).

Other frameworks unify by reparameterizing the classical KD loss itself. NKD normalizes the non-target distributions over $\mathbb{D}(g_{S\sharp}P,g_{T\sharp}P)$6 so that the non-target CE can fully align teacher and student, and combines this with the original hard-label cross-entropy through

$\mathbb{D}(g_{S\sharp}P,g_{T\sharp}P)$7

USKD uses the same target/non-target loss form but replaces teacher signals with student-derived soft target labels and Zipf-law-based non-target labels (Yang et al., 2023). In mAKD, the unified objective is

$\mathbb{D}(g_{S\sharp}P,g_{T\sharp}P)$8

with

$\mathbb{D}(g_{S\sharp}P,g_{T\sharp}P)$9

which makes the relation-based KD pipeline explicitly compositional (Li et al., 2022).

Task-specific unified frameworks extend the same pattern. DETRDistill sums Hungarian-matching logits distillation across decoder stages, target-aware feature distillation, and query-prior assignment distillation on top of the standard detection loss (Chang et al., 2022). MLLMEmbed-ReID uses

$4 \times 5 \times 4 = 80$0

where $4 \times 5 \times 4 = 80$1 projects teacher and student features onto the top-$4 \times 5 \times 4 = 80$2 principal axes from the teacher SVD and $4 \times 5 \times 4 = 80$3 preserves Gram-matrix relations (Jiang et al., 13 Feb 2026). UHKD combines feature-level MSE in the frequency domain with softened KL on logits and CE on labels,

$4 \times 5 \times 4 = 80$4

typically with $4 \times 5 \times 4 = 80$5 and $4 \times 5 \times 4 = 80$6 (Yu et al., 28 Oct 2025).

These formulations show that unification rarely eliminates multi-objective optimization. Instead, it reorganizes multi-source supervision so that disparate signals are made commensurate—through a common divergence family, a common representation space, or a common training loop.

4. Heterogeneity, modality, and deployment

A central technical driver of unified KD is the need to distill across incompatible representation geometries. Heterogeneous KD papers make this explicit. FOFA identifies “view-mismatch” and “teacher-unawareness” as the two main obstacles in feature distillation across CNNs, ViTs, and MLPs, and addresses them through Region-Aware Attention and Adaptive Feedback Prompts (Lin et al., 15 Jan 2025). UHKD states that most existing KD methods degrade in heterogeneous scenarios, especially when intermediate features are involved, and proposes frequency-domain alignment to alleviate semantic discrepancies (Yu et al., 28 Oct 2025). The audio reasoning framework similarly treats source-wise and layer-wise discrepancies as distinct axes, distilling from a textual teacher for reasoning and from a frozen acoustic teacher for modality-consistent hidden-state supervision (Yang et al., 23 Sep 2025).

Cross-modal and cross-domain settings further broaden the meaning of unification. MLLMEmbed-ReID adapts a Qwen2-VL teacher with hierarchical LoRA-SFT and instructional prompting so that RGB, IR, sketch, and text are embedded in a shared space, then distills to a CLIP-based student for edge deployment (Jiang et al., 13 Feb 2026). BEV-LGKD uses LiDAR only at training time to generate foreground and view-dependent BEV masks that guide KD between camera-only models, while separately distilling depth because “the quality of depth estimation is crucial for BEV perception” (Li et al., 2022). The multi-dataset medical framework uses source teachers trained with domain-adversarial encoders, cross-attention fusion into a joint teacher, and multi-level feature, contrastive, and cosine alignment to a compact task-specific student across segmentation, classification, and detection (Ciprian-Mihai et al., 2 May 2026).

Some frameworks unify not only model components but also data regimes. UKDSL is built for settings in which labeled impression data are much smaller than unlabeled consideration data, and where multi-stage ranking induces calibration bias after top-$4 \times 5 \times 4 = 80$7 selection. Its unified objective reuses late-stage predictions or foundation-model predictions as teacher signals on both labeled and unlabeled traffic (Eghbalzadeh et al., 5 Feb 2025). The digital staining framework similarly unifies unpaired and paired-but-misaligned training: both use the same teacher–student pipeline, while the paired-but-misaligned regime adds a Learning-to-Align module with content consistency and smoothness losses (Xu et al., 14 Apr 2025).

This suggests that recent unified KD research increasingly targets the mismatch itself—architectural, modal, geometric, or operational—as the primary design object.

5. Empirical behavior across tasks

The empirical record is broad but task-dependent. On CIFAR-100, UniKD reports top-1 accuracy gains such as ResNet56$4 \times 5 \times 4 = 80$8ResNet20: student alone $4 \times 5 \times 4 = 80$9, KD $L$0, DKD $L$1, FitNet $L$2, and “Ours” $L$3; ResNet110$L$4ResNet32: student alone $L$5 and “Ours” $L$6 (Huang et al., 2024). KD$L$7M reports that all variants significantly outperform the ResNet-18 student-only baseline on SVHN, CIFAR-10, and CIFAR-100, with class-conditional Wasserstein on empirical activations giving the best overall gains (Montesuma, 2 Apr 2025). NKD boosts ImageNet top-1 accuracy of ResNet18 from $L$8 to $L$9 with a ResNet-34 teacher, while USKD yields gains such as MobileNet $R$0 and DeiT-Tiny $R$1 in self-KD (Yang et al., 2023).

For detection, DETRDistill consistently improves various DETRs by more than $R$2 mAP on COCO. In the reported identical-stage setting, AdaMixer improves from $R$3 AP to $R$4 AP, Deformable DETR from $R$5 AP to $R$6 AP, and Conditional DETR from $R$7 AP to $R$8 AP; students with fewer encoder/decoder stages gain between $R$9 and $R$0 AP (Chang et al., 2022). BEV-LGKD on nuScenes raises a ResNet-18 student from NDS $R$1, mAP $R$2 to NDS $R$3, mAP $R$4, while improving depth AbsRel from $R$5 to $R$6 and depth $R$7 from $R$8 to $R$9 (Li et al., 2022). In medical detection and other medical tasks, the cross-domain joint-teacher framework reports gains such as DeepLesion mAP@[.50:.95] $\mathbb{R}^d$0 for Faster R-CNN and $\mathbb{R}^d$1 for DETR over baseline (Ciprian-Mihai et al., 2 May 2026).

In heterogeneous and cross-modal transfer, FOFA reports an average absolute accuracy gain of $\mathbb{R}^d$2 over student-from-scratch on CIFAR-100 and $\mathbb{R}^d$3 on ImageNet-1K, with ConvNeXt-T$\mathbb{R}^d$4ResMLP-S12 reaching $\mathbb{R}^d$5 versus $\mathbb{R}^d$6 scratch (Lin et al., 15 Jan 2025). UHKD reports CIFAR-100 average student accuracy $\mathbb{R}^d$7, versus scratch $\mathbb{R}^d$8, OFA/FBT $\mathbb{R}^d$9, and on ImageNet-1K an average $\mathcal{L}_{FL}$0 versus scratch $\mathcal{L}_{FL}$1 (Yu et al., 28 Oct 2025). MLLMEmbed-ReID reports on QrCM-ReID benchmarks sketch$\mathcal{L}_{FL}$2RGB cloud R1 $\mathcal{L}_{FL}$3, mAP $\mathcal{L}_{FL}$4 and edge R1 $\mathcal{L}_{FL}$5, mAP $\mathcal{L}_{FL}$6; IR$\mathcal{L}_{FL}$7RGB cloud R1 $\mathcal{L}_{FL}$8, mAP $\mathcal{L}_{FL}$9 and edge R1 $\mathcal{L}_{L}$0, mAP $\mathcal{L}_{L}$1; text$\mathcal{L}_{L}$2RGB cloud R1 $\mathcal{L}_{L}$3, mAP $\mathcal{L}_{L}$4 and edge R1 $\mathcal{L}_{L}$5, mAP $\mathcal{L}_{L}$6 (Jiang et al., 13 Feb 2026).

Large-scale application papers also show that unification can be operational rather than purely algorithmic. UKDSL reports offline reduction of the pre-ranking vs. ranking calibration gap on unlabeled traffic from $\mathcal{L}_{L}$7, approximately $\mathcal{L}_{L}$8 NE reduction on unlabeled traffic, a further $\mathcal{L}_{L}$9 NE reduction on impressions from SSL-from-foundation, and online lifts of $^{2}$0 overall CTR and $^{2}$1 RPM with no measurable latency increase (Eghbalzadeh et al., 5 Feb 2025). GPFM reports an average rank of $^{2}$2, with $^{2}$3 tasks ranked $^{2}$4, whereas UNI attains an average rank of $^{2}$5 with only $^{2}$6 tasks ranked $^{2}$7 on the established pathology benchmark (Ma et al., 2024).

6. Limitations, misconceptions, and open directions

Several papers explicitly caution against reducing unified KD to a simplistic “more losses are better” heuristic. UniKD argues that pure feature-based KD enforces hard pixel-level constraints that may hamper student expressivity, pure logits-based KD ignores rich multi-scale semantics in intermediate layers, and simple hybrids can yield conflicting gradient signals and unstable early training; it also notes that its distribution-level constraint gives smaller gains on dense tasks than on classification (Huang et al., 2024). SOKD shows that combining offline KD and peer-teaching can improve both student and teacher, but it still requires a pre-trained large teacher, introduces a Knowledge Bridge Module during training, and leaves hyper-parameter tuning as an open practical issue (Liu et al., 2021).

A common misconception is that unification requires either a single teacher or a single loss. The recent literature shows otherwise. GPFM uses three external expert teachers plus self-distillation (Ma et al., 2024). The audio reasoning framework uses a textual teacher and a frozen acoustic teacher, with top-layer and intermediate-layer Jensen–Shannon constraints (Yang et al., 23 Sep 2025). UKDSL combines supervised loss, teacher-guided labeled KD, and teacher-guided unlabeled KD in the same production-scale training loop (Eghbalzadeh et al., 5 Feb 2025). The medical multi-dataset framework uses multiple source teachers, a target teacher, a fused joint teacher, and then student distillation (Ciprian-Mihai et al., 2 May 2026).

Open directions are stated across several papers. UHKD proposes extension to object detection, segmentation, and video classification through the same frequency-domain alignment principle (Yu et al., 28 Oct 2025). The digital staining framework suggests stronger perceptual losses, transformer backbones, other contrast modalities, and 3D volume alignment (Xu et al., 14 Apr 2025). GPFM highlights the need for more external validation, more diverse experts, multimodal pathology foundation models, and long-sequence whole-slide encoders (Ma et al., 2024). SOKD identifies multi-teacher extension, convergence analysis in the semi-online regime, and automated Knowledge Bridge Module design as future directions (Liu et al., 2021).

Taken together, these works indicate that unified KD is best understood as a research program for coordinating heterogeneous supervision sources under a coherent optimization architecture. The main unresolved question is not whether knowledge can be unified, but which representation, discrepancy, and coupling strategy yields the most faithful transfer under a given combination of modality gap, architectural mismatch, label scarcity, and deployment constraint.