Knowledge Distillation Detection

• Knowledge distillation detection is the task of determining if a student model was derived from a teacher by analyzing structural habits and internal routing patterns.
• Methodological families range from white-box MoE signature analysis to proxy-driven and API-only black-box approaches for robust teacher attribution.
• Empirical benchmarks show that structurally informed detection methods outperform basic response similarity, ensuring accurate model provenance verification.

Knowledge distillation detection is the task of determining whether a student model was distilled from a particular teacher rather than trained independently. In recent work, it is formulated as a model provenance problem under practical access constraints, including white-box access to internal routing signals, open student weights with teacher API access, and fully black-box teacher–student comparisons mediated by proxy models. The topic is motivated by unauthorized replication, “model theft” through API access or released checkpoints, and the risk that repeated distillation from a small set of teachers reduces model diversity (Li et al., 19 Oct 2025, Shi et al., 2 Oct 2025).

1. Formal problem statement and operational settings

A canonical formulation treats the problem as a binary hypothesis test between

$$H_1:~ f_S \in \mathrm{KD}(f_T) \quad\text{vs.}\quad H_0:~ f_S \notin \mathrm{KD}(f_T),$$

where $f_T$ is a suspected teacher and $f_S$ is a suspected student (Li et al., 19 Oct 2025). In a more practical attribution setting, the task is posed as multiple-choice classification over candidate teachers $\{f^{(1)},\dots,f^{(K)}\}$, with a detector $\mathcal{D}$ mapping the student $g_\theta$ to one of the teacher indices, and the predicted source given by

$$k^* = \arg\max_{k \in \{1,\dots,K\}} S(g_\theta, f^{(k)}, X),$$

where $X$ is a constructed query set and $S$ is a student–teacher alignment score (Shi et al., 2 Oct 2025).

The operational setting matters. One line of work assumes white-box access to sparse MoE routing, allowing direct comparison of internal expert activation patterns. Another targets a practical “student weights + teacher API” regime, where the teacher’s parameters, the original distillation data, and the training procedure are unavailable, so the detector must rely on synthesized inputs and response statistics. A black-box extension is also studied through auxiliary sparse proxy models, allowing comparison even when neither side exposes internal states (Li et al., 19 Oct 2025, Shi et al., 2 Oct 2025).

A binary decision can be recovered from the multiple-choice formulation by thresholding a teacher-specific score, but the multiple-choice setting avoids calibrating an absolute threshold and provides direct teacher attribution. This suggests that knowledge distillation detection is not only a yes/no authenticity test, but also a teacher identification problem.

2. What distillation transfers, and why it can be detected

The premise of distillation detection is that KD transfers more than final-task accuracy. In MoE LLMs, recent work argues that KD transfers “structural habits,” especially internal routing patterns, including expert specialization and expert collaboration. These routing habits create fingerprints that persist through the distillation process and can be more robust than surface-form responses (Li et al., 19 Oct 2025).

A broader architectural view comes from the KD literature for object detection, which distinguishes backbone-level, neck-level, head-level, and RPN/RoI-level distillation for CNN detectors, and query-level, feature-level, and logit-level distillation for Transformer detectors (Golizadeh et al., 5 Aug 2025). DETR-specific frameworks make this concrete: DETRDistill combines Hungarian-matching logits distillation, target-aware feature distillation, and query-prior assignment distillation, while KD-DETR introduces consistent distillation points sampling through specialized object queries (Chang et al., 2022, Wang et al., 2022).

This diversity of transferred signals is important for provenance analysis. In one-stage detectors, for example, Rank Mimicking distills the rank distribution over positive candidate boxes, and Prediction-guided Feature Imitation weights feature imitation by prediction mismatch; both target behavior that is more specific than ordinary soft labels (Li et al., 2021). Task Integration Distillation explicitly integrates classification and regression outputs into its feature weighting, and UET introduces Monte Carlo-dropout-based teacher uncertainty into feature transfer (Su et al., 2024, Yi et al., 2024). Structural KD replaces pointwise $\ell_p$ feature matching with SSIM-based structural similarity, and global-knowledge methods project instances into shared prototype coordinates across teacher and student spaces (Rijk et al., 2022, Tang et al., 2022).

A plausible implication is that provenance detectors should not be limited to output similarity alone. If KD can imprint routing habits, query assignment behavior, ranking structure, uncertainty-conditioned feature patterns, and prototype-space coordinates, then detection methods that probe internal or induced latent structure should have a stronger signal than methods that inspect only verbal self-identification or raw response overlap.

3. Methodological families

Recent methods fall into three broad families that differ mainly by access regime and the type of signal they measure.

Access setting	Primary signal	Representative framework
White-box MoE	Expert specialization and collaboration	MoE expert signatures
Semi-/pure black-box	Proxy routing signatures via auxiliary distillation	Shadow-MoE
Open student weights + teacher API	Synthetic-query response alignment	Data-free score-based attribution

In the white-box MoE setting, routing is observed directly. For a sparse MoE layer $f_T$0, binary expert activation is defined by

$f_T$1

from which a domain-conditioned expert specialization profile and an expert collaboration matrix are estimated. Because expert identities can be permuted across independently trained models, similarity is measured with permutation-invariant Wasserstein distances over specialization and collaboration, and the final detection score is

$f_T$2

so larger values indicate stronger evidence of distillation (Li et al., 19 Oct 2025).

The black-box extension, Shadow-MoE, constructs sparse MoE proxy models for arbitrary targets by auxiliary distillation:

$f_T$3

The regularizer $f_T$4 is a load-balancing term that prevents expert collapse and encourages diverse routing. After training separate proxies for teacher and student, the same specialization and collaboration signatures are extracted and compared (Li et al., 19 Oct 2025).

The open-weight/API-only framework uses no original data. It synthesizes inputs, computes student–teacher alignment scores, and predicts the teacher with highest score. In image classification, a mixup-based generator is trained against the student using hard-label confidence and batch-normalization-statistics matching. In text-to-image generation, the empty string is used as a prompt because classifier-free-guidance diffusion models are explicitly exposed to unconditional generation during training. The scoring stage includes point-wise inverse-discrepancy scores,

$f_T$5

set-level scores such as Aligned Cosine Similarity for classification, and LPIPS or CKA for text-to-image generation (Shi et al., 2 Oct 2025).

4. Benchmarks, datasets, and empirical findings

The MoE-signature line of work establishes a benchmark with 280 prompts sampled from Tülu 3 SFT mixture, a black-box DeepSeek-R1 teacher, Moonlight-16B-A3B as the proxy architecture, OLMoE-1B-7B as the student architecture, and 18 student checkpoints spanning 9 domain-specific datasets under distilled and from-scratch conditions (Li et al., 19 Oct 2025). In the semi-black-box setting, Shadow-MoE reports average pairwise accuracy of 94%, versus 88% for Idiosyncrasies, 54% for BERT, 46% for Linear, and 0% for Self-identity. Per task, the reported accuracies are 75% for Code and 100% for Math, Science, and Puzzle. In the pure black-box setting, Shadow-MoE reaches 100% accuracy across all task sets (Li et al., 19 Oct 2025).

The same study reports that distilled students consistently show lower Wasserstein distance to the teacher proxy than scratch-trained students, with relative reductions of 4% to 20% for specialization and 2% to 19% for collaboration in the semi-black-box setting, and 11% to 62% for specialization and 11% to 46% for collaboration in the pure black-box setting (Li et al., 19 Oct 2025). Layer ablation shows that the last MoE layer is most informative: in the semi-black-box setting, first-layer routing gives 46% average accuracy, median-layer routing gives 85%, and last-layer routing gives 94% (Li et al., 19 Oct 2025).

The open-weight/API-only framework evaluates on CIFAR-10, ImageNet, and text-to-image generation, and reports improvements over the strongest baselines by 59.6% on CIFAR-10, 71.2% on ImageNet, and 20.0% for text-to-image generation (Shi et al., 2 Oct 2025). On CIFAR-10, the KL-based detector reaches 0.62 accuracy / 0.75 AUC at $f_T$6, 0.86 / 0.92 at $f_T$7, and 0.87 / 0.94 at $f_T$8. On ImageNet, it reaches 0.47 / 0.68 at $f_T$9, 0.66 / 0.89 at $f_S$0, and 0.75 / 0.92 at $f_S$1. On text-to-image generation, the LPIPS-based point-wise score yields 0.89 / 1.00 at $f_S$2, 0.97 / 1.00 at $f_S$3, and 1.00 / 1.00 at $f_S$4 (Shi et al., 2 Oct 2025).

These results collectively show that provenance signals survive under very different conditions: direct access to internal MoE routing, proxy-induced routing recovery, and purely response-level attribution from synthetic inputs. They also show that structurally informed methods outperform self-identity probing and simple embedding similarity baselines.

5. Relation to knowledge distillation in object detection

The phrase “knowledge distillation detection” can be confused with “knowledge distillation for object detection.” The latter is a mature model-compression literature rather than a provenance literature, but it is directly relevant because it enumerates the mechanisms through which student models inherit teacher structure (Golizadeh et al., 5 Aug 2025).

In CNN-based detection, transfer targets include backbone features, multi-scale neck representations, task heads, and region-level object features. In Transformer-based detection, transfer targets include queries, memory features, and Hungarian-aligned logits (Golizadeh et al., 5 Aug 2025). DETRDistill reports a total loss

$f_S$5

with $f_S$6, $f_S$7, and $f_S$8, explicitly combining matched logits, target-aware features, and assignment priors (Chang et al., 2022). KD-DETR shows that DETR distillation depends on “sufficient and consistent distillation points,” introduced via specialized object queries shared between teacher and student (Wang et al., 2022).

Other detection methods emphasize distinct inherited structures. DRKD distills pixel-wise and instance-wise relations to address foreground/background imbalance and weak small-object representations, improving Faster R-CNN based on ResNet50 from 38.4% to 41.6% mAP and RetinaNet based on ResNet50 from 37.4% to 40.3% mAP on COCO 2017 (Ni et al., 2023). Structural Knowledge Distillation replaces pixelwise $f_S$9 feature matching with SSIM and reports 40.1 AP for RetinaNet R50 and 40.9 AP for Faster R-CNN R50 on MSCOCO, outperforming $\{f^{(1)},\dots,f^{(K)}\}$0, $\{f^{(1)},\dots,f^{(K)}\}$1, and several state-of-the-art KD methods (Rijk et al., 2022). UET augments detector KD with teacher uncertainty estimated by MC dropout and reports 44.1% mAP on COCO for ResNet50-based GFL, surpassing the baseline by 3.9% (Yi et al., 2024).

This literature does not itself solve provenance detection, but it reveals what a provenance detector might seek. A plausible implication is that detector-oriented KD detection will likely require task-aware signals that span classification, localization, ranking, query assignment, feature structure, and uncertainty, rather than a single generic similarity metric.

6. Limitations, robustness, and open research directions

The recent provenance literature is explicit about the failure modes of earlier approaches. Identity-based KD detection can be defeated by prompt engineering, jailbreaks, or fine-tuning that changes surface responses while leaving distilled capabilities intact. Output-similarity detectors can incur high false positives because models trained on similar data may look alike even when no distillation occurred (Li et al., 19 Oct 2025). These limitations motivate structural or query-synthesis-based tests rather than self-report or naïve embedding overlap.

Current methods also have nontrivial assumptions. The MoE-signature approach is strongest for MoE-based models; extending it to dense architectures is expected to require additional structure cues. Shadow-MoE introduces approximation error because the proxy is not the original model, and the method is computationally heavier than simple output-similarity detectors, although it is reported to run in minutes for the stated configuration (Li et al., 19 Oct 2025). The open-weight/API-only framework assumes access to each candidate teacher’s API and to an open-weight student, and its authors note that when the KD influence parameter $\{f^{(1)},\dots,f^{(K)}\}$2, the student may resemble an independently trained model almost as much as the teacher, making detection unreliable (Shi et al., 2 Oct 2025).

Robustness analysis points in two directions. First, internal structure appears harder to evade than surface outputs: routing signatures remain informative under prompt-based evasion, and last-layer routing is substantially more discriminative than early-layer routing (Li et al., 19 Oct 2025). Second, carefully constructed probes matter: synthetic data combined with aligned scoring clearly outperforms OOD-filtered inputs or generic set-level similarity alone in the open-weight setting (Shi et al., 2 Oct 2025).

Future directions are already explicit in the literature. Proposed extensions include attention-head usage, dense-model signatures, reward-model-mediated or RL-based distillation, structural watermarks, and routing randomization (Li et al., 19 Oct 2025). The object-detection KD literature suggests an additional direction: provenance detectors tailored to detectors may need architecture-aware tests over backbone, neck, head, RPN/RoI, and query-level transfer channels (Golizadeh et al., 5 Aug 2025). This suggests that knowledge distillation detection will evolve from generic response comparison toward mechanism-specific forensic analysis of how students inherit teacher computation.