Progressive Knowledge Distillation

• Progressive Knowledge Distillation is a staged knowledge transfer method that uses intermediate targets to bridge the capacity gap between teacher and student models.
• It employs mechanisms such as adaptive target smoothing, checkpoint-following, and decomposed transfer to mitigate optimization instability and enhance data efficiency.
• In applications like knowledge graph completion, PKD achieves significant parameter reduction while preserving performance through progressive compression methods.

Searching arXiv for the cited PKD-related papers to ground the article. arxiv_search.query({"^^^^^^^^2search_query2 OR ti:\2^^^^^^^^"Progressive Distillation Based on Masked Generation Feature Method for Knowledge Graph Completion\"","start":^^^^2search_query2^^^^,"max_results":5}) Progressive Knowledge Distillation (PKD) denotes a family of knowledge distillation paradigms in which supervision is not transferred in a single step from a fixed, fully mature teacher to a compact student, but is instead organized as a sequence of intermediate targets, intermediate teachers, intermediate students, or intermediate training stages. Across the literature, this progression is used to mitigate the capacity gap between teacher and student, reduce optimization instability, and improve knowledge transfer under architectural mismatch, data scarcity, domain shift, or severe compression constraints. In the knowledge graph completion setting, PKD has been instantiated as a progressive, mask-aware compression procedure for pre-trained LLMs, where a pre-distilled teacher and multi-grade students are connected by masked feature alignment and stage-wise distillation (&&&^^^^2search_query2^^^^&&&). Related formulations include adaptive progressive teachers via approximate mirror descent (Shi et al., 2021), teacher- and data-progressive distillation for dense retrieval (Lin et al., 2022), checkpoint-following distillation along the teacher’s training trajectory (Rezagholizadeh et al., 2021), and class-, modality-, or network-decomposition-based curricula (Li et al., 30 May 2025, Soufleri et al., 2024, Zhang et al., 2021).

^^^^2id:(Fan et al., 2024) OR ti:\2^^^^. Conceptual scope and defining properties

PKD differs from conventional knowledge distillation in that the distillation target is progressive rather than static. Standard KD typically forces the student to match the output distribution of a strong, converged teacher model. Several papers identify this as problematic when the student is much less expressive, since the target may be too difficult and may induce poor local optima or inefficient transfer (Shi et al., 2021, Rezagholizadeh et al., 2021, Lin et al., 2022).

Within this literature, “progressive” has several precise meanings. One formulation trains a progressive teacher whose trajectory is explicitly shaped by the student’s current state, so that the target is both informative and reachable (Shi et al., 2021). Another formulation makes the student follow the teacher’s own training footprints across checkpoints, thereby addressing the checkpoint-search problem and smoothing the teacher–student difficulty gap (Rezagholizadeh et al., 2021). A third formulation uses a sequence of teachers or students with increasing or decreasing capacity, so that the transfer occurs through intermediate models rather than a single abrupt compression step (Lin et al., 2022, &&&^^^^2search_query2^^^^&&&, &&&^^^^2id:(Fan et al., 2024) OR ti:\2^^^^5&&&).

The recurring principle is that knowledge transfer is staged so that the complexity of transferred information better matches the student’s representational capacity. This suggests that PKD is less a single algorithm than a design pattern for restructuring the supervision path in distillation. In the supplied literature, this restructuring occurs along several axes: model size, teacher strength, training time, data difficulty, class subsets, modality hierarchy, and decomposed teacher subnetworks (&&&^^^^2search_query2^^^^&&&, Lin et al., 2022, Li et al., 30 May 2025, Soufleri et al., 2024, Zhang et al., 2021).

2. Recurrent mechanisms of progression

A central mechanism in PKD is stepwise transfer across adjacent difficulty levels. In the KGC method “Progressive Distillation Based on Masked Generation Feature Method for Knowledge Graph Completion,” the pre-trained LLM is compressed from ^^^^2id:(Fan et al., 2024) OR ti:\22^^^^ to 9, 6, and 3 Transformer encoder layers, and each lower-grade student is distilled from the previous higher-capacity model rather than directly from the most compressed endpoint (&&&^^^^2search_query2^^^^&&&). In dense retrieval, PROD uses a ^^^^2id:(Fan et al., 2024) OR ti:\22^^^^-layer dual-encoder teacher, then a ^^^^2id:(Fan et al., 2024) OR ti:\22^^^^-layer cross-encoder, and finally a 24-layer cross-encoder, while the student is initialized from the previous stage’s checkpoint (Lin et al., 2022). In object detection for UAV imagery, a senior teacher first supervises a junior teacher, and the junior teacher then supervises the lightweight student (Yao et al., 2024). In few-shot KD, the teacher network is progressively replaced by student blocks through dual-stage network grafting (Shen et al., 2020).

A second mechanism is adaptive target smoothing. ProKT updates the teacher by minimizing a weighted combination of ground-truth cross-entropy and proximity to the student:

PRESERVED_PLACEHOLDER_^^^^2search_query2^^^^

so the teacher moves toward better classification while remaining close to the student’s current state (Shi et al., 2021). Pro-KD similarly anneals the distillation temperature as the teacher evolves during training, with the student learning from the current checkpoint rather than from a single mature teacher (Rezagholizadeh et al., 2021).

A third mechanism is progressive expansion of the supervision domain. PROD supplements teacher progression with data progressive distillation, selecting “confusing” cases where the teacher succeeds and the student fails (Lin et al., 2022). Progressive Class-level Distillation ranks teacher–student logit differences and performs stage-wise bidirectional distillation over class groups rather than over the entire class ensemble at once (Li et al., 30 May 2025). In compressed video action recognition, PKD follows the modality hierarchy motion vectors PRESERVED_PLACEHOLDER_^^^^2id:(Fan et al., 2024) OR ti:\2^^^^ residuals $\rightarrow$ intra-frames, transferring increasingly rich supervisory signals to internal classifiers (Soufleri et al., 2024).

A fourth mechanism is decomposed or partial knowledge transfer. PWKD reconstructs the teacher into weight-sharing sub-networks with the same depth but increasing channel width, then trains the student from partial to whole knowledge across stages (Zhang et al., 2021). Curriculum extraction derives a comparable curriculum from the final teacher alone by progressively matching random projections of hidden representations from shallow to deep layers before final-output distillation (&&&3^^^^2id:(Fan et al., 2024) OR ti:\2^^^^&&&).

3. Progressive Knowledge Distillation for knowledge graph completion

In knowledge graph completion, PKD is instantiated by the PMD framework, described as a “Progressive Distillation Method based on Masked Generation Features” for compressing PLM-based KGC models while preserving performance (&&&^^^^2search_query2^^^^&&&). The method begins with a teacher model based on a pre-trained LLM such as BERT-base, strengthens that model by a pre-distillation phase, and then constructs multi-grade student models by progressively compressing the PLM. The example architecture reduces Transformer encoder depth from $12 \rightarrow 9 \rightarrow 6 \rightarrow 3$, with parameter counts $$210\text{M} \rightarrow 176\text{M} \rightarrow 133\text{M} \rightarrow 91\text{M}$$ (&&&^^^^2search_query2^^^^&&&).

The pre-distillation stage uses Masked Generation Feature Distillation (MGFD) to produce a stronger teacher. The motivation is that traditional feature distillation transfers only the unmasked representation, limiting information diversity. MGFD instead masks parts of the input sequence, including triples and descriptions, and aligns teacher and student feature vectors only at masked positions:

$$\mathcal{L}_{MGFD}= MSE\big(Mask(\mathcal{F}^{S}|\lambda) - Mask(\mathcal{F}^{T}|\lambda)\big).$$

Because the model is forced to reconstruct or infer masked parts, the transferred representation is described as richer and more generalizable (&&&^^^^2search_query2^^^^&&&).

The progressive distillation stage then trains each student grade using three supervision terms: cross-entropy on true labels, score regression to the teacher, and MGFD:

$$\mathcal{L} = (1 - \alpha - \beta)\, \mathcal{L}_{CE} + \alpha\, \mathcal{L}_{SCORE} + \beta\, \mathcal{L}_{MGFD}.$$

The masking rate $\lambda_g$ decreases with each grade, with the supplied example $[20\%, 10\%, 5\%, 0\%]$, so that the information quantity transferred is matched to student capacity (&&&^^^^2search_query2^^^^&&&).

On WN^^^^2id:(Fan et al., 2024) OR ti:\2^^^^8RR, the pre-distilled teacher PMD$_{12}$ achieves MRR PRESERVED_PLACEHOLDER_^^^^2id:(Fan et al., 2024) OR ti:\2search_query2^^^^ and Hits@^^^^2id:(Fan et al., 2024) OR ti:\2search_query2^^^^ PRESERVED_PLACEHOLDER_^^^^2id:(Fan et al., 2024) OR ti:\2id:(Fan et al., 2024) OR ti:\2^^^^, compared with a baseline MRR PRESERVED_PLACEHOLDER_^^^^2id:(Fan et al., 2024) OR ti:\22^^^^ and Hits@^^^^2id:(Fan et al., 2024) OR ti:\2search_query2^^^^ PRESERVED_PLACEHOLDER_^^^^2id:(Fan et al., 2024) OR ti:\2^^^^3. The progressive students report PMDPRESERVED_PLACEHOLDER_^^^^2id:(Fan et al., 2024) OR ti:\2^^^^4: MRR PRESERVED_PLACEHOLDER_^^^^2id:(Fan et al., 2024) OR ti:\2^^^^5, Hits@^^^^2id:(Fan et al., 2024) OR ti:\2search_query2^^^^ PRESERVED_PLACEHOLDER_^^^^2id:(Fan et al., 2024) OR ti:\2^^^^6, PMDPRESERVED_PLACEHOLDER_^^^^2id:(Fan et al., 2024) OR ti:\2^^^^7: MRR PRESERVED_PLACEHOLDER_^^^^2id:(Fan et al., 2024) OR ti:\2^^^^8, Hits@^^^^2id:(Fan et al., 2024) OR ti:\2search_query2^^^^ PRESERVED_PLACEHOLDER_^^^^2id:(Fan et al., 2024) OR ti:\2^^^^9, and PMD$\rightarrow$^^^^2search_query2^^^^: MRR $\rightarrow$^^^^2id:(Fan et al., 2024) OR ti:\2^^^^, Hits@^^^^2id:(Fan et al., 2024) OR ti:\2search_query2^^^^ $\rightarrow$2. The lowest-grade student has $\rightarrow$3 parameters, a $\rightarrow$4 reduction from the $\rightarrow$5 baseline (&&&^^^^2search_query2^^^^&&&).

The paper characterizes this as the first description-based KGC distillation method for PLM-based KGC rather than structure-based models. It also presents masked generation feature distillation and a progressive, mask-rate-aligned schedule as its main innovations. A plausible implication is that, in description-based KGC, feature richness and capacity matching are treated as coupled problems rather than separate ones.

4. Major variants across domains

PKD has been adapted to substantially different tasks, and the definition of “progression” changes accordingly. The following variants illustrate the breadth of the concept.

Domain	Progressive unit	Representative formulation
Dense retrieval	Teacher strength and data difficulty	PROD (Lin et al., 2022)
Vision and NLP classification/QA	Teacher checkpoints during training	Pro-KD (Rezagholizadeh et al., 2021)
Adaptive KD	Teacher targets projected into student region	ProKT (Shi et al., 2021)
UAV object detection	Senior teacher $\rightarrow$6 junior teacher $\rightarrow$7 student	Domain-invariant PKD (Yao et al., 2024)
Few-shot KD	Sequential block replacement and network grafting	Network Grafting (Shen et al., 2020)
Compressed video action recognition	Modality order MV $\rightarrow$8 R $\rightarrow$9 I-frame	PKD with WISE (Soufleri et al., 2024)
Class-level logit transfer	Stage-wise class groups and bidirectional refinement	PCD (Li et al., 30 May 2025)

In dense retrieval, PROD combines Teacher Progressive Distillation and Data Progressive Distillation. The student first distills from a ^^^^2id:(Fan et al., 2024) OR ti:\22^^^^-layer dual-encoder, then from a ^^^^2id:(Fan et al., 2024) OR ti:\22^^^^-layer cross-encoder, and finally from a 24-layer cross-encoder, while hard negatives and confusing subsets are introduced progressively. On MS MARCO Passage, a 6-layer PROD student reaches MRR@^^^^2id:(Fan et al., 2024) OR ti:\2search_query2^^^^ $12 \rightarrow 9 \rightarrow 6 \rightarrow 3$^^^^2search_query2^^^^, compared with $12 \rightarrow 9 \rightarrow 6 \rightarrow 3$^^^^2id:(Fan et al., 2024) OR ti:\2^^^^ for RocketQA v2 and $12 \rightarrow 9 \rightarrow 6 \rightarrow 3$2 for CL-DRD (Lin et al., 2022).

In Pro-KD, progression is temporal: the student learns from teacher checkpoints as the teacher itself is trained. This addresses both the checkpoint-search problem and the capacity-gap problem. Reported results include CIFAR-^^^^2id:(Fan et al., 2024) OR ti:\2search_query2^^^^ accuracy $12 \rightarrow 9 \rightarrow 6 \rightarrow 3$3 and CIFAR-^^^^2id:(Fan et al., 2024) OR ti:\2search_query2search_query2^^^^ accuracy $12 \rightarrow 9 \rightarrow 6 \rightarrow 3$4 for Pro-KD, compared with $12 \rightarrow 9 \rightarrow 6 \rightarrow 3$5 and $12 \rightarrow 9 \rightarrow 6 \rightarrow 3$6 for Annealing-KD in the supplied ResNet-^^^^2id:(Fan et al., 2024) OR ti:\2id:(Fan et al., 2024) OR ti:\2search_query2^^^^ to ResNet-8 setting (Rezagholizadeh et al., 2021).

In ProKT, progression is local and adaptive rather than checkpoint-based. The teacher is updated toward the labels while remaining close to the student, and the student then learns from this updated, easier-to-reach teacher. The paper states that ProKT consistently outperforms vanilla KD and RCO/TA across vision and NLP settings, and reports, for example, CIFAR-^^^^2id:(Fan et al., 2024) OR ti:\2search_query2search_query2^^^^ accuracy $12 \rightarrow 9 \rightarrow 6 \rightarrow 3$7 for MobileNetV2 distilled from ResNet5^^^^2search_query2^^^^, compared with $12 \rightarrow 9 \rightarrow 6 \rightarrow 3$8 for KD and $12 \rightarrow 9 \rightarrow 6 \rightarrow 3$9 for RCO (Shi et al., 2021).

In UAV-based object detection, PKD is paired with FFT-based feature alignment that keeps only the phase spectrum for distillation. On VisDrone, YOLOv7-Tiny improves from mAP $$210\text{M} \rightarrow 176\text{M} \rightarrow 133\text{M} \rightarrow 91\text{M}$$^^^^2search_query2^^^^ to $$210\text{M} \rightarrow 176\text{M} \rightarrow 133\text{M} \rightarrow 91\text{M}$$^^^^2id:(Fan et al., 2024) OR ti:\2^^^^, while the teacher YOLOv7-L is $$210\text{M} \rightarrow 176\text{M} \rightarrow 133\text{M} \rightarrow 91\text{M}$$2; on SynDrone, the proposed method reaches mAP $$210\text{M} \rightarrow 176\text{M} \rightarrow 133\text{M} \rightarrow 91\text{M}$$3, compared with $$210\text{M} \rightarrow 176\text{M} \rightarrow 133\text{M} \rightarrow 91\text{M}$$4 for the teacher and $$210\text{M} \rightarrow 176\text{M} \rightarrow 133\text{M} \rightarrow 91\text{M}$$5 for BCKD (Yao et al., 2024).

In compressed video action recognition, PKD exploits the hierarchy that intra-frame networks converge to flatter minima than residual networks, which in turn are flatter than motion-vector networks. It trains internal classifiers progressively from motion vectors, then residuals, then intra-frames. Reported IC accuracy gains reach $$210\text{M} \rightarrow 176\text{M} \rightarrow 133\text{M} \rightarrow 91\text{M}$$6 on UCF-^^^^2id:(Fan et al., 2024) OR ti:\2search_query2id:(Fan et al., 2024) OR ti:\2^^^^ and $$210\text{M} \rightarrow 176\text{M} \rightarrow 133\text{M} \rightarrow 91\text{M}$$7 on HMDB-5^^^^2id:(Fan et al., 2024) OR ti:\2^^^^, while WISE adds up to $$210\text{M} \rightarrow 176\text{M} \rightarrow 133\text{M} \rightarrow 91\text{M}$$8 and $$210\text{M} \rightarrow 176\text{M} \rightarrow 133\text{M} \rightarrow 91\text{M}$$9, respectively (Soufleri et al., 2024).

5. Empirical patterns, benefits, and misconceptions

A repeated empirical claim is that the strongest teacher is not necessarily the best teacher for distillation. This statement appears explicitly in dense retrieval, where “the best teacher is not always the strongest,” and in Pro-KD, where the best-performing checkpoint of the teacher is not necessarily the best checkpoint for training the student (Lin et al., 2022, Rezagholizadeh et al., 2021). The literature therefore treats teacher quality and teacher teachability as distinct quantities.

A second recurrent pattern is that progression often functions as a curriculum over knowledge difficulty. In PWKD, the student learns from narrower to wider teacher subnetworks; in PCD, distillation progresses over class groups ranked by teacher–student discrepancy; in POCL for LLMs, training samples are ranked from easy to hard and introduced incrementally, with progressively rising temperatures (Zhang et al., 2021, Li et al., 30 May 2025, Liu et al., 6 Jun 2025). This suggests that PKD is closely related to curriculum learning, but the curriculum is imposed on supervision structure rather than only on raw data order.

A third pattern is that progressive schemes frequently preserve or improve compact-model performance under substantial compression. In the KGC PMD setting, a 3-layer student retains Hits@^^^^2id:(Fan et al., 2024) OR ti:\2search_query2^^^^ $$\mathcal{L}_{MGFD}= MSE\big(Mask(\mathcal{F}^{S}|\lambda) - Mask(\mathcal{F}^{T}|\lambda)\big).$$^^^^2search_query2^^^^ with a $$\mathcal{L}_{MGFD}= MSE\big(Mask(\mathcal{F}^{S}|\lambda) - Mask(\mathcal{F}^{T}|\lambda)\big).$$^^^^2id:(Fan et al., 2024) OR ti:\2^^^^ parameter reduction from baseline (&&&^^^^2search_query2^^^^&&&). In speech watermarking, PKDMark reports a $$\mathcal{L}_{MGFD}= MSE\big(Mask(\mathcal{F}^{S}|\lambda) - Mask(\mathcal{F}^{T}|\lambda)\big).$$2 FLOPS reduction, average detection F^^^^2id:(Fan et al., 2024) OR ti:\2^^^^ score $$\mathcal{L}_{MGFD}= MSE\big(Mask(\mathcal{F}^{S}|\lambda) - Mask(\mathcal{F}^{T}|\lambda)\big).$$3, PESQ $$\mathcal{L}_{MGFD}= MSE\big(Mask(\mathcal{F}^{S}|\lambda) - Mask(\mathcal{F}^{T}|\lambda)\big).$$4, and average BER $$\mathcal{L}_{MGFD}= MSE\big(Mask(\mathcal{F}^{S}|\lambda) - Mask(\mathcal{F}^{T}|\lambda)\big).$$5 across ^^^^2id:(Fan et al., 2024) OR ti:\2^^^^4 attacks, while direct KD is reported at BER $$\mathcal{L}_{MGFD}= MSE\big(Mask(\mathcal{F}^{S}|\lambda) - Mask(\mathcal{F}^{T}|\lambda)\big).$$6 and PESQ $$\mathcal{L}_{MGFD}= MSE\big(Mask(\mathcal{F}^{S}|\lambda) - Mask(\mathcal{F}^{T}|\lambda)\big).$$7 (Cui et al., 24 Sep 2025). In one-class novelty detection, P-KDGAN improves student GAN performance by $$\mathcal{L}_{MGFD}= MSE\big(Mask(\mathcal{F}^{S}|\lambda) - Mask(\mathcal{F}^{T}|\lambda)\big).$$8, $$\mathcal{L}_{MGFD}= MSE\big(Mask(\mathcal{F}^{S}|\lambda) - Mask(\mathcal{F}^{T}|\lambda)\big).$$9, and $$\mathcal{L} = (1 - \alpha - \beta)\, \mathcal{L}_{CE} + \alpha\, \mathcal{L}_{SCORE} + \beta\, \mathcal{L}_{MGFD}.$$^^^^2search_query2^^^^ on CIFAR-^^^^2id:(Fan et al., 2024) OR ti:\2search_query2^^^^, MNIST, and FMNIST at computation compression ratios $$\mathcal{L} = (1 - \alpha - \beta)\, \mathcal{L}_{CE} + \alpha\, \mathcal{L}_{SCORE} + \beta\, \mathcal{L}_{MGFD}.$$^^^^2id:(Fan et al., 2024) OR ti:\2^^^^, $$\mathcal{L} = (1 - \alpha - \beta)\, \mathcal{L}_{CE} + \alpha\, \mathcal{L}_{SCORE} + \beta\, \mathcal{L}_{MGFD}.$$2, and $$\mathcal{L} = (1 - \alpha - \beta)\, \mathcal{L}_{CE} + \alpha\, \mathcal{L}_{SCORE} + \beta\, \mathcal{L}_{MGFD}.$$3, respectively (Zhang et al., 2020).

A common misconception is that PKD simply means inserting one intermediate assistant model. The collected papers show a broader picture. Progression may occur through evolving teachers, evolving students, staged data subsets, modality order, layerwise curricula, classwise masks, or decomposed teacher subnetworks (Shi et al., 2021, Lin et al., 2022, Soufleri et al., 2024, Zhang et al., 2021). Another misconception is that PKD is defined only by better final accuracy. Several works instead emphasize smoother loss curves, controlled teacher–student divergence, convergence stability, reduced catastrophic forgetting, or improved trade-offs between computation and performance (Shi et al., 2021, Liu et al., 6 Jun 2025, Lu et al., 25 Jul 2025).

6. Limitations, open questions, and research directions

The literature also makes clear that PKD is not a single settled doctrine. Different works choose incompatible progressive variables: checkpoint time, teacher capacity, student capacity, class subsets, modality order, data hardness, or hidden-layer depth. This suggests that “progressive” is an architectural principle whose exact instantiation remains task-dependent rather than universal.

Storage and scheduling overhead are recurring issues. Checkpoint-based progressive distillation requires storing multiple intermediate teacher snapshots and selecting a schedule, which curriculum extraction explicitly criticizes as impractical for large-scale training. Curriculum extraction therefore proposes that a curriculum can be extracted from the fully trained teacher via random projections of hidden representations, reaching performance similar to progressive distillation in the sparse parity setting while avoiding checkpoint storage (&&&3^^^^2id:(Fan et al., 2024) OR ti:\2^^^^&&&). This indicates an unresolved question: whether the benefits of PKD arise primarily from temporal teacher evolution or from the existence of any ordered sequence of easier-to-harder supervisory targets.

Another open direction concerns what exactly should be aligned progressively. Different papers advocate logits, scores, hidden features, masked features, latent vectors, phase spectra, or mixed outputs. In TaPD for trajectory forecasting, PKD is implemented as hierarchical feature regression across adjacent observation lengths,

$$\mathcal{L} = (1 - \alpha - \beta)\, \mathcal{L}_{CE} + \alpha\, \mathcal{L}_{SCORE} + \beta\, \mathcal{L}_{MGFD}.$$4

with a cosine-annealed distillation weight to stabilize optimization under variable history lengths (Fan et al., 6 Mar 2026). In ProGMLP, PKD is embedded in a multi-student sequence and combined with Progressive Mixup Augmentation to provide runtime-adaptive GNN-to-MLP trade-offs (Lu et al., 25 Jul 2025). These formulations imply that the future of PKD may lie less in a fixed canonical loss than in systematic co-design of progression axis, alignment target, and scheduling policy.

Finally, several results suggest that progression can be beneficial even when it departs from the classic teacher-to-student compression narrative. PKKD jointly trains CNN teachers and ANN students in a higher-dimensional kernel space and reports that ANN-5^^^^2search_query2^^^^ reaches $$\mathcal{L} = (1 - \alpha - \beta)\, \mathcal{L}_{CE} + \alpha\, \mathcal{L}_{SCORE} + \beta\, \mathcal{L}_{MGFD}.$$5 top-^^^^2id:(Fan et al., 2024) OR ti:\2^^^^ on ImageNet, $$\mathcal{L} = (1 - \alpha - \beta)\, \mathcal{L}_{CE} + \alpha\, \mathcal{L}_{SCORE} + \beta\, \mathcal{L}_{MGFD}.$$6 higher than ResNet-5^^^^2search_query2^^^^ (Xu et al., 2020). PWKD decomposes the teacher itself and distills “knowledge quantity” from partial to whole (Zhang et al., 2021). POCL restructures sample exposure and temperature in white-box LLM KD (Liu et al., 6 Jun 2025). Taken together, these works indicate that PKD has evolved into a broad research program on shaping the path of supervision, not merely the endpoint of supervision.