PromptKD: Prompt-Mediated Distillation

• PromptKD is a family of prompt-mediated distillation methods that leverages specialized prompts to control and refine the teacher's supervision signal.
• It spans multiple settings including unsupervised vision-language, student-friendly LLM distillation, teacher-side response priming, and knowledge injection.
• Empirical studies show that PromptKD improves model efficiency, domain generalization, and performance metrics across diverse datasets and architectures.

Searching arXiv for papers on PromptKD and closely related prompt distillation variants. PromptKD is a family of prompt-mediated distillation methods rather than a single algorithm. In recent arXiv usage, the term covers at least four distinct settings: unsupervised prompt distillation for CLIP-style vision-LLMs, where cached teacher text features supervise a smaller student on unlabeled images (Li et al., 2024); student-friendly distillation for generative LLMs, where a frozen teacher is adapted by a tiny set of learned soft prompts (Kim et al., 2024); teacher-side response-priming for large-language-model distillation, where prompts shape the teacher’s reasoning traces before knowledge transfer (Goyal et al., 2024); and prompt distillation for knowledge injection, where privileged document context in the teacher prompt is compressed into LoRA weights of an otherwise identical student (Kujanpää et al., 2024). Across these variants, prompts function less as a deployment interface than as a mechanism for controlling, reshaping, or caching the supervision signal seen during distillation.

1. Semantic scope and recurring structure

Despite the shared label, PromptKD spans different model families, supervision regimes, and prompt loci. In CLIP-style work, the prompt may reside in the teacher during few-shot pretraining and in the student image encoder during unlabeled distillation; in generative LLM work, the prompt is often attached only to the frozen teacher; in response-primed KD, the prompt modifies teacher generations when constructing the transfer set; and in knowledge-injection prompt distillation, the prompt supplies privileged context that the student will not see at inference (Li et al., 2024).

Variant	Prompt locus	Distillation signal
Unsupervised CLIP PromptKD	Teacher text branch cached as class vectors; student learns visual prompts	KL on teacher/student logits over unlabeled images
Student-friendly generative PromptKD	Soft prompts prepended to frozen teacher	Token-level KL on student-generated responses
Response-priming LLM PromptKD	Teacher prompted during transfer-set creation	Hard CE plus reverse KL over reasoning and answer tokens
Knowledge-injection prompt distillation	Teacher receives privileged context in prompt	Per-token KL on answer positions
Multi-teacher and refiner variants	Teacher ensemble or refined class text embeddings	Same KD backbone with altered targets or refined text features

This suggests that the unifying abstraction is not a specific architecture but a design principle: prompt the teacher, the shared text interface, or the transfer-set generator so that the distilled distributions are more informative, more domain-aware, or more student-compatible.

2. Unsupervised PromptKD for vision-LLMs

The CLIP-oriented formulation in “PromptKD: Unsupervised Prompt Distillation for Vision-LLMs” is a two-stage framework. Stage 1 adapts a large teacher CLIP, typically ViT-L/14, with few-shot labels using PromptSRC or MaPLe. After pretraining, the teacher text encoder is run once on class names and templates to produce normalized class vectors

$$\mathbf{v}_c=\frac{1}{M}\sum_{m=1}^M \frac{f_T(t_m(c);\theta_T)}{\|f_T(t_m(c);\theta_T)\|_2},$$

which are stacked into $W \in \mathbb{R}^{N \times d_T}$. Stage 2 distills a smaller student, typically ViT-B/16, on unlabeled domain images while freezing the student backbone and training only visual prompts plus a lightweight projector $P(\cdot)$. Teacher and student logits are computed against the same cached $W$, and the loss is

$$\mathcal{L}_{\text{KD}}(x;\tau)=\tau^2\,\mathrm{KL}\!\left(\mathrm{softmax}(\mathbf{z}^T(x)/\tau)\,\|\,\mathrm{softmax}(\mathbf{z}^S(x)/\tau)\right).$$

The method exploits CLIP’s decoupled modalities to remove runtime text encoding during both distillation and inference (Li et al., 2024).

Empirically, the ViT-B/16 student with a ViT-L/14 teacher reaches average Base 86.96, Novel 80.73, and HM 83.73 over 11 datasets, improving over PromptSRC by +2.70, +4.63, and +3.76 respectively. On ImageNet, the reported Base/Novel/HM is 80.83/74.66/77.62; on DTD it is 85.84/71.37/77.94; on EuroSAT it is 97.54/82.08/89.14. In cross-dataset transfer from ImageNet 16-shot pretraining, the average across 10 targets is 71.33%, +5.52% over prior state of the art. In domain generalization from ImageNet to ImageNet-V2, -Sketch, -A, and -R, the average is 71.47%, versus 63.55% for PromptAlign. The efficiency argument is equally central: on SUN397, PromptKD reduces test GFLOPs from approximately 162.5 to 42.5 and raises FPS from approximately 1380 to 1710 while improving HM from 79.97 to 83.73. The paper also reports that KL-based logit distillation is superior to feature alignment, that a 2-layer projector is best, that prompt depth 9 with visual prompt length 4 works well, and that $\tau=1$ is optimal in this setting (Li et al., 2024).

3. Multi-teacher and prompt-refinement extensions in vision-language distillation

The multi-teacher extension “The Professor” generalizes PromptKD from a single PromptSRC-finetuned ViT-L/14 teacher to a fixed two-teacher ensemble consisting of that domain-finetuned teacher and a zero-shot EVA-CLIP-L/14 teacher with cached logits. Two ensemble rules are studied. Equal-probability ensembling uses

$$p^{*}(y|x)=\tfrac{1}{2}\big(p_1(y|x)+p_2(y|x)\big),$$

whereas confidence-weighted ensembling uses per-image weights

$$w_i(x)=\frac{c_i(x)}{c_1(x)+c_2(x)+10^{-8}},\qquad c_i(x)=\max_y p_i(y|x),$$

and then

$p^{*}(y|x)=w_1(x)p_1(y|x)+w_2(x)p_2(y|x).$

The student still minimizes $\tau^2 \mathrm{KL}(p^{*}\|q_\theta)$ with $W \in \mathbb{R}^{N \times d_T}$0, 20 epochs, and batch size 8. On the four-dataset sweep over Caltech-101, DTD, UCF101, and EuroSAT, average HM rises from 87.52 for the single-teacher baseline to 88.88 for equal averaging and 89.28 for confidence weighting. Gains are dataset dependent: +0.16 HM on Caltech-101, +0.62 on UCF101, and +5.78 on EuroSAT for confidence weighting, supporting the claim that complementary supervision matters most under domain shift (Algadhi et al., 22 Jun 2026).

A different extension, TextRefiner, leaves the PromptKD backbone intact but refines class text embeddings using internal visual features from the CLIP image branch. Local image tokens $W \in \mathbb{R}^{N \times d_T}$1 are clustered into an attribute storage $W \in \mathbb{R}^{N \times d_T}$2, a 2-layer MLP aligns those local tokens with the text space, and the original class embedding $W \in \mathbb{R}^{N \times d_T}$3 is combined with a cache-aggregated fine-grained descriptor $W \in \mathbb{R}^{N \times d_T}$4 to form

$W \in \mathbb{R}^{N \times d_T}$5

When attached to PromptKD, TextRefiner is reported to produce state-of-the-art performance while remaining efficient in inference. The paper states that PromptKD with TextRefiner surpasses LLaMP by 1.06% on average harmonic mean and improves average novel accuracy by 1.97%; on DTD, novel accuracy rises from 69.15% to 71.01%, and on EuroSAT from 71.59% to 79.22%. It also reports 12793.26 FPS on an A800 GPU for PromptKD with TextRefiner, compared with 1473.46 FPS for LLaMP and 20.45 FPS for CoCoOp, while noting sensitivity to cache size $W \in \mathbb{R}^{N \times d_T}$6 and to the coefficients $W \in \mathbb{R}^{N \times d_T}$7, $W \in \mathbb{R}^{N \times d_T}$8, and $W \in \mathbb{R}^{N \times d_T}$9 (Xie et al., 2024).

4. PromptKD for student-friendly distillation in generative LLMs

In generative LLM compression, PromptKD denotes a parameter-efficient method for extracting student-friendly knowledge from a frozen teacher by tuning only a small set of soft prompt embeddings. The teacher receives prompt tokens $P(\cdot)$0 prepended to its input, while the student is updated to imitate the resulting teacher distributions. The training loop is explicitly student-guided. A request $P(\cdot)$1 is sampled, the student generates a response $P(\cdot)$2, and then the prompt is updated using

$P(\cdot)$3

where $P(\cdot)$4 and $P(\cdot)$5. With the updated prompt fixed, the student minimizes

$P(\cdot)$6

The paper emphasizes reverse KL for generation, pseudo-target generation at temperature 1.0, and a prompt length of $P(\cdot)$7 initialized from the text “Suppose you are a student.” Random initialization is reported to fail. The prompt adds approximately 11,200 parameters, about 0.0007% of a 1.5B GPT-2 XL teacher (Kim et al., 2024).

The experimental setup uses GPT-2 XL as teacher and GPT-2 Base, Medium, and Large as students on Dolly, SelfInst, Vicuna, S-NI, and UnNI. PromptKD is reported as state of the art among KD methods across most settings. For the 340M student, the reported ROUGE-L scores are 27.3 on Dolly, 15.0 on SelfInst, 17.6 on Vicuna, 27.1 on S-NI, and 32.6 on UnNI, with “+” marks indicating surpassing the teacher on all five datasets. The 120M student reaches 25.6, 13.1, 16.8, 26.8, and 28.9 respectively, and the 760M student reaches 26.9, 16.4, 17.8, 29.5, and 34.8. The mechanistic interpretation centers on exposure bias: the paper measures $P(\cdot)$8 and reports that PromptKD keeps it close to 0 across generation steps and throughout training, whereas classical KD baselines show increasing mismatch under free-run generation (Kim et al., 2024).

5. Teacher-side prompting in large-language-model distillation and knowledge injection

A later LLM variant uses PromptKD to mean response-priming prompting inside the distillation pipeline. The teacher is a quantized Llama 3.1 405B Instruct model, the student is a Llama 3.1 8B Instruct model fine-tuned with LoRA, and prompts are applied only when constructing the transfer set. The paper distinguishes “Teacher prompting,” “Ground Truth prompting,” and “Confidence prompting,” with exact templates such as “As a teacher, guide your student through solving the question below. Provide a clear, simple explanation for someone unfamiliar with the problem.” Teacher outputs contain both intermediate reasoning tokens and final answers, and the student is trained with

$P(\cdot)$9

using $W$0, $W$1, and reverse KL for the soft loss over the full prompted sequence. On GSM8K, the reported accuracies are 12.20% for the base model, 25.01% for “No KD finetuned,” 30.62% for Base KD, 34.04% for Confidence KD, 42.30% for Teacher KD, and 48.14% for Ground Truth KD. The paper describes Ground Truth KD as a 55% performance increase over Base KD and reports attention-layer correlates of success: lower late-layer attention entropy, lower self-attention focus, and higher inter-layer similarity in the stronger prompted students (Goyal et al., 2024).

A separate line of work uses prompt distillation for knowledge injection rather than model compression in the usual teacher–student sense. Here the teacher and student are the same base Llama-3-8B-Instruct model; the student differs only by a LoRA adapter. The teacher sees privileged context $W$2 together with the question $W$3 and answer $W$4, while the student sees only $W$5 and $W$6. Distillation minimizes the mean KL over answer positions at temperature $W$7, with high-temperature data generation at $W$8 used to create diverse Q/A pairs. The method is evaluated on four domains derived from Squadshifts—Amazon, New Wiki, NYT, and Reddit—and reports closed-book correctness of 86.1 ± 0.2, 94.4 ± 0.3, 93.6 ± 0.6, and 79.5 ± 1.4 respectively, substantially above supervised fine-tuning and unsupervised fine-tuning baselines. The paper further reports that PromptKD approaches base+RAG in closed-book performance and that PromptKD+RAG can reach 88.5 ± 0.3 on Amazon, 96.7 ± 0.2 on New Wiki, and 96.9 ± 0.2 on NYT. It also argues that PromptKD is markedly more sample-efficient than SFT: on NYT, closed-book PromptKD rises from 83.0% with 5 questions per context to 93.7% with 30, whereas SFT rises from 72.4% to 87.0% at 30 and requires roughly 200 questions to reach about 93.3% (Kujanpää et al., 2024).

6. Practical distinctions, misconceptions, and open problems

A common misconception is that PromptKD always means prompt tuning of the student. In the CLIP formulation, only student visual prompts and a projector are trained while the text branch is replaced by cached teacher class vectors (Li et al., 2024). In the generative student-friendly formulation, the student may receive no prompts at inference at all; the prompt is attached to the frozen teacher solely to adapt the supervision distribution (Kim et al., 2024). In response-primed LLM distillation, the student is explicitly not prompted during training; the prompt enters only through the teacher distribution $W$9 (Goyal et al., 2024). In knowledge injection, the prompt is privileged context available only to the teacher, and the student’s objective is to internalize that context into LoRA weights (Kujanpää et al., 2024).

The main technical fault lines are therefore elsewhere. One is teacher quality and calibration: PromptKD gains diminish with weaker or redundant teachers, whereas multi-teacher gains can be large when the second teacher contributes complementary supervision under domain shift, as on EuroSAT (Algadhi et al., 22 Jun 2026). Another is prompt sensitivity: random prompt initialization can fail in generative PromptKD, Confidence prompting can induce long, meandering reasoning that risks truncation in response-primed KD, and knowledge-injection PromptKD inherits errors from poor teacher prompts or low-quality context (Kim et al., 2024). A third is hyperparameter sensitivity and adaptation overhead: CLIP refiners depend on cache size and fusion coefficients, LLM response-priming depends on $$\mathcal{L}_{\text{KD}}(x;\tau)=\tau^2\,\mathrm{KL}\!\left(\mathrm{softmax}(\mathbf{z}^T(x)/\tau)\,\|\,\mathrm{softmax}(\mathbf{z}^S(x)/\tau)\right).$$0 and $$\mathcal{L}_{\text{KD}}(x;\tau)=\tau^2\,\mathrm{KL}\!\left(\mathrm{softmax}(\mathbf{z}^T(x)/\tau)\,\|\,\mathrm{softmax}(\mathbf{z}^S(x)/\tau)\right).$$1, and knowledge-injection PromptKD requires high-temperature data generation plus nontrivial Q/A synthesis (Xie et al., 2024).

Open directions recur across the literature. Multi-prompt or multi-teacher ensembles, hidden-state or attention alignment, better calibration-aware weighting, and explicit separation of reasoning-token and answer-token losses are all proposed or implied extensions. More broadly, the literature suggests that PromptKD has evolved from a narrow vision-language distillation recipe into a broader research program centered on prompt-conditioned supervision: prompts can cache text features, soften or specialize teacher outputs, generate student-friendly targets, or compress external knowledge into weights. What remains unsettled is not whether prompting can assist distillation, but which prompt locus, objective, and teacher configuration best match a given compression or adaptation regime.