Soft Prompt Distillation Overview

• Soft Prompt Distillation is a technique that transfers a teacher’s prompt behavior to a student model by using learnable continuous prompts and structured instructions while keeping the backbone frozen.
• It encompasses methods such as continuous prompt transfer, non-parametric prompt compilation, and prompt-conditioned weight adaptation across various domains including vision-language and clinical NLP.
• This approach enhances efficiency, interpretability, and privacy by reducing computational demands and enabling fast, scalable, and transparent model deployment.

Soft prompt distillation denotes a family of teacher–student transfer methods in which prompts, rather than full model updates, become the primary carrier of distilled capability. In the literature, the distilled object may be a learnable continuous prompt, a prompt-conditioned logit distribution, a prompt-induced behavioral shift, a shared meta-prompt, or even a structured textual instruction set inserted into a system prompt. Correspondingly, the target may be a frozen backbone with trainable prompt parameters, a compact student model driven by stored class vectors, a prompt transferred across model families, or a student whose weights imitate a prompt-conditioned teacher. This breadth is evident across CLIP-style VLM adaptation, LLM reasoning, continual learning, recommendation, clinical NLP, and multimodal VQA (Li et al., 2024, Wang et al., 19 Jun 2025, Badhe et al., 24 Feb 2026).

1. Conceptual scope and terminological variants

“Soft prompt distillation” is not used uniformly. In its narrowest sense, it refers to transferring knowledge into learnable continuous prompt embeddings while keeping the backbone frozen. PromptKD, MoPD, PLPP, CDL-Prompt, FDBPL, MAPD, PromptMM, and the clinical multitask prompt framework all instantiate this prompt-centric PEFT view, although they differ in whether they distill logits, prompt representations, modality structure, or task-shared prompt factors (Li et al., 2024, Chen et al., 2024, Liu et al., 2024, Zhang et al., 2024, Zhang et al., 23 May 2025, Gupta et al., 7 Jun 2025, Wei et al., 2024, Peng et al., 8 Apr 2026).

A broader usage includes non-parametric prompt-space transfer. Prompt-Level Distillation (PLD) explicitly presents itself as a non-parametric form of soft prompt distillation: teacher reasoning is “compiled” into a structured list of natural-language instructions that is injected into the student’s system prompt, avoiding any parameter update while preserving explicit reasoning structure (Badhe et al., 24 Feb 2026). By contrast, “Knowledge Injection via Prompt Distillation” uses prompt distillation to move knowledge from a teacher’s privileged prompt into LoRA weights, not into a soft prompt representation (Kujanpää et al., 2024).

A further boundary case is discrete prompt distillation. DistillPrompt is explicitly a non-gradient autoprompting method whose final artifact is an interpretable textual prompt, not a learned continuous embedding; DeltaPrompts is a data-centric method for selecting and synthesizing high-divergence prompts for multimodal distillation, rather than a prompt-tuning method (Dyagin et al., 26 Aug 2025, Jung et al., 15 May 2026). This suggests that the phrase “prompt distillation” has become an umbrella term covering at least three regimes: continuous prompt transfer, non-parametric prompt compilation, and prompt-conditioned knowledge transfer into other parameterizations.

Family	Distilled object	Representative papers
Prompt-parameter distillation	Learnable continuous prompts or prompt-conditioned logits	PromptKD (Li et al., 2024), MoPD (Chen et al., 2024)
Prompt-space compilation	Structured textual instructions in the system prompt	PLD (Badhe et al., 24 Feb 2026)
Prompt-conditioned transfer beyond prompts	Prompt effects transferred into weights or shared low-rank factors	POST (Wang et al., 19 Jun 2025), clinical multitask prompt distillation (Peng et al., 8 Apr 2026), knowledge injection via prompt distillation (Kujanpää et al., 2024)

2. Core mechanisms and optimization objectives

Most soft prompt distillation methods are teacher–student systems in which the backbone is frozen and the prompt channel absorbs task adaptation. In PromptKD, the teacher CLIP text encoder produces stored class vectors

$$W = [w_1, w_2, \dots, w_N] \in \mathbb{R}^{N \times d},$$

and student training on unlabeled domain images minimizes KL divergence between teacher and student class distributions,

$L_{stu} = L_{kd}(q^t, q^s, \tau),$

with the student updated mainly through visual prompts and a small projector (Li et al., 2024). The key transfer object is thus the teacher’s prediction behavior over class logits, mediated by prompts rather than backbone finetuning.

Other methods distill prompt guidance directly. MoPD introduces single-prompt distillation via KL divergence from a hard-prompt teacher to a soft-prompt student, then generalizes it to mixture-of-prompts distillation with a gating network and the objective

$$\zeta = \alpha \zeta_{\mathrm{CE}} + \left(1-\alpha\right)\zeta_{\mathrm{MPD}} + \beta \zeta_{\mathrm{MPS}},$$

where the gating network selects instance-specific hard prompts from a prompt pool (Chen et al., 2024). CDL-Prompt goes further by projecting teacher prompts into student-compatible shared prompts through attention-based mapping,

$$P' = \sum_{m=1}^{M} \gamma(\mathrm{SPE}(x), k'_m) P'_m,$$

so that the student receives both its own prompts and teacher-derived shared prompts inside the frozen ViT backbone (Zhang et al., 2024).

A separate line treats prompt learning itself as a self-distillation problem. PLPP augments CoOp with a perplexity-based regularizer and mutual self-distillation,

$$\mathcal{L}_{PLPP} = \mathcal{L}_{CE} + \lambda \cdot \left(\alpha \cdot \mathcal{L}_{PPL} + (1-\alpha)\cdot \mathcal{L}_{IPPL}\right),$$

where prompt vectors are regularized against a language-model-style distribution derived from the frozen text encoder (Liu et al., 2024). The paper explicitly interprets this as self-distillation: shallow prompt embeddings learn from deeper transformed features.

In LLM transfer settings, the objective often shifts from prompt tuning to prompt portability. POST first distills a large provider model $\Phi_t$ into a small surrogate $\Phi_s$, then tunes a private soft prompt locally, and finally transfers it back to the target model with

$$\mathcal{L} = (1-\alpha)\mathcal{L}_1 + \alpha \mathcal{L}_2,$$

where $\mathcal{L}_1$ matches prompted outputs and $\mathcal{L}_2$ matches the prompt-induced directional change relative to each model’s zero-shot behavior (Wang et al., 19 Jun 2025). In the clinical multitask setting, each task prompt is decomposed as

$L_{stu} = L_{kd}(q^t, q^s, \tau),$0

with joint task, logit-distillation, and hidden-state-distillation losses,

$L_{stu} = L_{kd}(q^t, q^s, \tau),$1

and $L_{stu} = L_{kd}(q^t, q^s, \tau),$2 (Peng et al., 8 Apr 2026).

PLD formalizes a non-parametric variant. Given labeled data

$L_{stu} = L_{kd}(q^t, q^s, \tau),$3

teacher reasoning extraction yields

$L_{stu} = L_{kd}(q^t, q^s, \tau),$4

where $L_{stu} = L_{kd}(q^t, q^s, \tau),$5 is an executable rule. The workflow then applies clustering, synthesis, conflict resolution, and zero-shot deployment through the student’s system prompt (Badhe et al., 24 Feb 2026).

3. Vision-language adaptation and prompt-centric generalization

In CLIP-style VLMs, soft prompt distillation is primarily motivated by the base-to-new failure mode of prompt tuning. PromptKD addresses this through unsupervised domain prompt distillation: a ViT-L/14 teacher is first adapted with labeled domain data, its text features are frozen and stored as shared class vectors, and a smaller student such as ViT-B/16 is trained on unlabeled domain images by KL logit matching. Across 11 datasets, PromptKD reports average improvements over PromptSRC of +2.70% on base classes, +4.63% on novel classes, and +3.76% harmonic mean. On SUN397, it reports 42.5 GFLOPs and 1710 FPS, compared with 162.8 GFLOPs and 1380 FPS for PromptSRC, because only the student image branch is needed at inference (Li et al., 2024).

MoPD and PLPP both target prompt overfitting, but with different teacher signals. MoPD transfers knowledge from a pool of 12 manually created hard prompts into a learnable soft prompt, using a gating network with top-$L_{stu} = L_{kd}(q^t, q^s, \tau),$6 selection. In base-to-new generalization, it reports the highest harmonic mean on 10 out of 11 datasets; the average row gives new-class accuracy 74.69 and harmonic mean 77.90, compared with 73.61 and 77.00 for KgCoOp (Chen et al., 2024). PLPP instead regularizes CoOp with perplexity and inverted perplexity losses derived from the frozen text encoder and an untrained LM head. It reports the best average domain generalization accuracy, 60.8%, on ImageNet-V2, ImageNet-Sketch, ImageNet-A, and ImageNet-R, and it beats PromptSRC on 10/10 target datasets in the cross-dataset setting (Liu et al., 2024).

FDBPL extends the distillation-based prompt-learning line by removing repeated online teacher inference. Teacher outputs for random crops are precomputed once and stored in a Region Information Lookup table together with crop coordinates, augmentation type, and teacher-generated soft labels. The method then performs region-aware learning with dual positive-negative prompt spaces and cascaded difference learning. Across 11 datasets, it reports an average harmonic mean of 74.57 in base-to-new generalization, compared with 73.99 for KDPL, and an average $L_{stu} = L_{kd}(q^t, q^s, \tau),$7 faster training speed (Zhang et al., 23 May 2025).

Two further directions show that prompt distillation also serves continual and few-shot adaptation. CDL-Prompt distills prompts themselves in continual ViT learning through attention-based prompt mapping and improves the CODA-Prompt baseline with a ViT-Base student from 86.16% to 87.69% on Split CIFAR-100 and from 75.78% to 77.95% on Split ImageNet-R (Zhang et al., 2024). MAPD, built on LLaVA v1.5 with a CLIP ViT-L/14-336px visual encoder and Qwen2.5-7B-Instruct, jointly learns an attention-mapper and soft prompt tokens; it reports about 24M trainable parameters, a test-time FT advantage of about 21.2% on average over ICL, strict monotonic improvement as shots increase, and the smallest mean robustness drop under image perturbations, $L_{stu} = L_{kd}(q^t, q^s, \tau),$8 (Gupta et al., 7 Jun 2025).

4. LLM reasoning, prompt portability, and cross-task shared prompts

In LLM reasoning, the central attraction of soft prompt distillation is often not only PEFT but also the relocation of reasoning cost. PLD makes this explicit. A teacher is prompted to solve each labeled example with chain-of-thought reasoning and then abstract that reasoning into a generalized instruction. Those instructions are embedded with Gemini Embedding into 768-dimensional vectors, clustered with DBSCAN using cosine distance with $L_{stu} = L_{kd}(q^t, q^s, \tau),$9 and min_samples = 6, and synthesized by Gemini 3 Pro into master instructions. Closed-loop conflict resolution then iteratively revises contradictions until validation error converges. With Gemma-3 4B, PLD reports Macro-F1 gains from 0.57 to 0.90 on StereoSet and from 0.67 to 0.83 on Contract-NLI; the abstract presents these as improvements from 57% to 90.0% and from 67% to 83%, with negligible latency overhead. The paper also states that Gemma-3 4B is about 25× cheaper and roughly 80× faster than Gemini-3 Flash (Badhe et al., 24 Feb 2026).

POST addresses a different problem: soft prompts are tightly coupled to the model on which they were tuned. Its three-stage pipeline distills the provider model into a small surrogate, tunes the prompt locally on private data, optionally with PromptDPSGD, and transfers the prompt back to the target model using only a small public dataset. The paper reports prompts of length 100 by default, main DP experiments with $$\zeta = \alpha \zeta_{\mathrm{CE}} + \left(1-\alpha\right)\zeta_{\mathrm{MPD}} + \beta \zeta_{\mathrm{MPS}},$$0, strong transfer with fewer than 100 public examples, and roughly a 6x runtime reduction on SST-2 relative to full prompt tuning on GPT2-XL when tuning and transfer are considered together (Wang et al., 19 Jun 2025).

The clinical multitask prompt distillation framework reinterprets soft prompt distillation as multi-task compression into a shared metaprompt. It learns teacher prompts from 21 diverse clinical source tasks, distills them into a shared prompt representation, and adapts to 10 held-out target datasets across NER, RE, QA, NLI, and summarization. With LLaMA 3.1 8B, Meditron3 8B, and gpt-oss 20B, it uses fewer than 0.05% trainable parameters, outperforms LoRA by 1.5~1.7%, and exceeds single-task prompt tuning by 6.1~6.6%. At $$\zeta = \alpha \zeta_{\mathrm{CE}} + \left(1-\alpha\right)\zeta_{\mathrm{MPD}} + \beta \zeta_{\mathrm{MPS}},$$1 shot, it reports the largest gains over LoRA and PT, and it notes that zero-shot performance remains near zero across tasks (Peng et al., 8 Apr 2026).

PromptMM shows that prompt distillation is not limited to language generation or CLIP classification. In multimodal recommendation, it inserts a soft prompt module into the teacher’s modality reduction layer to bridge the semantic gap between multimodal content semantics and collaborative preference semantics, then distills pairwise ranking, disentangled listwise ranking, and modality-aware embeddings into a LightGCN student. On Netflix, it reports Recall@20 of 0.1864 and NDCG@20 of 0.0743, exceeding the strongest prior baselines cited in the paper (Wei et al., 2024).

5. Deployment properties: efficiency, privacy, interpretability, and compression

A recurrent systems claim is that prompt distillation preserves PEFT while improving deployment characteristics. PLD emphasizes interpretability most strongly: because the distilled reasoning is expressed as natural-language rules in the system prompt, the decision process is described as transparent and human-verifiable, with the paper explicitly highlighting law, finance, content moderation, high-volume settings, and edge devices as target deployments (Badhe et al., 24 Feb 2026). This differs from weight-space finetuning, where the learned behavior is hidden in parameters.

PromptKD and FDBPL highlight a different systems advantage: distillation can be reorganized so that the expensive teacher branch is amortized or removed. PromptKD pre-stores teacher text features as shared class vectors, so no text-branch forward pass is needed at inference (Li et al., 2024). FDBPL uses a “space-for-time” design in which teacher supervision is cached offline and sparsified with Marginal Smoothing or Marginal Re-Norm with Top-K, allowing fast lookup instead of repeated online teacher inference (Zhang et al., 23 May 2025).

POST places privacy at the center. Its DP guarantee applies only to the local prompt-tuning stage on the user’s private data; because of the post-processing property of DP, the learned prompt can be sent back to the provider without additional privacy loss. The paper is equally explicit that this does not protect test-time queries and does not fully study all possible leakage from the provider’s pretraining data during KD (Wang et al., 19 Jun 2025). This suggests that privacy-preserving soft prompt transfer is tractable, but only under a sharply delimited threat model.

Compression becomes important once prompts, logits, or teacher supervision are cached at scale. “Vector-Quantized Soft Label Compression for Dataset Distillation” is not a prompt-distillation paper, but it provides a relevant systems analogue: on ImageNet-1K, a VQAE achieves 30–40x additional compression over RDED, LPLD, SRE2L, and CDA while retaining over 90% of their original performance, and in GPT-2 token-level distillation it reduces a 112 GB soft-label cache to about 200 MB. A plausible implication is that large-scale soft prompt distillation may need analogous compression for prompt tensors or prompt-conditioned supervision signals when storage, transmission, or context budget becomes the bottleneck (Abbasi et al., 4 Mar 2026).

6. Limitations, misconceptions, and adjacent directions

The literature also converges on several limitations. PromptKD assumes CLIP-like decoupled image/text encoders, access to class names, and unlabeled domain images whose coverage is relevant to the target domain (Li et al., 2024). PLD is explicitly scoped to reasoning-intensive classification tasks with static decision boundaries; it cautions that arithmetic or symbolic proofs may not be fully externalizable into a concise prompt, and it does not model prompt scaling limits as instruction sets grow (Badhe et al., 24 Feb 2026). MAPD reports that FT at test time requires about 5× more compute than ICL, making it unattractive for resource-constrained deployment despite its accuracy gains (Gupta et al., 7 Jun 2025). The clinical multitask framework reports that zero-shot adaptation remains weak and still requires some target supervision (Peng et al., 8 Apr 2026).

Several controversies are terminological. DistillPrompt is a prompt distillation method, but its prompts are discrete natural-language instructions produced by a non-gradient search procedure; it is therefore not a soft-prompt method in the usual sense (Dyagin et al., 26 Aug 2025). “Knowledge Injection via Prompt Distillation” is similarly adjacent rather than canonical: the teacher sees new knowledge in its prompt, but the distilled artifact is a LoRA adapter in the student’s weights (Kujanpää et al., 2024). DeltaPrompts, finally, is data-centric. It shows that up to 69% of prompts in standard chart/document reasoning datasets are effectively zero-delta, meaning teacher and student already induce the exact same answer distribution; it then proposes answer divergence $$\zeta = \alpha \zeta_{\mathrm{CE}} + \left(1-\alpha\right)\zeta_{\mathrm{MPD}} + \beta \zeta_{\mathrm{MPS}},$$2 as the criterion for prompt usefulness and constructs a 200k synthetic high-divergence dataset (Jung et al., 15 May 2026). This suggests that prompt selection may be as important as prompt parameterization: a soft prompt can only distill what the training prompts actually expose.

Ethical concerns also appear directly in the primary papers. PLD notes that, because StereoSet concerns bias, the teacher could hallucinate or amplify bias during instruction extraction, and the consolidated prompt could encode discriminatory logic if not carefully reviewed (Badhe et al., 24 Feb 2026). Prompt-centric transfer is therefore not intrinsically safer or more neutral than weight-space transfer; it is merely more inspectable when the prompt remains textual.

Taken together, these results indicate that soft prompt distillation is best understood as a prompt-space transfer paradigm rather than a single algorithmic template. Its unifying idea is that capabilities ordinarily associated with larger models, longer contexts, or heavier finetuning can be relocated into prompts, prompt-conditioned supervision, or prompt-derived low-rank factors. The most successful formulations do not merely optimize prompts for task loss; they preserve teacher distributions, preserve prompt-induced behavioral changes, or preserve reusable structure across tasks and domains (Wang et al., 19 Jun 2025, Peng et al., 8 Apr 2026, Jung et al., 15 May 2026).