Prompt-Based Fine-Tuning Strategy (PFTS)

• PFTS is a family of parameter-efficient methods that inject learnable prompt embeddings into frozen pretrained models to steer their behavior.
• It encompasses static, dynamic, decomposed, and internal modulation techniques across domains like vision, NLP, code, and speech.
• This strategy balances high performance with minimal trainable parameters, enhancing training efficiency and deployment scalability.

Searching arXiv for the specified paper and closely related prompt-based fine-tuning work to ground the article. Prompt-based Fine-Tuning Strategy (PFTS) denotes a family of parameter-efficient adaptation methods in which a pretrained model is customized by prompts, prompt-conditioned generators, or prompt-aware modulation, while the main backbone remains frozen or largely frozen. In its simplest form, PFTS learns a small set of continuous prompt embeddings prepended to the input; in more elaborate forms, it injects prompts layer-wise, initializes them from downstream statistics, generates them dynamically from each instance, decomposes them into shorter prompts plus low-rank factors, or converts prompt information into internal multiplicative adjustments. Across vision, NLP, code, speech, and clinical information extraction, the shared objective is to preserve pretrained knowledge while reducing trainable parameters, storage, and task-specific duplication of the backbone (Wang et al., 2024, Lan et al., 2024, Ma et al., 2024).

1. Foundational definition and architectural interface

PFTS is typically positioned between full fine-tuning and lightweight probing. Full fine-tuning updates all model parameters and is accurate but computationally heavy; linear probing trains only a new classifier head over frozen features and is cheap but often underperforms. Prompt-based fine-tuning keeps the pretrained backbone frozen, updates only a small prompt parameter set and usually a task head, and seeks a compromise between accuracy and efficiency. This framing is explicit in both visual prompt tuning for ViTs and soft prompt tuning for encoder–decoder LMs (Wang et al., 2024, Shernazarov et al., 23 Mar 2026).

The canonical NLP interface prepends learned prompt vectors to the token sequence. In encoder–decoder summarization, for instance, the encoder input becomes $[p_1,\dots,p_m,x_1,\dots,x_n]$, where only the prompt parameters are updated and the base model remains frozen. In prompt-based classification with a masked LM, the model predicts a label word at a template-defined $[\mathrm{MASK}]$ position, and a verbalizer maps predicted tokens to class labels (Shernazarov et al., 23 Mar 2026, Wang et al., 2022).

The canonical vision interface injects prompt tokens into a ViT token stream. With patch embeddings $E_0=\mathrm{PatchEmbed}(x)+E_{\mathrm{pos}}$, shallow insertion uses

$[x_1,Z_1,E_1]=B_1([x_0,P_0,E_0]),$

whereas deep insertion uses

$[X_i,Z_i,E_i]=B_i([X_{i-1},P_{i-1},E_{i-1}]).$

Only the prompts and the task head are trained; patch embedding and transformer blocks remain frozen (Wang et al., 2024).

A broader formulation extends beyond explicit prepended tokens. Some methods leave the input length unchanged and instead treat the prompt as a conditioning signal that generates low-rank factors, gates, or multiplicative vectors inside the network. This suggests that PFTS is best understood not as a single insertion trick, but as a general strategy for steering a frozen pretrained model with a small prompt-dependent parameterization.

2. Mechanistic taxonomy

PFTS has diversified into several recurring mechanism classes.

Mechanism	Representative operation	Example papers
Static prompt tuning	Learn task-specific prompt tokens and freeze the backbone	(Wang et al., 2024, Shernazarov et al., 23 Mar 2026)
Dynamic or data-derived prompts	Initialize or generate prompts from downstream prototypes or instance features	(Wang et al., 2024, Xiao et al., 22 Mar 2025, Jain et al., 2024)
Decomposed or prompt-aware adaptation	Replace long prompts with shorter prompts plus low-rank factors, or modulate hidden states using prompt-conditioned vectors	(Lan et al., 2024, Shi et al., 2023, Liu et al., 3 Feb 2025)

A first axis is static versus dynamic prompting. Static methods learn one task-level prompt set. Dynamic methods generate prompts conditioned on target data or on the current instance. In Self-Prompt Tuning for ViTs, prompts are initialized from downstream token prototypes because prompt tokens and patch tokens exhibit increasing normalized mutual information during successful training. In V$^2$APT, a VAE maps image-specific features to instance prompts and concatenates them with domain prompts, keeping the total prompt token count equal to VPT’s budget (Wang et al., 2024, Xiao et al., 22 Mar 2025).

A second axis is prompt length versus factorization. EPT decomposes a long prompt $P\in\mathbb{R}^{l\times d}$ into a shorter prompt $P_s\in\mathbb{R}^{s\times d}$ and low-rank matrices $A\in\mathbb{R}^{m\times r}$ and $B\in\mathbb{R}^{r\times d}$ under the constraint

$[\mathrm{MASK}]$0

DePT uses the related decomposition $[\mathrm{MASK}]$1 so that a shorter prompt is paired with a global low-rank embedding update, reducing the effective sequence length without changing trainable parameter size (Lan et al., 2024, Shi et al., 2023).

A third axis is input-level prompts versus internal prompt-aware modulation. LoPA keeps the FM frozen and constructs a prompt

$[\mathrm{MASK}]$2

where $[\mathrm{MASK}]$3 is a shared task component and $[\mathrm{MASK}]$4 is a low-rank instance-specific component produced by an encoder. PARA moves further inward: a vector generator pools prompt hidden states and outputs $[\mathrm{MASK}]$5, $[\mathrm{MASK}]$6, and $[\mathrm{MASK}]$7, which multiplicatively scale $[\mathrm{MASK}]$8, $[\mathrm{MASK}]$9, and the FFN up-projection stream $E_0=\mathrm{PatchEmbed}(x)+E_{\mathrm{pos}}$0 at each transformer layer. This preserves a prompt-centric adaptation view while avoiding explicit prompt-token growth inside the sequence (Jain et al., 2024, Liu et al., 3 Feb 2025).

A fourth axis is regularized prompting. PTP identifies instability in vanilla prompt tuning as a consequence of a precipitous loss landscape and adds random-noise or adversarial perturbation regularizers in text or embedding space. The perturbed objective smooths the local landscape and improves both mean performance and variance across random seeds (Chen et al., 2023).

3. Vision-specific development

Vision has produced some of the clearest PFTS formulations. Visual Prompt Tuning inserts learned tokens into frozen ViTs, but early work exposed three recurrent weaknesses: initialization sensitivity, prompt-length sensitivity, and subpar performance on self-supervised backbones such as MAE and MoCo-v3. Self-Prompt Tuning addresses these by computing layer-wise downstream token prototypes from target data and using them to initialize prompts. Under MAE pretraining, it improves average accuracy by 10–30% relative to VPT, surpasses full fine-tuning in 19 out of 24 FGVC and VTAB-1K tasks, and keeps learnable parameters below 0.4% (Wang et al., 2024).

Dynamic prompting in vision is exemplified by V$E_0=\mathrm{PatchEmbed}(x)+E_{\mathrm{pos}}$1APT. A VAE encoder produces a Gaussian posterior $E_0=\mathrm{PatchEmbed}(x)+E_{\mathrm{pos}}$2, samples $E_0=\mathrm{PatchEmbed}(x)+E_{\mathrm{pos}}$3, and decodes $E_0=\mathrm{PatchEmbed}(x)+E_{\mathrm{pos}}$4 into instance-dependent prompts that are concatenated with static domain prompts. The training objective combines task loss and KL regularization,

$E_0=\mathrm{PatchEmbed}(x)+E_{\mathrm{pos}}$5

On HTA, V$E_0=\mathrm{PatchEmbed}(x)+E_{\mathrm{pos}}$6APT improves over VPT-Deep by +3.2%, and across FGVC, HTA, and VTAB-1k it reports an average +2.0% improvement, with the strongest gains on Structured tasks (Xiao et al., 22 Mar 2025).

High-resolution medical imaging has motivated a distinct prompt formulation. Fine-grained Prompt Tuning freezes a ViT-B encoder on $E_0=\mathrm{PatchEmbed}(x)+E_{\mathrm{pos}}$7 images, trains a lightweight side network on $E_0=\mathrm{PatchEmbed}(x)+E_{\mathrm{pos}}$8 inputs, and uses fine-grained prompts as cross-attention queries into frozen high-resolution features. Important-token selection and preloading reduce both training cost and memory. On four medical datasets, FPT reaches 92.26 average AUC versus 93.96 for full fine-tuning, while using 1.81% of trainable parameters and 3,182 MB rather than 24,116 MB at $E_0=\mathrm{PatchEmbed}(x)+E_{\mathrm{pos}}$9 (Huang et al., 2024).

Pro-tuning generalizes the vision prompt idea beyond ViTs to CNNs and dense prediction. After each stage, a prompt block produces a residual $[x_1,Z_1,E_1]=B_1([x_0,P_0,E_0]),$0 and blends it as $[x_1,Z_1,E_1]=B_1([x_0,P_0,E_0]),$1, with the backbone frozen and only prompt blocks plus head trained. The method reports 25x–97x fewer trainable parameters than full fine-tuning and shows advantages not only on generic classification but also on class imbalance, image corruption, adversarial robustness, out-of-distribution generalization, object detection, and semantic segmentation (Nie et al., 2022).

Federated vision tuning has introduced a further specialization. PEP-FedPT keeps prompts global but personalizes them per sample with Class-Contextualized Mixed Prompt. A bank of class-specific prompt tokens is mixed using weights derived from global class prototypes and client class priors, without storing client-specific trainable parameters. On CIFAR-100 with pathological heterogeneity, it reaches 95.46% mean and 84.74% worst-client accuracy; on Tiny-ImageNet pathological splits it reaches 91.52% and 77.33%, and held-out client performance remains strong rather than collapsing as in some personalized baselines (Yashwanth et al., 29 Oct 2025).

4. Language, code, and clinical text adaptation

In NLP, the simplest PFTS remains soft prompt tuning over a frozen LM. For PubMed medical summarization with Flan-T5, only encoder-side prompt vectors are trained, initialized from the embedding of the instruction text “Summarize the following medical article:”. With Flan-T5-Base, prompt tuning peaks around 20–40 prompt tokens, and with Flan-T5-Large it achieves $[x_1,Z_1,E_1]=B_1([x_0,P_0,E_0]),$2 ROUGE-1 using 0.005% trainable parameters. The same study reports prompt-tuning storage per task of approximately 160 KB, while also noting that LoRA remains stronger on this benchmark (Shernazarov et al., 23 Mar 2026).

Discrete prompting has also been unified across tasks. Unified Prompt Tuning introduces the Prompt-Options-Verbalizer paradigm and a Knowledge-enhanced Selective Masked Language Modeling objective so that a BERT-style encoder acquires task-invariant prompting knowledge before few-shot adaptation. With self-ensemble, UPT-SE reports an average accuracy of $[x_1,Z_1,E_1]=B_1([x_0,P_0,E_0]),$3 across nine datasets, compared with $[x_1,Z_1,E_1]=B_1([x_0,P_0,E_0]),$4 for LM-BFF and $[x_1,Z_1,E_1]=B_1([x_0,P_0,E_0]),$5 for P-Tuning in the same setting (Wang et al., 2022).

Prompt-based Continued Pre-training addresses a different mismatch. Conventional TAPT frequently harms prompt-based fine-tuning, especially on sentence-pair tasks, because it exposes the model to in-domain text but not to prompt templates or label words. PCP first pseudo-labels unlabeled data with a prompt-based classifier, then renders a corpus where the $[x_1,Z_1,E_1]=B_1([x_0,P_0,E_0]),$6 is replaced by the label word inside the prompt template, and finally performs MLM-style continued pre-training on that corpus before prompt-based fine-tuning. On semi-supervised QNLI with soft prompts, the reported gain is 64.2 to 84.3, a +20.1 absolute improvement (Shi et al., 2023).

Code intelligence has shown similar behavior. On defect prediction, code summarization, and code translation with CodeBERT and CodeT5, prompt tuning consistently outperforms fine-tuning. In low-resource code summarization, the reported BLEU improvement over fine-tuning is more than 26% on average, and even zero-shot prompting with a verbalizer can surpass some few-shot fine-tuning settings (Wang et al., 2022).

Prompting can also inject task-specific side information. Feature Prompt Tuning for few-shot readability assessment converts handcrafted linguistic features into trainable soft prompts via a MultiHeadMLP and adds a ListMLE-based similarity calibration loss over class-level feature relations. The method outperforms prior prompt-based and feature-fusion baselines across ChineseLR, WeeBit, and Cambridge, and also surpasses gpt-3.5-turbo-16k in most reported settings (Wang et al., 2024).

Clinical IE has moved prompt-based PEFT toward instruction-following generation. In a five-dataset study of patient information extraction, decoder-only LLMs with LoRA and instruction prompts reach the best average scores: GatorTronLlama-8B + LoRA reports 0.8981 micro-F1 for concept extraction and 0.8978 for relation extraction, while multi-task instruction tuning substantially improves zero-shot and few-shot transfer under leave-one-dataset-out evaluation (Peng et al., 5 Sep 2025).

5. Efficiency, transfer, and systems deployment

A major branch of PFTS research concerns training efficiency rather than only parameter count. Fast Prompt Tuning exploits the empirical similarity of prompts learned by partial PLMs of different widths and depths. It first tunes prompts on smaller partial models and progressively expands toward the full model, recycling the learned prompt at each stage. On T5-LARGE, compound reduction uses 65% FLOPs and 70% wall-clock relative to vanilla prompt tuning, corresponding to roughly 35% FLOPs and 30% time saved while maintaining comparable average performance across five tasks (Huang et al., 2022).

In few-shot NLG, Scaled Prompt-Tuning modifies prompt tuning with a trainable per-prompt-token scaling vector $[x_1,Z_1,E_1]=B_1([x_0,P_0,E_0]),$7 so that the encoder input becomes $[x_1,Z_1,E_1]=B_1([x_0,P_0,E_0]),$8. On T5-large, both PT and SPT use only 0.007% trainable parameters, but SPT consistently improves over PT on WebNLG 2020, E2E, and DART. For example, on WebNLG 2020 at 50-shot, PT reports 44.9 BLEU and SPT 50.2; on E2E at 50-shot, PT reports 55.2 and SPT 60.0 (Hu et al., 2023).

Deployment-oriented PFTS has emphasized single-backbone multi-tenant serving. LoPA keeps the FM frozen, computes instance-aware prompts client-side, and requires no server-side adapter. Averaged across all tasks and foundation models in the reported evaluation, it improves 28.52% over PT and 20.16% over IDPG, trails LoRA by only 0.54%, and outperforms LoRA in 11 out of 24 cases (Jain et al., 2024).

PARA targets the same setting from inside the transformer. Each layer pools the instruction hidden states, passes them through a low-rank vector generator, and produces multiplicative vectors for $[x_1,Z_1,E_1]=B_1([x_0,P_0,E_0]),$9, $[X_i,Z_i,E_i]=B_i([X_{i-1},P_{i-1},E_{i-1}]).$0, and $[X_i,Z_i,E_i]=B_i([X_{i-1},P_{i-1},E_{i-1}]).$1. On LLaMA-2 7B with rank $[X_i,Z_i,E_i]=B_i([X_{i-1},P_{i-1},E_{i-1}]).$2, PARA uses 8.9M tunable parameters and matches or exceeds LoRA and AdaLoRA on most reported tasks. In a multi-tenant inference setting, it delivers 32.8 tokens/s at beam size 1 versus 25.1 for LoRA, with slightly lower peak memory (Liu et al., 3 Feb 2025).

Speech adaptation has produced another deployment-relevant variant. For LLM-based ASR, two-step soft prompt fine-tuning first learns a domain-specific pseudo audio prompt $[X_i,Z_i,E_i]=B_i([X_{i-1},P_{i-1},E_{i-1}]).$3 on text-only data with all ASR parameters frozen, then freezes $[X_i,Z_i,E_i]=B_i([X_{i-1},P_{i-1},E_{i-1}]).$4 and fine-tunes the decoder conditioned on that prompt. The method is explicitly designed to reduce prompt mismatch between text-only adaptation and audio-conditioned inference. On target domains, it reports relative improvements of up to 9% WER reduction and up to 18% EER reduction, and external LM fusion adds a further relative 2–5% EER improvement (Ma et al., 2024).

6. Limitations, failure modes, and open directions

Several recurring limitations cut across the literature. Static prompts can be fragile with respect to initialization and prompt length, especially in vision. VPT variants show notable sensitivity to random initialization and prompt length, whereas SPT was proposed precisely because prompt tokens and patch tokens exhibit a measurable correlation structure that random initialization ignores. The same general issue appears in language summarization, where prompt tuning peaks around 20–40 tokens and slightly degrades beyond that range (Wang et al., 2024, Shernazarov et al., 23 Mar 2026).

Training instability is another recurrent failure mode. PTP attributes large variance across random seeds to a sharp loss landscape in vanilla prompt tuning and shows that perturbation-based regularizers can smooth this landscape. The reported gains are +1.94% on SuperGLUE and +2.34% on FewGLUE relative to strong prompt-tuning baselines, with large reductions in seed variance on tasks such as RTE and WSC (Chen et al., 2023).

Prompt-model mismatch remains unresolved in several settings. PCP shows that continued pre-training on unlabeled task text alone can be detrimental for prompt-based fine-tuning, especially on sentence-pair tasks, because the model is not exposed to the same prompt templates and label words that will appear at adaptation time. In ASR, text-only decoder fine-tuning without a learned pseudo audio prompt creates an analogous mismatch between text-conditioned training and audio-conditioned inference (Shi et al., 2023, Ma et al., 2024).

Dynamic prompting introduces its own costs. V$[X_i,Z_i,E_i]=B_i([X_{i-1},P_{i-1},E_{i-1}]).$5APT adds a VAE encoder and decoder and requires tuning a KL-regularized latent model; its behavior depends on latent dimensionality, prompt length, and KL weight, and the paper does not report numeric ablations for these hyperparameters. More generally, decomposed and prompt-aware methods such as EPT, DePT, LoPA, and PARA replace prompt-length sensitivity with new trade-offs in rank, subspace count, gating behavior, or auxiliary generator design (Xiao et al., 22 Mar 2025, Lan et al., 2024, Shi et al., 2023, Liu et al., 3 Feb 2025).

Theory is still incomplete. Several papers offer empirical or local explanatory analyses—SPT uses normalized mutual information, PEP-FedPT derives a quadratic upper bound and an MMSE interpretation for CCMP, and PTP visualizes sharp versus smooth loss surfaces—but SPT explicitly notes that no formal theory is provided. This suggests that PFTS remains primarily an engineering discipline organized around interface alignment: the most successful variants systematically reduce the mismatch between the frozen backbone’s pretrained computation and the downstream task’s conditioning structure (Wang et al., 2024, Yashwanth et al., 29 Oct 2025).

A plausible synthesis is that current progress is converging on three design principles. First, prompts work best when they are statistically aligned with the target data, as in prototype initialization and feature-derived prompting. Second, prompts benefit from being structurally matched to the frozen model’s interface, as in PCP for masked LMs and soft pseudo-audio prefixes for ASR. Third, prompt efficiency increasingly depends on systems considerations such as multi-tenant serving, federated aggregation, and high-resolution memory constraints, rather than on parameter count alone.