Prompt-Based Fine-Tuning

• Prompt-based fine-tuning is a parameter-efficient method that adapts pre-trained models by optimizing a small set of tunable prompts while keeping most model parameters fixed.
• It employs techniques like continuous soft prompts, deep layerwise injection, and hybrid prompt-parameter co-optimization to steer model behavior efficiently.
• This approach enables robust performance in low-resource, cross-lingual, and multimodal settings, offering significant parameter savings compared to full fine-tuning.

Prompt-based fine-tuning is a parameter-efficient paradigm that adapts pre-trained models to downstream tasks by optimizing a small set of “prompts” while keeping the majority of model parameters frozen. Unlike traditional fine-tuning approaches, which update the entire network, prompt-based methods steer model behavior either through the tuning of continuous prompt embeddings, discrete templates, or lightweight architectural modules inserted at key points in the network. Prompt-based fine-tuning is now established in NLP, code intelligence, and vision, and it provides competitive or superior performance to full fine-tuning across classification, structured prediction, generative, and cross-lingual tasks, especially in low-resource regimes. Furthermore, it is foundational for state-of-the-art parameter-efficient transfer learning (PEFT) systems and is under active development for robustness, efficiency, and generalizability.

1. Principles and Formalism of Prompt-Based Fine-Tuning

Prompt-based fine-tuning methods operate by learning a compact set of tunable parameters—such as continuous prompt tokens, small neural prompt generators, or specialized module weights—while freezing the pre-trained backbone. The core mechanisms include:

• Continuous (Soft) Prompts: Learnable vectors $\{p_1,\ldots,p_m\}$ appended to or injected at various layers of a frozen Transformer. These prompt tokens are optimized via task-specific loss functions (such as cross-entropy or MLM loss), with the main network weights held constant (Liu et al., 2021).
• Layerwise (Deep) Prompting: Extension of soft prompting where prompts are injected deeply at each layer. For each layer $\ell$, a learnable matrix $P_\ell \in \mathbb{R}^{m \times d}$ is prepended, yielding direct hierarchical control of model predictions and increased capacity (Liu et al., 2021, Liu et al., 19 Oct 2025).
• Unified Formulation: Let $\theta$ be the frozen pre-trained model parameters, $P$ the learnable prompts, and $(x_i, y_i)$ the dataset. The prompt-tuning loss is

$$\mathcal{L}(P) = -\sum_{i} \log p(y_i | x_i; P, \theta)$$

where $P$ may be continuous embeddings, a learned function of instance metadata, or architectural weights. Only $P$ (and possibly a light classifier or prompt encoder) is updated; all others are frozen (Liu et al., 2021, Liu et al., 19 Oct 2025, Liu et al., 2023).

• Prompt Generators and Attribute Control: Some approaches learn prompt embeddings that are instance-conditioned via a small prompt encoder $f_\varphi$ (often an MLP or shallow transformer), so that prompts are generated or predicted per input attribute (e.g., dialogue act or persona) (Liu et al., 2023).
• Hybrid Prompt-Parameter Co-Optimization: Recent frameworks directly optimize both prompts and parameter-efficient model deltas (e.g., LoRA), coupled through shared layers and regularization, to capture both explicit task specification and implicit representation adaptation (cf. MetaTuner) (Bo et al., 29 Sep 2025).

2. Prompt-Based Fine-Tuning Methodologies

A variety of prompt-fine-tuning strategies have been established, often differing in architecture, granularity, and application domain:

• P-Tuning v2: Deep prompt tuning, where soft prompt embeddings are injected at every transformer layer, achieving task universality and matching full fine-tuning with only 0.1%–3% of the parameters for diverse NLU tasks and model sizes in the 300M–10B range (Liu et al., 2021).
• Capsule Prompt-Tuning (CaPT): At each layer, a single vector formed by summing a trainable task vector and a mean-pooled instance embedding is prepended, serving as an “attention anchor.” CaPT eliminates prompt-length search, and with $\ell$0 parameters yields state-of-the-art PEFT scores on SuperGLUE, even surpassing full fine-tuning (Liu et al., 19 Oct 2025).
• XPrompt: Leverages the lottery ticket hypothesis for prompts, hierarchically pruning negative tokens and token-pieces to find a “winning ticket” prompt subspace. This rewardingly results in strong performance with an extremely compressed prompt, closing the fine-tune gap for moderate model sizes (Ma et al., 2022).
• Prompt-Agnostic Fine-Tuning (PAFT): To maximize prompt robustness and reduce overfit to one formulation, PAFT dynamically samples synthetic prompts during training (from a large LLM-generated pool), optimizing for performance averaged over many prompt surface forms. This yields superior cross-prompt generalization and substantial inference-time speed-ups (Wei et al., 18 Feb 2025).
• Attribute-Controlled and Instance-Specific Methods: Attribute Controlled Dialogue Prompting (ACDP) uses a prompt encoder to map each instance’s control code to a shallow or deep prompt embedding, achieving controllability with only 5–6% of the underlying model’s parameters (Liu et al., 2023).
• Ultra-Low-Dimensional Prompt Tuning (ULPT): Projects prompts from a low (e.g., 2D) space to the model’s dimension via a frozen random matrix, plus learnable shift and scale, drastically reducing parameters while retaining nearly all performance (Wu et al., 6 Feb 2025).

The underlying architectural motifs are well summarized below:

Method	Prompt Mechanism	Tuned Params	Highlights
P-Tuning v2	Deep soft prompts (per layer)	0.1%–3%	Universality across backbone/size/task
CaPT	Single capsule per layer	0.003%	Instance-aware, no prompt-length search
XPrompt	Hierarchical sparsified prompt	0.02%	Winning-ticket subspace, SOTA on T5
PAFT	Synthetic pool, dynamic swap	—	Robustness to prompt variation
ULPT	Ultra-low (r≪d) projections	≤2%	Minimal parameters, theoretical backing

3. Applications Across Domains

Prompt-based fine-tuning has demonstrated wide applicability and notable performance:

• Natural Language Processing: In NLU, deep prompt-tuning (P-Tuning v2) yields task-universal, parameter-efficient adaptation on classification, sequence labeling, and extractive QA across GLUE, SuperGLUE, and CoNLL-03. Prompt-tuning consistently matches or exceeds full fine-tuning across tasks/models (Liu et al., 2021).
• Cross-Lingual Transfer: Prompt-tuning excels at zero-shot cross-lingual transfer in multilingual encoders (e.g., XLM-R), achieving 0.1%–0.3% tuned parameters and yielding higher average accuracy and lower transfer gap compared to full fine-tuning across over 15 languages (Tu et al., 2022).
• Vision Models: Prompt-based fine-tuning (VPT, Pro-tuning, SPT) for vision transformers and CNNs demonstrates robust transfer and strong performance on classification, detection, and segmentation tasks. Advanced initialization (prototype-based) and hierarchical/semantic prompt designs further improve adaptation in low-data and self-supervised settings (Wang et al., 2024, Nie et al., 2022). Vision prompt tuning surpasses full fine-tuning in robustness (adversarial and distribution shift), class imbalance, and dense prediction, with parameter savings of up to 30× (Nie et al., 2022, Wang et al., 2024).
• Code Intelligence: Prompt tuning for code summarization, translation, and defect detection on models like CodeBERT and CodeT5 leads to higher BLEU, accuracy, and CodeBLEU scores than traditional fine-tuning, especially in low-resource scenarios. BLEU improvements reach +26% in few-shot summarization (Wang et al., 2022).
• Speech & Multimodal: In LLM-based ASR, two-step soft prompt adaptation enables domain-specific text injection using only a (pseudo-audio) prompt, reducing WER by up to 9% and EER by up to 18% with parameter-efficient training (Ma et al., 2024).
• Retrieval and Zero-Shot Entity Typing: Fine-grained object retrieval and fine-grained entity typing are both amenable to prompt-based fine-tuning, with discriminative perturbation prompts and prompt-verbalizer architectures substantially outperforming full fine-tuning in low-data scenarios (2207.14465, Ding et al., 2021).

4. Comparative Performance and Scaling Properties

Empirical results across tasks and model sizes consistently show the following properties:

• Parameter Efficiency: Prompt-based fine-tuning typically tunes 0.1%–3% of backbone parameters versus 100% for full fine-tuning. Ultra-low (0.003%–0.02%) regimes are now feasible with little to zero accuracy drop (Liu et al., 2021, Liu et al., 19 Oct 2025, Ma et al., 2022).
• Universality: Properly optimized deep prompt-tuning architectures (P-Tuning v2, CaPT) are applicable across GLUE/SuperGLUE, NER, QA, sequence labeling, and more, reliably matching or outperforming full fine-tuning (Liu et al., 2021, Liu et al., 19 Oct 2025).
• Low-Resource and Cross-Domain Superiority: Prompt-tuning vastly outperforms fine-tuning under few-shot, zero-shot, and domain-shifted evaluation (Tu et al., 2022, Wang et al., 2022, Ding et al., 2021). On fine-grained entity typing with 1–8 shots, prompt-based FT achieves 43.9% accuracy versus 8.9% for standard fine-tuning and maintains strong results in zero-shot contrastive settings (Ding et al., 2021).
• Ablation Insights: Gains arise from hierarchical and instance-aware prompt mechanisms (e.g., CaPT, XPrompt), robust initialization (downstream token prototypes), and multi-task pre-training of prompt modules (Liu et al., 19 Oct 2025, Ma et al., 2022, Wang et al., 2024).
• Efficiency: Training and inference memory and compute are substantially reduced. Prompt-tuned models are faster to update and deploy, allowing storage of multiple tasks as small prompt files, facilitating multi-tenancy and rapid task switching (Liu et al., 2021, Wei et al., 18 Feb 2025).

5. Recent Advances: Robustness, Co-Optimization, and Augmentation

Prompt-based fine-tuning has seen multiple refinements and extensions:

• Prompt Robustness: PAFT injects dynamic prompt variability during fine-tuning, training models to be insensitive to specific prompt phrasings. PAFT achieves the highest accuracy and lowest performance variance across held-out prompts, with a 4.25% average improvement and 3.25× faster inference vs. single-prompt LoRA baselines (Wei et al., 18 Feb 2025).
• Joint Prompt and Parameter Optimization: MetaTuner co-optimizes prompts and LoRA adapters (parameter deltas) via shared meta-encoders and a supervised regularization objective. This approach yields 5–10% gains over previous hybrid or single-branch approaches and is robust to out-of-distribution task generalization (Bo et al., 29 Sep 2025).
• Prompt Compression and Ultra-Low-Dimensionality: ULPT compresses prompt vectors into arbitrarily low dimensions (e.g., $\ell$1). Even with 2% of the prompt parameters, accuracy on GLUE/SuperGLUE and sixteen NLP/QA tasks remains within $\ell$2 of full prompt tuning (Wu et al., 6 Feb 2025).
• Contrastive and Data-Augmented Prompt Training: Contrastive paraphrasing-guided prompt-based augmentation (LM-CPPF) employs LLM-generated paraphrases to augment few-shot training and pairs a supervised contrastive loss with prompt-based MLM/CLS objectives. LM-CPPF outperforms back-translation, multi-templates, and EDA with notable gains across sentiment and entailment datasets (Abaskohi et al., 2023).
• Unified/Multi-Task Prompting: Unified Prompt Tuning (UPT) pre-trains PLMs in a prompt-style, multi-task setup with a prompt-options-verbalizer scheme, enabling rapid adaptation to unseen few-shot tasks and yielding further ensemble gains (Wang et al., 2022).
• Instance-Specific Advance: Attribute-Controlled Prompting using instance-level encoders significantly boosts control and flexibility for structured outputs or attribute-guided generation, approaching fine-tuning performance with parameter savings (Liu et al., 2023).

6. Practical Considerations and Best Practices

• Prompt Length and Placement: Empirical results recommend prompt lengths $\ell$3 for classification, $\ell$4 for harder tasks. Deep and per-layer injection is critical for sequence labeling and generative tasks (Liu et al., 2021, Liu et al., 19 Oct 2025).
• Regularization and Initialization: Random initialization suffices for prompt embeddings; downstream token prototype initialization provides further boosts in vision/multimodal settings (Wang et al., 2024). Layer normalization, AdamW, and careful learning rate scheduling remain essential.
• Task Adaptation and Transfer: Prompt-tuned models demonstrate robust cross-lingual and cross-domain transfer, preserving underlying embedding geometry and generalizing decision boundaries better than full fine-tuning (Tu et al., 2022).
• Hyperparameter Sensitivity: Prompt-finetuning is less sensitive to prompt length when proper initialization and deep architectures are used; in ultra-low-dimensional setups, prompt length and projection rank must be tuned for task complexity (Wu et al., 6 Feb 2025).
• Robustness and Efficiency: PAFT and similar prompt-agnostic methods provide strong invariance to prompt surface variability and robustness to distribution shift; such methods are recommended when deployment environments or prompt formats are expected to change (Wei et al., 18 Feb 2025).
• Compression and Storage: PEFT recipes (prompt, LoRA, adapters) enable the deployment of tens or hundreds of task-specific adapters atop a giant shared model without redundancy or high inference cost (Liu et al., 19 Oct 2025).

7. Limitations and Research Frontiers

• Task and Domain Range: Most benchmarks are NLU, classification, or structured prediction; generative, open-ended, or reinforcement learning tasks remain underexplored (Liu et al., 19 Oct 2025).
• Model Size Dependence: On very small models ($\ell$5M), full fine-tuning may retain an edge; prompt-tuning excels as model size increases ($\ell$6M) (Liu et al., 2021).
• Prompt Discovery and Automation: Automated or meta-learned prompt discovery (structure search, neural prompt generators) is an open research direction. Current approaches depend on hand-designed templates or prompt classifier heads (Wang et al., 2022).
• Theoretical Understanding: Compression theorems (e.g., Johnson–Lindenstrauss for random projections in ULPT (Wu et al., 6 Feb 2025)) and lottery ticket hypothesis for prompt sparsification (Ma et al., 2022) are promising but only partially explain empirical findings in transfer and robustness.
• Instance and Context Awareness: Fully dynamic, input-aware prompts outperform static task prompts, but require more flexible encoder architectures and possibly higher training cost (Liu et al., 2023, Liu et al., 19 Oct 2025).
• Scalability to Multimodal and Streaming Domains: Emerging work in ASR (Ma et al., 2024) and retrieval (2207.14465) highlights the promise of prompt-based fine-tuning beyond NLP and vision, but robust benchmarks and adaptation recipes are still maturing.

Prompt-based fine-tuning continues to evolve as a dominant strategy in parameter-efficient model adaptation, offering state-of-the-art accuracy, practical deployment advantages, and principled extensions for cross-task and domain invariance. Key methodological advances in deep prompting, prompt compression, instance-aware control, and robust dynamic sampling have made prompt-based fine-tuning a cornerstone of contemporary adaptive AI systems.