Fine-Tuning & Prompt Optimization

• Fine-tuning and prompt optimization are complementary strategies that adapt large pre-trained models: fine-tuning adjusts model weights while prompt optimization conditions fixed models with task-specific prompts.
• Fine-tuning offers maximum task adaptation with complete parameter updates but is resource-intensive, whereas prompt optimization uses discrete or soft prompts to achieve efficiency with minimal updates.
• Integrated frameworks that combine both methods can offer synergistic benefits, enhancing scalability, robustness, and multi-task performance in modern AI applications.

Fine-tuning and prompt optimization are two central strategies for adapting large pre-trained models—especially in natural language processing and computer vision—to diverse downstream tasks. Fine-tuning traditionally entails updating all or a subset of model parameters, enabling task-specific adaptation at the cost of high resource consumption and reduced modularity. Prompt optimization instead conditions a fixed backbone model using either discrete (textual) prompts or learnable "soft" prompts (continuous embeddings), enabling efficient task transfer and multi-task serving with a dramatically reduced tuning footprint. Recent research demonstrates that both paradigms have unique strengths, and their integration yields synergistic benefits. Below is a comprehensive overview of their principles, methodological advancements, scalability, practical implications, and emerging trends.

1. Foundations of Fine-Tuning and Prompt Optimization

Fine-tuning is the process of continuing gradient-based updates for all or many parameters of a pre-trained model using supervision from a downstream task. This results in maximum task adaptation capacity, but each task requires storing an independent set of large model weights. In contrast, prompt optimization specializes the model for a downstream task by conditioning it with a prompt rather than modifying the core model weights, thus preserving storage efficiency and deployment modularity.

Prompt-based transfer employs either:

• Discrete prompts: Manually engineered or automatically discovered phrases or templates, as in GPT-3 few-shot prompting.
• Soft prompts: Continuous vectors prepended to the input embeddings and optimized through backpropagation while the model backbone remains frozen (Lester et al., 2021).

Key mathematical formalism:

• Fine-tuning (supervised objective): $P_\theta(Y \mid X)$, $\theta$ updated.
• Prompt tuning: $P_{\theta, 0_p}(Y \mid [P; X])$, $\theta$ frozen, soft prompt parameters $P$ optimized.

Recent frameworks unify both approaches, extending the optimization space to include both prompts and model parameters jointly (Bo et al., 29 Sep 2025).

2. Mechanisms and Advances in Prompt Optimization

Prompt optimization has evolved along several dimensions:

Soft Prompt Tuning

Introduced as a parameter-efficient alternative to full model adaptation, soft prompt tuning learns a small set of additional task-specific prompt embeddings, typically requiring less than 0.1% of the total model parameters (Lester et al., 2021). The prompt embeddings $P_e \in \mathbb{R}^{p \times e}$ are prepended to the embedding sequence of each input. The optimization objective becomes: $$\max_{P_e} \mathbb{E}_{(X, Y) \sim D} \log P_{\theta, 0_p}(Y \mid [P; X])$$ where $\theta$ remains fixed.

Deep and Multi-Layer Prompting

Prompting at multiple layers ("deep prompt tuning") increases representational capacity and bridges the performance gap with full-model fine-tuning on both simple and complex NLU tasks (Liu et al., 2021). For each transformer layer, unique prompt tokens are learned and injected directly into the intermediate representations.

Prompt Compression and Structured Pruning

Hierarchical pruning (as in XPrompt) removes uninformative or deleterious tokens and sub-token pieces, exploiting the lottery ticket hypothesis to retain only the most effective components (Ma et al., 2022). Structured decomposition methods (e.g., truncated SVD or compressed outer product (Lan et al., 16 Feb 2025)) further reduce trainable parameters while maintaining or enhancing intrinsic semantic associations.

Prompt Initialization and Self-supervised Pretraining

Prompt initialization can be greatly improved by leveraging clustering or statistical pooling over downstream token embeddings, ensuring high mutual information between prompt and input token distributions (Wang et al., 4 Feb 2024). This leads to more stable convergence and higher final accuracy, especially in self-supervised visual adaptation scenarios.

Loss-driven and Merit-guided Prompt Optimization

Optimization frameworks such as PMPO utilize token-level cross-entropy loss as a direct and lightweight signal for prompt selection and refinement, ensuring model-aligned improvements without human-labeled preference data or expensive output sampling (Zhao et al., 22 May 2025). Other merit-guided approaches, such as MePO, emphasize prompt clarity, precision, and reasoning brevity to guarantee downward compatibility with lightweight inference models (Zhu et al., 15 May 2025).

3. Parameter and Resource Efficiency

Prompt optimization consistently demonstrates drastic reductions in per-task storage, memory, and computational requirements compared to full fine-tuning. For example, tuning a T5-XXL model (11B parameters) via prompt tuning requires updating less than 0.01% of parameters, versus full parameter adaptation (Lester et al., 2021). Methods such as ULPT reduce trainable prompt dimensions to mere 2D latent spaces by projecting into the model's original embedding space with a fixed random up-projection, retaining at least 97% of vanilla prompt tuning's performance with only 2% of trainable parameters (Wu et al., 6 Feb 2025).

Decomposition-based frameworks (EPT, LAMP) combine short soft prompts with low-rank updates or truncated SVD-based representations, efficiently enriching prompt representations while maintaining fixed total parameter budgets and reducing training time by up to 14% (Lan et al., 19 May 2024, Lan et al., 16 Feb 2025).

Parameter-efficient methods also facilitate serving multiple tasks with a single backbone, requiring storage and deployment only of lightweight, task-specific prompt modules.

4. Empirical Performance, Robustness, and Transferability

Soft prompt tuning, when applied to sufficiently large models, matches or even surpasses full-model fine-tuning on a wide range of tasks and resource regimes. Notable empirical findings include:

• Prompt tuning outperforms GPT-3's few-shot learning by a large margin on SuperGLUE, and a prompt-tuned T5-Large can outperform GPT-3 175B on several tasks (Lester et al., 2021).
• In NLU sequence-labeling and extractive QA, deep prompt tuning approaches or matches fine-tuning performance even at moderate model scales (Liu et al., 2021).
• For code intelligence, prompt tuning on models such as CodeBERT and CodeT5 outperforms fine-tuning in all evaluated tasks and demonstrates substantial gains (26%+ in BLEU) under low-resource settings (Wang et al., 2022).
• In cross-lingual NLU, prompt tuning achieves better cross-lingual transfer (and decision boundary alignment) than fine-tuning, with only 0.1–0.3% of parameters updated (Tu et al., 2022).

Robustness to prompt formulation and domain adaptation can be enhanced by regularizing the decision boundary (OPTIMA (Guo et al., 2022)), randomizing prompt selection during fine-tuning (PAFT (Wei et al., 18 Feb 2025)), and using multi-task or meta-learning frameworks (UPT (Wang et al., 2022)). Prompt-tuned models often generalize better across domains and unseen prompts, preserving the original knowledge of the pre-trained backbone.

5. Synergistic and Unified Optimization Frameworks

Recent research demonstrates that fine-tuning and prompt optimization are not mutually exclusive and may be integrated for improved downstream performance. Alternating or joint optimization of prompt templates and model weights (e.g., the BetterTogether strategy (Soylu et al., 15 Jul 2024) and MetaTuner (Bo et al., 29 Sep 2025)) leads to higher accuracy and sample efficiency than either approach alone. These frameworks typically employ:

• Interleaved prompt and weight updates, where the LM "teaches itself" with self-generated traces or refined outputs.
• Separate neural networks for prompt and parameter generation, with shared encoder layers facilitating knowledge exchange.
• Surrogate supervised regularization losses to unify discrete prompt generation (non-differentiable) with continuous model parameter updates.

Empirical studies reveal performance gains up to 60% over weight-only optimization and up to 6% over prompt-only optimization, as well as enhanced robustness when generalizing to new prompts or downstream tasks.

6. Limitations, Challenges, and Future Research Directions

Several open problems remain:

• The performance gap between prompt tuning and fine-tuning can persist for moderate and small model sizes; sophisticated pruning, compression, or initialization is required to "close the gap" (Ma et al., 2022).
• Prompt tuning is data-hungry and may underperform in extremely low-resource settings without regularization or multi-task pre-exposure (Guo et al., 2022).
• Sensitivity to prompt formulation and poor generalization to unseen prompts remain critical challenges; frameworks such as PAFT mitigate this by randomizing prompt exposure (Wei et al., 18 Feb 2025).
• Downward compatibility of prompts created by large LLMs for lightweight or non-instruction-tuned models is not guaranteed without explicit merit or compatibility controls (Zhu et al., 15 May 2025).
• Automated prompt engineering, including hybrid discrete-continuous search, evolutionary optimization, constrained and bi-level optimization, and agent-oriented prompt design, constitute major research frontiers (Li et al., 17 Feb 2025).

Continued integration of fine-tuning and prompt optimization—augmented with token-level metrics, preference datasets, meta-learning, and agent-based strategies—promises further advances in model efficiency, robustness, scalability, and cross-modal generalization.

7. Mathematical Formalisms and Representative Algorithms

Several mathematical expressions underpin these techniques:

• Conditional likelihood for prompt tuning:

$P_{\theta; 0_p}(Y \mid [P; X])$

with $\theta$ fixed and $P$ learned by maximizing the log-likelihood over data.

• Prompt decomposition (truncated SVD):

$$P \approx U_{:r} \cdot \mathrm{diag}(Q_{:r}) \cdot V_{:r}^\top$$

where $U$, $Q$, and $V$ are the truncated singular vectors/values of the prompt embedding.

• ULPT up-projection:

$$\mathbf{\tilde{e}}_{ij} = \left( \sum_{k=1}^r z_{ik} \tilde{p}_{kj} \right) \cdot s_j + b_j$$

for ultra-low-dimensional prompt vectors $z$, random projection $\tilde{p}$, and learnable shift/scale $b$, $s$ (Wu et al., 6 Feb 2025).

• Cross-entropy objective for PMPO prompt optimization:

$L(P) = -\sum_{i=1}^n \log P(w_i \mid w_{1:i-1}; P)$

and prompt search:

$P^* = \arg \min_P L(P)$

• Joint co-optimization (MetaTuner):

$$\min_{\theta, p_i} \sum_{i=1}^N \mathcal{L}(M_\theta(p_i, x_i), y_i)$$

where $p_i$ is generated by a prompt network and $\theta$ by a parameter network with shared representations to enable co-adaptation.

These formulations provide the basis for systematic, scalable, and efficient fine-tuning and prompt optimization across a wide spectrum of model architectures and downstream domains.