Prompt Tuning Frameworks

• Prompt Tuning Frameworks are methods that inject a small, learnable prompt into pretrained models to enable efficient task adaptation with minimal trainable parameters.
• They encompass diverse strategies—such as encoder-based, decomposition, and mixture-of-experts approaches—that address challenges like sensitivity, instability, and domain transfer.
• Applications span language, vision, and multimodal tasks, achieving competitive performance to full fine-tuning while significantly reducing computational and memory costs.

Prompt tuning frameworks are parameter-efficient adaptation methods for large pretrained models, leveraging learnable prompt tokens or modules—rather than full model tuning—to enable rapid adaptation to new tasks. These frameworks encompass a growing taxonomy of architectures, parameterizations, and transfer strategies for language, vision, and multimodal domains. Modern prompt tuning methods achieve near-parity with full fine-tuning on many downstream tasks while using orders-of-magnitude fewer trainable parameters, making them central to contemporary parameter-efficient transfer learning.

1. Fundamental Concepts in Prompt Tuning

Prompt tuning freezes the majority (often all) of the underlying pretrained model’s weights and prepends or injects a small, learnable prompt, typically realized as a sequence of trainable vectors $P \in \mathbb{R}^{k \times d}$. Here, $k$ is the prompt length and $d$ the embedding dimension. This prompt is combined with the input (either via concatenation for LLMs, or by injecting tokens/features for vision models) and only the prompt parameters are updated during downstream training. For typical set-ups (e.g., $k=100$, $d=768$, $|\theta| \sim 220$M), this yields less than $0.1\%$ additional trainable parameters (Razdaibiedina et al., 2023, Li et al., 8 Jul 2025).

The standard learning objective (language setting) freezes model weights $\theta$ and learns $P$ to optimize:

$$L(P) = \mathbb{E}_{(x,y)\sim\mathcal{D}} [ \ell( f_\theta([P; x]), y ) ]$$

where $\ell$ is the downstream loss and only $P$ is updated (Li et al., 8 Jul 2025).

Despite extreme parameter-efficiency, vanilla prompt tuning can be prone to instability, sensitive to initialization and hyper-parameters, and often delivers lower accuracy on smaller models or in few-shot regimes compared to full fine-tuning or more sophisticated PEFT (parameter-efficient fine-tuning) methods (Razdaibiedina et al., 2023).

2. Taxonomy of Prompt Tuning Frameworks

Prompt tuning approaches can be organized along two principal axes: (i) prompt parameterization/learning strategy, and (ii) transfer/initialization protocol (Li et al., 8 Jul 2025). The main categories are:

Direct prompt learning: The prompt is initialized (either randomly or via heuristics) and trained directly for the target task, with no transfer from other tasks.

• General optimization approaches: Pure soft-prompt tuning [Lester et al., 2021].
• Encoder-based methods: The prompt is generated by a small model (e.g., MLP, LSTM, residual MLP) from a lower-dimensional code or past embedding (Razdaibiedina et al., 2023, Liu et al., 2022).
• Decomposition strategies: The prompt $P$ is factorized into low-rank matrices or decomposed into multiple submodules for efficiency and modularity.
• Mixture-of-Experts (MoE) frameworks: A pool of expert prompts is combined via gating/routing depending on each input (Li et al., 8 Jul 2025).

Transfer learning prompt tuning: Leverages knowledge from previously trained prompts, either through initialization, joint multitask tuning, or prompt composition, to improve adaptation on the target or downstream tasks.

• General transfer approaches: Universal or source-specific prompts adapted to new tasks (e.g., SPoT, ATTEMPT).
• Encoder-based transfer methods: Dual encoder architectures for task-specific and universal prompt encoding.
• Decomposition-based transfer strategies: Parameter-sharing via low-rank, shared components across tasks (e.g., MPT).

A high-level summary of the taxonomy and representative frameworks is given in the following table:

Framework Category	Representative Methods	Noteworthy Properties
General Optimization	Prompt Tuning, P-Tuning v2	k·d params, highly hyperparam. sensitive
Encoder-based	RPT, Prefix Tuning, LPT	Added encoder structure, improved stability
Decomposition	DPT, DePT, EPT	Low-rank, fusion, multi-space, efficient
MoE	SMoP, PT-MoE, ComPT, DynaPrompt	Dynamic adaptation, modularity
General Transfer	SPoT, ATTEMPT, MVLPT, ComPT	Fast transfer, effective few-shot
Encoder-based Transfer	TransPrompt, CTPT	Dual encoder, handles zero-shot/tasks
Decomposition Transfer	MPT	Shared+task-specific decomposition

3. Technical Innovations and Representative Frameworks

3.1. Encoder-based Prompt Tuning

Encoder-based variants use functionally parameterized modules to improve the expressivity and robustness of prompt adaptation. For example, Residual Prompt Tuning (Razdaibiedina et al., 2023) applies a two-layer MLP with residual addition to each prompt token, yielding:

$\varphi(P_i) = P_i + f_\theta(P_i)$

where $f_\theta$ is the MLP. This structure improves both optimization smoothness and learning-rate/init robustness, yielding +7–9 points improvement over vanilla prompt tuning on SuperGLUE, with insensitivity to prompt length and initialization scheme.

Late Prompt Tuning (Liu et al., 2022) injects the prompt not at input but at an intermediate layer ($\ell$) of the transformer, generated by a neural module conditioned on the pre-prompted hidden states. This shortens the loss gradient path and enables instance-aware prompt generation. LPT matches full fine-tuning with 2x speedup and 56.6% memory reduction in large models.

Global Prompt Cell (Liu et al., 2023) instead maintains a dynamic prompt state at each transformer layer, fusing prior prompt states via learned remembering (W_R) and forgetting (W_F) matrices. This preserves rich prompt signal and accelerates convergence, with +5.8% SuperGLUE gains over vanilla prompt tuning.

3.2. Decomposition and Fusion

Decomposition approaches factorize soft prompts for parameter reduction and flexibility. For instance, Efficient Prompt Tuning (EPT) (Lan et al., 2024) decomposes a long prompt $P$ into a shorter prompt $P_s$ and two low-rank matrices A, B, and then applies prompt fusion with attention and multi-space projection:

• Reduces training time by 14%
• Gains +1.0% GLUE and +28.8% relative SuperGLUE accuracy over vanilla prompt tuning at fixed parameter budget

Structured Prompt Tuning (Liu et al., 2022) generates per-task soft prompts via a hypernetwork $H(e_t)$ mapping compact task embeddings to full prompt vectors, providing parameter sharing and increased stability (+1.2–1.5 GLUE points) especially in multi-task settings.

DePT (Zhang et al., 2023) isolates base-specific and task-shared knowledge into separate feature spaces via learnable channel-wise transformations, addressing the base-new tradeoff in transfer scenarios.

3.3. Mixture-of-Experts, Modular, and Dynamic Prompting

Mixture-of-Experts frameworks, such as SMoP, PT-MoE, and ComPT (Pouramini et al., 2024), allow dynamic prompt composition from a shared bank of expert prompts, often via softmax gating or learned weights. This improves adaptation, modularity, and robustness in multitask and few-shot regimes.

• ComPT composes each target prompt from private (task-specific) and shared (source) prompts, outperforming vanilla PT by 10+ GLUE points in 8-shot setups.

Dynamic frameworks, such as DynaPrompt (Xiao et al., 27 Jan 2025), maintain an online buffer of prompts optimized selectively at test-time via entropy/sensitivity-based selection. DynaPrompt improves OOD robustness by 1–3% over static and per-sample test-time tuning.

3.4. Multimodal and Vision-specific Prompt Tuning

Prompt frameworks have been extended to vision (Le et al., 31 Jan 2025), vision-language (Shen et al., 2022), and speech domains (Chang et al., 2022). Visual Prompt Tuning (VPT) and variants, such as VAPT, interpret prompt tokens as additional "experts" in a MoE self-attention, and further improve expressiveness by introducing adaptive prompt experts parameterized by downstream input (Le et al., 31 Jan 2025).

Attribute- and structure-aware frameworks (e.g., IntCoOp (Ghosal et al., 2024)) infuse semantic compositionality via explicit attribute-token injection, boosting few-shot transfer and interpretability on CLIP and derived tasks.

In federated contexts, PEP-FedPT (Yashwanth et al., 29 Oct 2025) introduces per-sample, class-contextualized mixed prompts, enabling robust adaptation across heterogeneous clients with only globally shared prompt parameters.

4. Empirical Performance and Comparative Analysis

Prompt tuning frameworks consistently achieve strong parameter efficiency, with trainable parameter counts in the $0.001\%$–$0.1\%$ range relative to full fine-tuning. On standard NLU and NLG benchmarks (GLUE, SuperGLUE, etc.):

• Direct prompt tuning can lag full fine-tuning by 1–2% on large LMs, but can underperform by 7–10% on smaller models (Razdaibiedina et al., 2023, Li et al., 8 Jul 2025).
• Encoder-based and decomposition strategies (ResPT, GPC, EPT) recover much of the lost accuracy, improve convergence, and sharply reduce variance/sensitivity to initialization and LR (Razdaibiedina et al., 2023, Liu et al., 2023, Lan et al., 2024).
• Modular/mixture/multitask frameworks (ComPT, MVLPT) yield pronounced advantages in few-shot and domain-transfer settings, outperforming vanilla PT by 7–12% relative accuracy in 8–shot regimes (Pouramini et al., 2024, Shen et al., 2022).
• Vision and multimodal frameworks (VAPT, Pro-tuning, IntCoOp) reach or surpass full-tuned baselines using less than 1% of the parameter budget (Le et al., 31 Jan 2025, Nie et al., 2022, Ghosal et al., 2024).
• For domain generalization and OOD robustness, prompt tuning substantially outperforms static or head-only adaptation (Xiao et al., 27 Jan 2025).

5. Practical Considerations and Application Guidelines

The selection of a prompt tuning framework should be guided by task requirements, compute/memory constraints, and domain. Key recommendations from meta-studies (Li et al., 8 Jul 2025, Pouramini et al., 2024):

• For compute-constrained scenarios and large LLMs: vanilla or P-Tuning v2 is efficient, but one should pivot to residual or structured parametric encoders (RPT, LPT, GPC, EPT) if training is unstable.
• To maximize few-shot performance and domain robustness, use mixture-of-experts (SMoP, PT-MoE, DynaPrompt) or modular transfer (ComPT, MVLPT) approaches.
• For multitask or multi-domain adaptation, prompt composition frameworks that balance shared and private prompts (ComPT's SSUM/MSUM strategies, MVLPT's multitask groups) yield superior sample efficiency.
• In federated or privacy-sensitive settings, approaches like PEP-FedPT allow per-sample adaptation without local parameter proliferation.
• For vision and multimodal tasks, use architecture-aligned prompt insertion (deep visual token prepending for ViTs; attribute-injection for vision-language) to maximize representational synergy.

General hyperparameter guidelines:

• Moderately long prompts ($k=60$–$100$) strike a balance between efficiency and accuracy; fusion and decomposition recover expressivity with shorter $k$.
• Encoder depth and insertion position are critical; late or multi-layer prompt injection improves gradient flow and adapts the effective receptive field (Liu et al., 2022).
• Warm-start prompt initialization from pretrained embeddings or multitask seeds enhances stability and transfer.

6. Limitations and Open Challenges

• Accuracy gap: While reduced, full fine-tuning still outperforms prompt tuning by 1–10% depending on scale and model/domain (Razdaibiedina et al., 2023, Li et al., 8 Jul 2025).
• Convergence time: Some methods require many more epochs to reach peak accuracy compared to fine-tuning or adapters (Yang et al., 2022).
• Sensitivity: Despite progress, vanilla prompt tuning remains highly sensitive to initialization and learning rate, especially in low-shot regimes (Razdaibiedina et al., 2023).
• Expressiveness: Soft prompt methods are limited to biasing the representation manifold and may not achieve arbitrary function transfer (Le et al., 31 Jan 2025). Input-adaptive and multimodal prompt experts are active areas of development.
• Transfer and Multi-domain Robustness: Solving the base-new tradeoff (BNT) and task interference for multitask prompts is nontrivial; techniques like DePT and modular composition are partial solutions (Zhang et al., 2023, Pouramini et al., 2024).
• Scaling to new domains: Application to generative, sequential, or underexplored domains (e.g., speech, video) is nascent (Chang et al., 2022, Yang et al., 2022).

Major open directions include more robust and theoretical justified prompt pretraining, meta-learned or instance-adaptive prompt generators, deeper integration of composition and mixture-of-experts logic, and efficient transfer protocols for ever larger, cross-modal models (Li et al., 8 Jul 2025).

7. Impact and Future Directions

Prompt tuning frameworks have enabled scalable, parameter-efficient adaptation of foundation models across language, vision, speech, and multimodal applications, dramatically reducing the environmental and logistical cost per downstream task. Their modular and transfer-aware designs facilitate rapid deployment in compute- and data-limited settings and create new avenues for scaling multitask, federated, and continual learning systems. Future work will address the limitations of expressiveness, transfer stability, and automated prompt architecture design, aiming for universal, robust, and explainable adaptation modules.

References: (Razdaibiedina et al., 2023, Li et al., 8 Jul 2025, Liu et al., 2022, Liu et al., 2023, Pouramini et al., 2024, Le et al., 31 Jan 2025, Lan et al., 2024, Liu et al., 2022, Shen et al., 2022, Yang et al., 2022, Chang et al., 2022, Yashwanth et al., 29 Oct 2025)