Parameter-Efficient Fine-Tuning (PEFT)

• PEFT is a set of techniques that selectively updates parts of pre-trained models to adapt efficiently to new tasks.
• It reduces computational, memory, and storage resources by fine-tuning only key parameters, enabling rapid deployment.
• PEFT balances efficiency with performance, mitigating overfitting and catastrophic forgetting while trading off some capacity.

Parameter-Efficient Fine-Tuning (PEFT) is a class of adaptation techniques for large pre-trained models that aims to retain high task performance while modifying only a small portion of model parameters. The primary motivation behind PEFT arises from the prohibitive computational, memory, and storage costs associated with full fine-tuning of large models, especially in the context of LLMs, vision transformers, and multimodal foundation models. PEFT strategies address efficiency, overfitting, and catastrophic forgetting by optimizing the most relevant or efficient subset of parameters, allowing for rapid adaptation to downstream tasks with minimal resource overhead.

1. Foundational Concepts and Taxonomy

PEFT approaches are generally categorized by how they restrict, augment, or reparameterize the base model during adaptation. The principal families include:

• Selective Fine-Tuning: Only a carefully selected subset of original model parameters are updated. This includes fixed strategies (e.g., tuning just the last layers, only biases in BitFit) as well as data-driven automatic selection methods (e.g., magnitude-based masking, Hessian-informed methods) (Zhang et al., 23 Jan 2025, Xu et al., 18 May 2025).
• Additive Methods: Lightweight modules such as adapters or soft prompts are inserted into the architecture, with all backbone parameters kept frozen (Han et al., 21 Mar 2024, Prottasha et al., 19 Apr 2025). Adapters can be serial (interleaved between layers) or parallel (added alongside main computations).
• Reparameterization Methods: Learnable, typically low-rank updates are introduced, as in LoRA, where a large weight matrix $W$ is updated as $W + \Delta W$ with $\Delta W$ factorized into low-rank matrices $A$ and $B$ (Zhang et al., 23 Jan 2025, Han et al., 21 Mar 2024).
• Hybrid and Unified Frameworks: These methods combine aspects of the above, such as adapters fused with LoRA-style low-rank updates, with or without prompt-based front ends. Recent work further integrates MoE-style (Mixture-of-Experts) routing and selection within the PEFT framework (Liu et al., 12 Nov 2024, Prottasha et al., 19 Apr 2025).

The mechanisms governing “which” parameters are updated, “where” in the network adaptation occurs, and “how” new modules or low-rank matrices are added define the distinctions and trade-offs between each category.

2. Representative Methodologies and Mathematical Formulations

PEFT methods are concretely characterized by their update equations and architectural integration:

• Selective Methods: BitFit updates only bias parameters (e.g., in attention layers), so that in the formula $Q(x) = W_q x + b_q$, only $b_q$ is trainable (Zhang et al., 23 Jan 2025).
• Additive Methods: Bottleneck adapters perform a down-projection, nonlinearity, and up-projection, then add the projected output back to the residual:

$$\text{Adapter}(x) = W_\text{up} \sigma(W_\text{down} x) + x$$

• LoRA: The reparameterized update is expressed as

$$W' = W + \Delta W, \quad \text{with} \quad \Delta W = B A$$

where $A \in \mathbb{R}^{r \times d}$ and $B \in \mathbb{R}^{d \times r}$ and $r \ll d$ (Zhang et al., 23 Jan 2025).

• Prompt Tuning: Soft or continuous prompt vectors are learned either at the input or at specific model layers and concatenated/inserted without changing backbone weights.
• Advanced Selection: Hessian-informed techniques (e.g., AdaPEFT) quantify the influence of each parameter group via a second-order approximation, casting subset selection as a 0-1 knapsack problem under Pareto optimality to maximize loss reduction per parameter trained (Xu et al., 18 May 2025).
• Matrix Decomposition Perspective: Recent analysis reframes PEFT as subspace reconstruction (adjusting singular vectors/values for improved alignment with optimal weight space) and subspace extension (adding low-rank corrections), providing a unified mathematical lens for both additive and reparameterization-based methods (Si et al., 7 Jul 2024).

3. Empirical and Domain-Specific Applications

PEFT methods have demonstrated effectiveness in a wide spectrum of domains:

• LLMs: Techniques such as LoRA, BitFit, and adapters obtain near full fine-tuning performance on benchmarks like GLUE and SuperGLUE by updating less than 1% of parameters (Pu et al., 2023, Han et al., 21 Mar 2024, Prottasha et al., 19 Apr 2025).
• Vision and Vision-LLMs: Parallel and serial adapters, as well as prompt-based methods, have been extended to vision transformers and segmentation models, with cross-block orchestration further improving transferability in high-dimensional output spaces (Peng et al., 2023).
• Multi-Profile Personalization: X-PEFT leverages binary mask tensors to combine a library of adapters for efficient personalization in multi-profile deployments, reducing per-profile storage by factors up to 10,000 (Kwak et al., 29 Jan 2024).
• Specialized Science and Engineering: In full-waveform seismic inversion, LoRA-based PEFT achieves strong generalization and memory efficiency when adapting foundational models across diverse geological scenarios (Ghosal et al., 27 Dec 2024).
• Other Modalities: Spectral domain adaptations (PointGST) for point cloud learning (Liang et al., 10 Oct 2024) and experiment-driven approaches for code change learning (Liu et al., 9 Feb 2024) confirm the suitability of PEFT in non-traditional domains.

4. Computational Efficiencies and Trade-Offs

PEFT’s main resource advantages are:

• Parameter Reduction: Typical PEFT setups update fewer than 1% of parameters—three to four orders of magnitude less than full fine-tuning—which results in lower GPU memory requirements and accelerated adaptation cycles (Zhang et al., 23 Jan 2025, Han et al., 21 Mar 2024, Balne et al., 21 Apr 2024).
• Storage and Deployment: For multi-task or federated settings, only adapter weights, low-rank matrices, or mask tensors need to be stored or transferred, facilitating deployment on storage-constrained devices or privacy-preserving cloud-offsite architectures (Liao et al., 2023, Kwak et al., 29 Jan 2024).
• Training Speed and Stability: While full fine-tuning often converges faster in low-resource settings, PEFT methods become more performant and stable with abundant data, and allow rapid model proliferation for serving many downstream tasks or users (Pu et al., 2023).
• System Integration: Scalable system-level serving (e.g., PetS, Offsite-Tuning) leverages single-copy backbone models with modular PEFT head swapping for efficient inference across tasks (Han et al., 21 Mar 2024).

The principal trade-off is that aggressive parameter reduction can limit representational capacity, especially in complex, knowledge-intensive regimes or tasks requiring substantial projection reconfiguration.

5. Challenges, Limitations, and Theoretical Considerations

Despite substantial empirical gains, open challenges persist:

• Hyperparameter Sensitivity: Choices such as adapter bottleneck size, LoRA rank, or soft prompt length have pronounced, sometimes non-monotonic, performance impact, requiring careful tuning and highlighting the need for robust, adaptive selection schemes (Wu et al., 23 Feb 2024, He, 25 Nov 2024).
• Subset Selection: As shown by AdaPEFT, not all parameter groups are equally important; the selection of which subset to adapt is best determined via principled, data-driven strategies that consider their influence on downstream loss, leveraging gradient and Hessian information (Xu et al., 18 May 2025).
• Convergence Speed: In low data regimes, PEFT can converge more slowly than full fine-tuning and may require larger data volumes or adapted hyperparameters to achieve parity in training efficiency (Pu et al., 2023).
• Capacity, Modularity, and Compositionality: There is a trade-off between the expressiveness of lightweight modules and their parameter cost. Recent approaches (e.g., cross-block orchestration, hybrid or routed MoE-PEFT frameworks) aim to address the limitations posed by strict locality or static routing (Peng et al., 2023, Liu et al., 12 Nov 2024).
• Theoretical Unification: Recent decomposition-based perspectives aim to provide a unified understanding of PEFT’s effectiveness and guide the design of improved low-rank, adapter, and soft prompt modules (Si et al., 7 Jul 2024).

6. Future Directions and Ongoing Developments

Anticipated research frontiers include:

• Automated Hyperparameter and Subset Selection: Integration of automatic, task-aware selection methods (e.g., BIPEFT’s budget-guided iterative search, Hessian-based influence ranking) may render PEFT tuning more robust and transferable (Chang et al., 4 Oct 2024, Xu et al., 18 May 2025).
• Generalization to Diverse Architectures: Expanding PEFT principles beyond transformer backbones—to MoE models (with dedicated routing adaptation (Liu et al., 12 Nov 2024)), state-space architectures (e.g., Mamba (Yoshimura et al., 6 Nov 2024)), and spectral graph networks (Liang et al., 10 Oct 2024)—is an active area of development.
• Scalability and System Integration: Addressing real-world bottlenecks, such as distributed/federated adaptation, communication constraints, and hardware-aware partitioning of adaptation modules, will become increasingly important as models and deployments scale (Liao et al., 2023, Han et al., 21 Mar 2024, Hao et al., 7 Jun 2024).
• Interpretability and Layer Allocation: Comprehensive studies on which layers are most critical for adaptation, the interpretability of adapter and low-rank modules, and the relationship between pre-trained network structure and adaptation success are ongoing (Zhang et al., 23 Jan 2025, Si et al., 7 Jul 2024).
• Unified Benchmarks and Evaluation: There is a need for standardized evaluation protocols and comparative benchmarks (analogous to Hugging Face PEFT, AdapterHub) for fair assessment of new methods’ trade-offs and effectiveness (Zhang et al., 23 Jan 2025, Prottasha et al., 19 Apr 2025).

7. Summary Table of Category Properties

Category	Mechanism	Typical Parameter %	Notable Methods
Selective	Subset of original weights	0.01–1%	BitFit, AdaPEFT
Additive	Small adapter modules	0.1–5%	Adapter, LoReFT
Reparameterization	Low-rank matrix updates	0.1–2%	LoRA, AdaLoRA
Prompt Tuning	Soft or hard prompt vectors	≪1%	Prefix, P-Tuning v2
Hybrid/Unified	Mixed/stacked strategies	variable	UniPELT, PERFT, X-PEFT

This taxonomy highlights how PEFT techniques allow practitioners to choose methods appropriate to their task requirements, resource constraints, and deployment scenario, thereby enabling scalable, generalizable, and efficient adaptation of large-scale models across natural language, vision, and multimodal domains.