---
title: Parameter-Efficient Finetuning Techniques
url: https://www.emergentmind.com/topics/parameter-efficient-finetuning-techniques
type: topic
---

# Parameter-Efficient Finetuning Techniques

Parameter-efficient finetuning (PEFT) comprises a diverse family of methods designed to adapt large pre-trained foundation models to downstream tasks while updating only a small fraction (typically 0.01–1%) of parameters. By freezing the majority of model weights and introducing minimal adaptation modules or modifications, PEFT achieves nearly full fine-tuning performance, while substantially reducing computational, storage, and optimization overheads, enabling scalable adaptation across tasks and domains [2407.05417].

## 1. Motivation and Problem Context

With the rapid proliferation of foundation models containing billions of parameters, full fine-tuning for each downstream task is increasingly impractical due to computational demands and prohibitive storage associated with maintaining separate model copies [2407.05417]. PEFT directly targets these bottlenecks by freezing the pretrained weights $W\in\mathbb{R}^{n\times m}$ and introducing a small delta module $\phi$—often with only 0.01–1% of the total parameters—to approximate the performance $P(W^*)$ of full fine-tuning $W^*$, while minimizing both parameter count and training/inference time.

Design constraints for PEFT methods include minimizing trainable parameters, preserving throughput (no significant increase in forward/backward cost), and achieving downstream performance (e.g., accuracy, F1) close to that of fully tuned models [2407.05417].

## 2. Unifying Mathematical Framework: Subspace Decomposition

PEFT methods can be unified under a decomposition-based framework, where the pretrained weight matrix is viewed as spanning a subspace $S(W)$. Adaptation is then realized either by reconstructing/rescaling the original subspace (subspace reconstruction) or augmenting it with additional directions (subspace extension):

- **Subspace Reconstruction:** Apply a transformation $\phi(W) = g(f(W))$ where $f$ is a (possibly structured) reconstruction/scaling (e.g., singular-value modification, diagonal scaling), and $g$ may introduce new directions.
    - **Mode 1:** Only singular values $\Sigma$ are updated, keeping $U,V$ fixed (e.g., SAM-PARSER).
    - **Mode 2:** Scale left ($D_1$) and/or right ($D_2$) singular vector spaces: $\phi(W) = D_1 W D_2$ (diagonal).
      - Examples: (IA)$^3$ ($\phi(W)=W D_2$), SSL ($\phi(W)=D_1 W$), SSB ($\phi(W)=D_1 W D_2$).
    - **Mode 3:** Perturb $U$ or $V$ (e.g., BitFit, prefix or prompt tuning).
- **Subspace Extension:** Add a low-rank update $\Delta W$ to expand $S(W)$:
    - **LoRA family:** $\Delta W=AB$ ($A\in\mathbb{R}^{n\times r}, B\in\mathbb{R}^{r\times m}$, $r\ll \min(n,m)$), and extensions such as AdaLoRA ($\Delta W=ADB$ with diagonal $D$), FLoRA/TriLoRA ($\Delta W=AGB$ with unrestricted $G$).
    - **Adapters:** Insert a small MLP in parallel, yielding $\phi(W): xW \mapsto xW + h(xA)B$.

The unified training objective is to produce an adapted weight $\phi(W)$ whose span projects maximally onto the unknown optimal $W^*$:
$$
\min_\phi \| W^* - \phi(W) \|_F
$$
[2407.05417].

## 3. Key Parameter-Efficient Finetuning Methods

PEFT strategies can be grouped as follows:

| Method                      | Adaptation Mechanism           | Typical Parameter Overhead (%) | Workspace              |
|-----------------------------|-------------------------------|-------------------------------|------------------------|
| BitFit                      | Bias-only tuning              | 0.01–0.1                      | Biases (all layers)    |
| (IA)$^3$, SSL, SSB          | Diagonal scaling (row/col)    | 0.01–0.1                      | Weights                |
| LoRA, AdaLoRA, FLoRA        | Low-rank update ($AB$, $ADB$) | 0.1–1.0                       | Attention, FFN         |
| Adapter (Houlsby, Pfeiffer) | Bottleneck MLP insertion      | 0.1–4.0                       | After attention/FFN    |
| Prefix/Prompt tuning        | Learnable embeddings          | 0.05–0.5                      | Attention keys/values  |
| SVD-based methods           | Modifying singular values     | 0.01–0.5                      | Weights                |
[2407.05417]

**Novel classes:** SSL (scale-left subspace: row scaling only, $n$ scalars per layer), SSB (scale-both subspace: input and output scaling, $n+m$ scalars), both offer extreme sparseness and reach up to 85–99% of full fine-tuning performance while tuning under 0.1% of parameters [2407.05417].

## 4. Comparative Theoretical Analysis

Empirical and theoretical insights indicate that not just low-rankness, but the form of decomposition and implicit pattern constraints, critically influence the learning capability of PEFT schemes:

- **Expressivity Hierarchy:** Adapters using $\Delta W = AGB$ (unconstrained $G$) outperform diagonal-constrained forms (AdaLoRA: $G$ diagonal, LoRA: $G=I$). Fewer coupling/orthogonality constraints allow $A$ and $B$ to adapt more freely to $W^*$, improving both optimization and convergence [2407.05417].
- **Matrix Pattern Constraints (MPC):** Adding explicit regularization (e.g., enforce (semi-)orthogonality or diagonal structure) can improve both performance and stability for low-rank PEFT methods, acting as a plug-and-play regularizer without extra parameters:
    - MPC$_o$: $R_o(A,B) = \|A^TA - I_r\|_F^2 + \|BB^T - I_r\|_F^2$
    - MPC$_d$: $R_d(A,B) = \|A^TA - I_r\|_F^2 + \|BB^T - \text{diag}(BB^T)\|_F^2$
    - Adding these as regularizers yields up to 1–2 average GLUE points of improvement for LoRA-family methods [2407.05417].

## 5. Empirical Results and Practical Trade-offs

Extensive experiments were conducted on foundation models such as RoBERTa-base/large and DeBERTaV3-base on the GLUE benchmark (nine tasks). Key findings:

- **SSL** achieves 82–85% of full fine-tuning with just 0.02% of model parameters.
- **SSB** bridges the performance gap to full fine-tuning (≈99% performance) at <0.1% parameter overhead.
- **FLoRA** and unconstrained low-rank architectures outperform AdaLoRA and LoRA, confirming the importance of avoiding unnecessary matrix constraints.
- **MPC** regularization delivers consistent 0.3–1.0 point improvements (GLUE average) and reduces variance for the LoRA family, without introducing new trainable parameters.

These empirical results directly inform several practical implications:
- Subspace scaling approaches (SSL/SSB) are optimal in memory-constrained environments.
- Matrix decomposition forms with relaxed constraints (e.g., FLoRA) facilitate faster convergence and higher final accuracy.
- MPC regularization modules are orthogonal enhancements, compatible with all low-rank PEFT methods [2407.05417].

## 6. Design Space and Strategy Patterns

Recent work has highlighted that the design space of PEFT encompasses four key axes: layer grouping, parameter allocation, tunable group selection, and strategy assignment. Greedy exploration reveals the following effective patterns in practice:

- **Spindle grouping:** Finer granularity at input/output layers, wider groups in the middle.
- **Uniform allocation:** Trainable parameter budgets uniformly distributed across groups.
- **Tune all groups:** Consistent tuning across all layer groups yields superior performance.
- **Strategy assignment:** Assigning adapters, prefix, BitFit, or LoRA in a group-wise manner—rather than uniformly—enables performance gains surpassing full fine-tuning in some regimes [2301.01821].

PEFT strategies are thus not monolithic; optimal configurations depend on the foundation model’s structure, the downstream task, and resource constraints.

## 7. Future Directions and Open Challenges

Future research directions include:
- **Unified Decomposition Theory:** Further theoretical examination of the correspondence between different PEFT forms under the lens of matrix decomposition, exploring the optimal subspace expansions for various architectures and tasks.
- **Automated Pattern Selection:** Learning to adapt not only the parameter delta but also decomposition strategy, matrix pattern constraints, and module positioning (potentially through meta-learning or differentiable search).
- **Ultra-lightweight PEFT:** Continued exploration of methods that drive parameter overheads toward negligible fractions—without loss in accuracy—by leveraging advanced subspace decompositions and task-aligned regularization.
- **Generalization Across Domains:** Characterization of transferability and adaptation gaps for PEFT modules across tasks or model families, particularly how decomposition-based methods interact with architectures and pretraining biases.

The decomposition-centric view of PEFT introduced in [2407.05417] provides not only a unified classification and analysis scheme but also inspires novel approaches—such as SSL and SSB—that push the frontier of parameter efficiency and adaptation capacity in large-scale foundation models.

Source: https://www.emergentmind.com/topics/parameter-efficient-finetuning-techniques