---
title: Adapter-Based Finetuning
url: https://www.emergentmind.com/topics/adapter-based-finetuning
type: topic
---

# Adapter-Based Finetuning

Adapter-based finetuning is a parameter-efficient transfer learning paradigm that injects lightweight, trainable modules—adapters—into frozen, large-scale pre-trained models. Instead of updating all model parameters for each downstream task, only the adapter modules are trained, yielding substantial memory and computation savings while retaining, and often improving, task performance and generalization. Adapter-based finetuning has been adopted and rigorously evaluated across natural language processing, speech, computer vision, and multimodal domains, with evolving architectural variants and empirical insights.

## 1. Concept and Motivation

Adapter modules implement a compact two-layer bottleneck: an input is down-projected to a low-dimensional subspace, processed with a nonlinearity, then re-projected to the original dimension and added back via a residual connection. For hidden state $h\in \mathbb{R}^d$, a typical adapter applies:
$$
h' = h + W_\text{up}\, \sigma(W_\text{down} h)
$$
where $W_\text{down} \in\mathbb{R}^{r\times d}$, $W_\text{up} \in\mathbb{R}^{d\times r}$, $r \ll d$, and $\sigma$ is a nonlinearity (e.g., ReLU, tanh). Only the adapter weights are updated during finetuning; all backbone parameters (e.g., Transformer blocks) remain frozen.

This design offers several advantages:
- Dramatic reduction in trainable parameters and optimizer state; typical added parameter budget is 0.3–10% of the full model [2509.03057, 2405.05493, 2407.21066].
- Preservation of original representations and in-context learning capacity [2112.05253, 2106.03164].
- Modularity—separate adapters per task, language, or domain in a single backbone [2106.01463, 2401.08992].
- Mitigation of catastrophic forgetting and increased robustness to hyperparameter choices [2106.03164].

## 2. Core Architectures and Variants

The classical adapter, as popularized by Houlsby et al., is a two-layer serial bottleneck inserted after self-attention and feed-forward sublayers in each Transformer block [2305.07491]. Variants and extensions include:

- **Parallel adapters:** Placed in parallel to the sublayer and summed with the main output for reduced inference latency [2305.07491].
- **LoRA:** Low-rank adaptation, injecting updatable low-rank matrices directly into existing weight matrices [2501.08271, 2305.07491].
- **Prefix/prompt tuning:** Trainable tokens prepended to the input sequence or key/value matrices [2501.08271].
- **Compacter, AdapterFusion, IA³:** Variants using Kronecker factorization, multi-adapter fusion, or learned per-dimension scaling [2501.08271].
- **Dynamic and structure-learnable adapters:** Activation and placement are learned via differentiable gating and sparsity controls, enabling task-specific network substructures [2509.03057].

In computer vision, vision-specific adapters integrate convolutions or multi-scale filters (e.g., Mona) [2311.15010] and block-specific designs such as dynamic routing and prompt generators (e.g., Adapter-X) [2406.03051] have demonstrated significant gains.

## 3. Insertion Policies and Freezing Strategies

Adapters are typically inserted:
- After attention and feed-forward sublayers in Transformer-based architectures [2106.03164, 2406.03051].
- Only after feed-forward layers or in the encoder/decoder selectively for efficiency [2211.00585, 2106.01463].
- In parallel or sequential arrangements, sometimes as stacked modules [2501.08271, 2405.05493].

All original backbone weights (attention, MLPs, embeddings, positional encodings) are frozen. Only the adapter weights (and sometimes task-specific head layers) are trainable. This strict freezing is central to memory and compute efficiency and prevents catastrophic shifts in generic representations [2106.03164, 2112.05253]. Selective adapter freezing (SAFE) further improves memory/computation efficiency by dynamically freezing unimportant adapters during training using activation similarity metrics (CKA) [2412.03587].

## 4. Training Objectives, Optimization, and Resource Profiles

Adapter-based finetuning adopts the standard task loss (cross-entropy, CTC, MSE, contrastive, etc.) as in full fine-tuning [2210.14161, 2211.00585, 2408.05749]. The optimizer (typically Adam or AdamW) and schedules usually mirror full fine-tuning but employ higher learning rates for adapter parameters—often 5–10× main-model fine-tuning rates due to the reduced parameter count [2106.01463].

Resource footprint is consistently lower:
- Adapter parameter budget: typically 0.3–10% of the model; as low as 0.2% for Adapter-X (CV) or LoRA (NLP), up to 27% for multi-task speech models [2406.03051, 2406.14747, 2401.08992].
- FLOPs and memory: significant reductions in optimizer state, with activation memory saved further by selective freezing (SAFE) [2412.03587].
- Faster convergence, stronger regularization, and improved generalization in low-resource or multi-task scenarios [2501.08271, 2106.03164, 2406.14747].

Care must be taken with batch size and learning rate to maintain efficiency, especially in high-throughput or streaming applications [2401.08992, 2211.00585].

## 5. Empirical Performance Across Domains

**Language:** Adapter-tuned models routinely match or slightly underperform full fine-tuning on large-scale NLU benchmarks (GLUE, SuperGLUE) within 0.5–2.0 points, but show marked superiority in low-resource [2106.03164, 2405.05493], cross-lingual [2106.01463], and multi-task settings [2509.03057, 2412.03587].

**Speech:** Adapter-based methods in ASR and speech processing deliver WER reductions (12.2% average in challenging multilingual dictation for 0.4% per-language parameters [2401.08992]), outperform or match full fine-tuning across ASR, speaker/intent/emotion tasks, and enable rapid, modular adaptation [2211.00585, 2407.21066, 2406.14747].

**Vision and Vision-Language:** In visual tasks, advanced adapter designs such as Mona and Adapter-X match or exceed full fine-tuning in image classification, detection, and segmentation—sometimes at less than 2% of trainable parameters [2311.15010, 2406.03051]. For VLMs and segmentation, VLSM-Adapter and R-Adapter enable robust, OOD-resistant finetuning with strong gains in both data-rich and few-shot/zero-shot settings [2405.06196, 2408.05749].

**Multimodal/Few-shot/Hierarchical:** Adapter-based finetuning paired with attribute prompts and hierarchical regularization achieves state-of-the-art on few-shot VLM transfer and robust multimodal alignment [2508.11176]. Gate-controlled, structure-learning adapters yield superior accuracy and task-dependent efficiency [2509.03057].

Summary of typical quantitative results (accuracy/f1/BLEU, parameter fraction):

| Model/task                | Adapter perf.   | Full-tune perf. | Adapter param % | Source         |
|-------------------------- | --------------- | --------------- | --------------- |---------------|
| RoBERTa-base, GLUE (avg)  | 85.6            | 86.4            | 8.9             | [2405.05493]  |
| ELECTRA, SuperGLUE        | 0.782           | 0.750           | 2–5             | [2501.08271]  |
| WavLM ASR, WER (%)        | 9.39            | 9.41            | 10              | [2407.21066]  |
| Mona, COCO instance seg.  | AP=53.4         | AP=52.4         | 4.7             | [2311.15010]  |
| Adapter-X, VTAB           | 76.2            | 68.9            | 0.2             | [2406.03051]  |
| CLIP, ImageNet OOD acc.   | 54.3            | 44.2            | 13              | [2408.05749]  |

## 6. Limitations and Trade-offs

Despite strong parameter efficiency, adapters can incur higher training compute and slightly increased inference latency versus full fine-tuning for moderate-size models (up to several hundred million parameters), mainly due to non-trivial backward passes through each adapter [2305.07491]. In these regimes, multi-task full fine-tuning may match or surpass adapters in total resource cost and maintainability. For extremely large models (LLMs, ViTs), adapter-based approaches remain the only tractable solution for scalable, modular, and continually adaptive finetuning.

Certain tasks and domains—especially extremely small data settings or those demanding architectural reconfiguration—may require refined adapter placement, hybrid PEFT, or adapters with dynamic insertion and activation [2509.03057, 2406.03051].

## 7. Practical Implementation and Emerging Trends

Implementation is supported by libraries such as AdapterHub, HuggingFace Transformers, and task-specific frameworks. Best practices include:
- Serial insertion after every attention/FFN block; $r$ (bottleneck) set to $d/4$–$d/16$ [2106.03164, 2406.03051].
- Stacking/fusion of adapters for related multi-task sets [2406.14747].
- Freezing schedule and module selection automated via structural gating or importance scores [2509.03057, 2412.03587].
- For robust OOD and multi-positive alignment, self-ensemble adapters with dropout and EMA are effective [2408.05749].
- Domain-specific modifications (e.g., depthwise/spatial adapters in vision, hierarchical/prompt adapters in VLMs) yield further gains [2311.15010, 2508.11176].

Recent architectural and analytical advances include frequency-aware adapters (FAA) with dynamic channel modulation [2512.22378], hyperbolic attribute bridging for one-to-many VLM mapping [2508.11176], and unified adapters for multi-task and continual learning [2407.21066, 2412.03587].

## References

- "MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuning" [2112.05253]
- "Adapter-Based Extension of Multi-Speaker Text-to-Speech Model for New Speakers" [2211.00585]
- "Structure-Learnable Adapter Fine-Tuning for Parameter-Efficient Large Language Models" [2509.03057]
- "ELP-Adapters: Parameter Efficient Adapter Tuning for Various Speech Processing Tasks" [2407.21066]
- "Adapter is All You Need for Tuning Visual Tasks" [2311.15010]
- "Fine-Grained VLM Fine-tuning via Latent Hierarchical Adapter Learning" [2508.11176]
- "VLSM-Adapter: Finetuning Vision-Language Segmentation Efficiently with Lightweight Blocks" [2405.06196]
- "Lightweight Adapter Tuning for Multilingual Speech Translation" [2106.01463]
- "Parameter-Efficient Fine-Tuning With Adapters" [2405.05493]
- "Adapter-X: A Novel General Parameter-Efficient Fine-Tuning Framework for Vision" [2406.03051]
- "AAT: Adapting Audio Transformer for Various Acoustics Recognition Tasks" [2401.10544]
- "A Comprehensive Analysis of Adapter Efficiency" [2305.07491]
- "Towards Efficient Post-Training via Fourier-Driven Adapter Architectures" [2512.22378]
- "Not All Adapters Matter: Selective Adapter Freezing for Memory-Efficient Fine-Tuning of Language Models" [2412.03587]
- "An Adapter-Based Unified Model for Multiple Spoken Language Processing Tasks" [2406.14747]
- "Efficient Adapter Finetuning for Tail Languages in Streaming Multilingual ASR" [2401.08992]
- "Comparative Analysis of Efficient Adapter-Based Fine-Tuning of State-of-the-Art Transformer Models" [2501.08271]
- "On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation" [2106.03164]
- "Efficient and Versatile Robust Fine-Tuning of Zero-shot Models" [2408.05749]

Source: https://www.emergentmind.com/topics/adapter-based-finetuning