---
title: Adapter-Based Continual Pre-Training
url: https://www.emergentmind.com/topics/adapter-based-continual-pre-training
type: topic
---

# Adapter-Based Continual Pre-Training

Adapter-based continual pre-training is a paradigm for extending the capabilities of large pre-trained neural networks to new tasks, languages, or domains while mitigating catastrophic forgetting and avoiding the inefficiencies of full model re-training. This approach instantiates small, often layer-wise, task-specific modules—adapters—into a frozen backbone, enabling modular, parameter-efficient transfer and specialization. Contemporary research demonstrates that such methods yield strong empirical performance across vision, language, speech, and multi-modal domains, with only a minor fraction of the backbone model’s parameters being updated per task.

## 1. Core Principles and Adapter Architectures

Adapter-based continual pre-training is predicated on the decoupling of feature extraction (handled by a fixed backbone, typically a Transformer) from continual adaptation (handled by lightweight adapters). The standard adapter is a two-layer feed-forward MLP "bottleneck" with down-projection, non-linearity, and up-projection, all sandwiched by a residual connection:
\[
\mathrm{Adapter}(h) = h + W_u\,\sigma(W_d\,h)
\]
where \( h \in \mathbb{R}^d \), \( W_d \in \mathbb{R}^{r \times d} \) (with \( r \ll d \)), \( W_u \in \mathbb{R}^{d \times r} \), and \( \sigma \) is a nonlinearity (e.g., ReLU, GELU) [2203.10484][2107.13530][2312.08192][2211.11363].

Variants include:
- **Hierarchical adapters**: Partition adapters into base (multi-task-capturing general skills) and task-specific (specializing on each new task), yielding hierarchical inductive transfer [2203.10484].
- **Linked or lateral adapters**: Fuse the outputs of all (past, present, and even future) task adapters via attention-weighted sums, computed by a trainable MLP over task embeddings [2412.10687].
- **Self-expanding adapters**: Dynamically allocate new adapters only when significant distribution shift is detected, employing a z-score outlier test on representation descriptors [2403.18886].
- **Label-specific adapters**: Each class in continual learning is equipped with a small prototype memory; feature aggregation occurs over all class adapters for inference [2505.23271].
- **Multi-modal adapters**: Adapters are placed not just in unimodal blocks but also in cross-modal fusion layers (e.g., vision-language transformers) [2410.10923].
- **Domain-specific heads/units**: Parallel domain-heads/hidden-units are added to self-attention and FFN, increasing capacity adaptively [2211.11363].

## 2. Training Methodologies and Continual Learning Protocols

The prototypical continual pre-training protocol involves freezing the pre-trained backbone and successively training or extending adapters as tasks arrive:
- **Sequential task introduction**: For each new task or domain, either a fresh adapter slice (e.g., new down/up projection) is introduced, or an existing set of adapters is extended [2407.10281][2107.13530][2211.11363]. Old adapters are frozen after being trained.
- **Two-stage training (ATLAS)**: First, an "experience-based" stage reuses knowledge via weighted sums of old adapters; second, a "novel knowledge expansion" stage introduces a new adapter if needed [2410.10923].
- **Replay/memory-free**: Most methods avoid storing raw data or gradient rehearsal; stability is enforced by architectural isolation and sometimes regularization (e.g., Fisher penalty in EWC, or orthogonality constraints between new and old adapters) [2412.10687][2407.10281][2211.11363].
- **Adapter consolidation or distillation**: Where memory is limited or redundancy is detected, adapter fusion or MSE-based distillation over unlabeled buffer pools consolidates multiple adapters [2206.14085].
- **Momentum accumulation**: Parallel online/offline adapters are maintained, with the offline adapter updated by exponential moving average (EMA) for cross-task stability [2303.10070].

A representative procedure is shown below for hierarchical adapters [2203.10484]:

1. Multi-task train the base adapter on all old tasks; freeze.
2. For each task, train its specific adapter on that task, keeping base (general) adapters frozen.
3. When a new task arrives, add a new task-specific adapter and train only it, keeping both base and all previous adapters fixed.

## 3. Objectives, Regularization, and Stability

The primary loss is typically cross-entropy (classification), retrieval, or contrastive/self-supervised objectives (for SSL or MAE). Adapter-based approaches introduce distinctive regularizers or losses:
- **Orthogonal loss**: Enforce new adapter directions orthogonal to all previous adapters to minimize interference [2407.10281].
- **EWC-style penalty**: Online computation of the Fisher information matrix constrains adapter-attention network weights to prevent forgetting [2412.10687].
- **Feature distillation**: For per-label adapters, old label embeddings are preserved by L2 regularization over class prototypes [2505.23271].
- **Adaptive expansion**: A pre-trained autoencoder per adapter (at each representation level) signals whether a new adapter is needed, driving sub-linear parameter growth by amortizing adaptation over the distributional complexity rather than total number of tasks [2403.18886].

In self-supervised or domain-adaptive settings (e.g., PAD for infrared images [2312.08192]) the adapter is trained via a masked reconstruction loss (e.g., MAE) with only adapters and the decoder updated, while all backbone parameters remain fixed.

## 4. Empirical Evaluation and Comparative Results

Multiple experimental protocols are established across vision, dialogue, language, speech, and multi-modal domains:
- **Parameter efficiency**: Adapter-based continual pre-training consistently reduces the trainable parameter count per task, often to <5% of the full model [2203.10484][2107.13530][2312.08192][2303.10070].
- **Catastrophic forgetting**: By freezing both the backbone and non-current adapters, prior task performance remains stable, outperforming full fine-tuning or regularization-only baselines [2107.13530][2211.11363][2412.10687].
- **Downstream transfer**: Tasks such as dialogue response retrieval, semantic segmentation, ASR, classification, and multi-modal inference benefit from continual adapter-based pre-training, often surpassing prompt-based PETs, naive replay, and even some rehearsal-based methods [2203.10484][2509.17816][2410.10923].
- **Quantitative gains**: For instance, AdaHIT attains average hits@1 of 0.8158 with only 4% per-task parameter cost (vs. 100% for full fine-tuning) [2203.10484]; SEMA achieves 86.98% accuracy on CIFAR-100 incrementally vs. ≈80% for prompt-based L2P [2403.18886]; LADA shows 2–3% gains over MoE-style adapter routing on X-TAIL [2505.23271].

| Method           | Backbones                | Continual Setting   | Notable Results                       |
|------------------|-------------------------|---------------------|---------------------------------------|
| AdaHIT           | Poly-Encoder (Dialog)   | Multi-task and incremental | 0.8158 avg hits@1 @ 4.2% ∆params [2203.10484] |
| cwav2vec 2.0     | wav2vec 2.0 (Speech)    | Language-incremental | <0.5% WER drop, 0% forgetting, 32% faster [2107.13530] |
| ADA              | ViT/DeiT (Vision)       | Adapter-pool/binary/multi-class | ≈95M params for all tasks @ pool K=4 [2206.14085] |
| SEMA             | ViT-B (Vision)          | Class-incremental, VTAB | Sub-linear growth, 6.5× lower than linear [2403.18886] |
| Linked Adapters  | ViT (Vision)            | Task-incremental    | +1–2% over standalone adapters [2412.10687] |
| C-ADA            | ViT (Vision)            | Class/Domain-incremental | +2.5–6% over CODA/L2P, 0% rehearsal [2407.10281] |
| PAD              | ViT (Vision, IR-SSL)    | Domain transfer     | +0.9 to +2.4 mIoU over full MAE ft [2312.08192] |
| LADA             | CLIP ViT                | Label-incremental   | +0.8–2.9% avg/last over strong PET [2505.23271] |
| GLARE            | ViT (Semantic Seg)      | Domain-SSL          | +0.4–0.6 mIoU, ≈3% param overhead [2509.17816] |
| AF-Adapter       | RoBERTa (Chinese, Biomed) | Domain-continual   | +2% downstream vs. PCL-MedBERT, –11% forgetting [2211.11363] |
| ATLAS            | ViLT (Vision-Language)  | Multi-modal, upstream| +0.7–1.4% avg, <1% forgetting [2410.10923] |

## 5. Extensions: Modularity, Knowledge Sharing, and Sub-Linear Growth

Advanced architectures address several challenges beyond naive per-task adapter assignment:
- **Knowledge sharing and lateral transfer**: Attention-based mechanisms (e.g., Linked Adapters [2412.10687], ATLAS [2410.10923]) achieve both forward and backward knowledge transfer by learning per-layer, per-task attention weights via MLPs or cross-attention. During inference, the system leverages not just own-task but all adapters, mitigating under-sharing and redundancy.
- **Self-regulating growth**: Expansion is triggered only by genuine shifts in representation space (quantified by descriptor AEs), yielding parameter growth proportional to intrinsic task diversity rather than total task count [2403.18886].
- **Distillation/consolidation**: Adapter fusion and pool-size management (e.g., ADA with K=4) maintain a memory-efficient pool, replacing or distilling only as needed [2206.14085].
- **Per-label scaling**: In label-incremental CLIP settings, dedicated, lightweight memory per label sidesteps router errors endemic to MoE or prompt-pool architectures [2505.23271].
- **Patchwise scaling**: Dynamic patchwise scaling in vision-specific adapters further improves domain adaptation (e.g., in IR) by allowing fine spatial control over adapter influence [2312.08192].

## 6. Domains of Application and Current Limitations

Adapter-based continual pre-training has been scaled across the following axes:
- **Vision**: Class/domain/task-incremental learning with ViT/DeiT, semantic segmentation with adapter-only pre-training (UniAdapter in GLARE [2509.17816]), and IR-specific SSL (PAD [2312.08192]).
- **Language**: Domain-incremental masked language modeling (e.g., biomedical adaptation in AF-Adapter [2211.11363]), cross-lingual SSL in speech (wav2vec 2.0 + Language Adapters [2107.13530]).
- **Multi-modal**: Unified handling of image, text, and combinations thereof, leveraging two-stage learning (ATLAS [2410.10923]).
- **Rehearsal- and memory-free CL**: All architectures by design avoid or minimize the need for replay buffers or task data storage.

Current limitations include:
- **Parameter growth**: Most per-task or per-label adapter instantiations scale at least linearly with the number of tasks or labels; progressive self-expansion and adapter compression mechanisms aim to mitigate this overhead [2403.18886][2412.10687].
- **Domain shift/extrapolation**: The freezing of the entire backbone strongly constrains representational adaptation; in cases of extreme domain shift, existing frozen features may not suffice [2312.08192].
- **Task identity**: Most knowledge fusion or attention-based reuse schemes assume task labels are known at inference; class-incremental or task-free regimes require further methodological advances [2412.10687].
- **Adapter placement**: Current conventions often inject adapters only into MLP blocks or upper transformer layers; the optimal per-layer allocation remains under-explored.

## 7. Outlook and Future Directions

Recent advances suggest several key research trajectories:
- **Improved knowledge transfer**: Learning sophisticated, dynamic fusion of all adapters (forward/backward), possibly leveraging meta-learning or generative router networks for adaptive weighting [2412.10687][2410.10923].
- **Adapter compression/merging**: Approaches that consolidate redundant adapters post hoc, reducing parameter and compute footprint for long-sequence continual learning [2206.14085].
- **Cross-modal continual adaptation**: Extension of modular adapters to ever more complex combinations (vision, audio, language), with multi-modal fusion modules trained in continual settings [2410.10923].
- **Task-agnostic and streaming settings**: Removal of reliance on explicit task labels at inference, demanding more distributed, context-aware routing [2412.10687].
- **Integration with memory/rehearsal**: Hybrid architectures combining adapter isolation with external memory, replay, or generative rehearsal for theoretical guarantees of no forgetting.
- **AutoML/NAS for adapter placement and sizing**: Search for per-layer, per-task optimal adapter capacity, balancing stability and plasticity on-the-fly [2211.11363].

Adapter-based continual pre-training enables highly parameter-efficient, robust extension of large pre-trained models. Through methodological advances in modular isolation, hierarchical transfer, adaptive attention, and self-regulating growth, these approaches have enabled continual learning systems to match or surpass traditional fine-tuning methods with orders-of-magnitude smaller per-task overhead, broad applicability, and greatly increased resistance to catastrophic forgetting [2203.10484][2412.10687][2403.18886][2505.23271][2410.10923].

Source: https://www.emergentmind.com/topics/adapter-based-continual-pre-training