Parameter-Efficient Continual Learning (PECL)

• Parameter-Efficient Continual Learning (PECL) is a framework that adapts large, pre-trained models to new tasks using minimal parameter updates, thereby addressing catastrophic forgetting.
• It leverages methods like LoRA, adapters, and prompts to introduce low-dimensional updates, balancing stability on previous tasks with plasticity for new challenges.
• Evaluations in PECL combine performance metrics with resource usage measures, highlighting improvements in accuracy, reduced forgetting, and efficient compute/storage trade-offs.

Parameter-Efficient Continual Learning (PECL), also termed Parameter-Efficient Continual Fine-Tuning (PECFT), studies how a large pre-trained model can be adapted over a sequence of tasks while updating and storing only a small fraction of its parameters, rather than repeatedly fine-tuning the full network. In the survey formulation, a pre-trained model $f_{\Theta_0}$ is continually adapted by task-specific updates $\Delta\Theta^t$ through objectives of the form

$$\min_{\Delta\Theta^t}\; \mathcal L_M\bigl(D^t,M^t;\Theta_0+\sum_{k=1}^t\Delta\Theta^k\bigr) \quad \text{s.t.} \quad \|\Delta\Theta^t\|_0 \ll \|\Theta_0\|_0,$$

with variants that replace replay $M^t$ by regularization, routing, or subspace constraints (Coleman et al., 18 Apr 2025). PECL emerged from the conjunction of continual learning and parameter-efficient fine-tuning, and now spans class-incremental vision, language-model adaptation, multimodal instruction learning, semantic segmentation, and automatic speech recognition.

1. Formal problem setting and evaluation

Continual learning assumes a stream of tasks $D^1,\dots,D^T$ arriving sequentially, with the central difficulty being catastrophic forgetting: adaptation to task $t$ degrades performance on earlier tasks. Parameter-efficient fine-tuning, by contrast, freezes most of a pre-trained model and introduces only a low-dimensional update or a small module. PECL combines these constraints: it seeks stability on old tasks, plasticity on new tasks, and small per-task training, storage, and inference overheads (Coleman et al., 18 Apr 2025).

Evaluation is modality-dependent. General continual-learning studies commonly report average accuracy $A_N$, average forgetting $F_N$, forward transfer, and backward transfer (Coleman et al., 18 Apr 2025). LLM-oriented PECL work uses Whole Accuracy and Average Accuracy in continual knowledge extraction, or AP, F.Ra, FWT, and BWT in instruction-following benchmarks (2309.14763). Class-incremental semantic segmentation work evaluates mean IoU and Forget Score, defined as $$FS=\text{mIoU}_{\mathrm{joint}}-\text{mIoU}_{\mathrm{CL}}$$ (Muralidhara et al., 26 Jul 2025). ASR work measures word error rate and BWT, where negative BWT indicates forgetting (Eeckt et al., 8 Jun 2026).

A distinctive feature of PECL is that task performance is often evaluated jointly with resource usage. The PECFT survey explicitly separates continual-learning metrics from parameter-efficiency metrics such as trainable parameters per task, model-size efficiency, and storage overhead (Coleman et al., 18 Apr 2025). CLoRA goes further by using NetScore $\Omega_N$, combining final mIoU, average trainable parameters, and training MACs, thereby making compute and memory part of the evaluation rather than a secondary implementation detail (Muralidhara et al., 26 Jul 2025).

2. Architectural building blocks and update regimes

The basic PECL design space mirrors PEFT. The survey divides PEFT mechanisms into additive methods such as adapters and prompts, reparameterization methods such as LoRA, and selective methods such as masking or BitFit (Coleman et al., 18 Apr 2025). In LoRA, a frozen weight $\Delta\Theta^t$0 receives a low-rank update

$\Delta\Theta^t$1

so the adapted forward map becomes $\Delta\Theta^t$2, with parameter overhead $\Delta\Theta^t$3 per layer (Muralidhara et al., 26 Jul 2025). This low-rank form underlies a large fraction of PECL work because it decouples the frozen backbone from a small trainable subspace.

The continual-learning side of the design space remains equally varied. The survey retains the classical distinctions among regularization-based, replay-based, architecture-based, and optimization-based continual learning, but shows that each can be instantiated with parameter-efficient modules rather than full-model updates (Coleman et al., 18 Apr 2025). ConPET is replay-based but task-number-independent in compute; LAE is memory-free and relies on online/offline PET dynamics; SAPT uses multiple PET blocks with learned selection; EWC-LoRA revisits regularization in low-rank space; and PIECE abandons inserted modules entirely by updating only the top $\Delta\Theta^t$4 of existing parameters in place (2309.14763).

A common misconception is that PECL is equivalent to prompt-based continual learning. LAE explicitly argues that prompting is only one instantiation of PET, and demonstrates a unified continual-learning framework with Adapter, LoRA, and Prefix modules under the same Learning–Accumulation–Ensemble recipe (Gao et al., 2023). This suggests that PECL is better understood as a family of update parameterizations and continual-learning controls, not as a single architectural pattern.

3. Shared-state and fixed-footprint methods

One line of work keeps the number of adapted parameters effectively constant across tasks. CLoRA is a direct example in class-incremental semantic segmentation: every multi-head attention query and value projection in a frozen ViT encoder is replaced by a LoRA decomposition, and the same adapter $\Delta\Theta^t$5 is reused for all tasks. Task $\Delta\Theta^t$6 is trained with standard cross-entropy, while later tasks optimize $\Delta\Theta^t$7, where the distillation term follows MiB’s background-aware strategy to avoid background shift. Because there is only one set of LoRA weights, inference requires no task-ID and no adapter merging; the update is simply folded once into $\Delta\Theta^t$8. With rank $\Delta\Theta^t$9, the adapted projections correspond to about $$\min_{\Delta\Theta^t}\; \mathcal L_M\bigl(D^t,M^t;\Theta_0+\sum_{k=1}^t\Delta\Theta^k\bigr) \quad \text{s.t.} \quad \|\Delta\Theta^t\|_0 \ll \|\Theta_0\|_0,$$0 of the full model’s parameters, decoder included. On PASCAL VOC, CLoRA improves over MiB on the 15–1 split from $$\min_{\Delta\Theta^t}\; \mathcal L_M\bigl(D^t,M^t;\Theta_0+\sum_{k=1}^t\Delta\Theta^k\bigr) \quad \text{s.t.} \quad \|\Delta\Theta^t\|_0 \ll \|\Theta_0\|_0,$$1 mIoU and $$\min_{\Delta\Theta^t}\; \mathcal L_M\bigl(D^t,M^t;\Theta_0+\sum_{k=1}^t\Delta\Theta^k\bigr) \quad \text{s.t.} \quad \|\Delta\Theta^t\|_0 \ll \|\Theta_0\|_0,$$2 FS to $$\min_{\Delta\Theta^t}\; \mathcal L_M\bigl(D^t,M^t;\Theta_0+\sum_{k=1}^t\Delta\Theta^k\bigr) \quad \text{s.t.} \quad \|\Delta\Theta^t\|_0 \ll \|\Theta_0\|_0,$$3 mIoU and $$\min_{\Delta\Theta^t}\; \mathcal L_M\bigl(D^t,M^t;\Theta_0+\sum_{k=1}^t\Delta\Theta^k\bigr) \quad \text{s.t.} \quad \|\Delta\Theta^t\|_0 \ll \|\Theta_0\|_0,$$4 FS, on 5–3 from $$\min_{\Delta\Theta^t}\; \mathcal L_M\bigl(D^t,M^t;\Theta_0+\sum_{k=1}^t\Delta\Theta^k\bigr) \quad \text{s.t.} \quad \|\Delta\Theta^t\|_0 \ll \|\Theta_0\|_0,$$5 and $$\min_{\Delta\Theta^t}\; \mathcal L_M\bigl(D^t,M^t;\Theta_0+\sum_{k=1}^t\Delta\Theta^k\bigr) \quad \text{s.t.} \quad \|\Delta\Theta^t\|_0 \ll \|\Theta_0\|_0,$$6 to $$\min_{\Delta\Theta^t}\; \mathcal L_M\bigl(D^t,M^t;\Theta_0+\sum_{k=1}^t\Delta\Theta^k\bigr) \quad \text{s.t.} \quad \|\Delta\Theta^t\|_0 \ll \|\Theta_0\|_0,$$7 and $$\min_{\Delta\Theta^t}\; \mathcal L_M\bigl(D^t,M^t;\Theta_0+\sum_{k=1}^t\Delta\Theta^k\bigr) \quad \text{s.t.} \quad \|\Delta\Theta^t\|_0 \ll \|\Theta_0\|_0,$$8, and on 10–1 from $$\min_{\Delta\Theta^t}\; \mathcal L_M\bigl(D^t,M^t;\Theta_0+\sum_{k=1}^t\Delta\Theta^k\bigr) \quad \text{s.t.} \quad \|\Delta\Theta^t\|_0 \ll \|\Theta_0\|_0,$$9 and $M^t$0 to $M^t$1 and $M^t$2. In the PASCAL-VOC 15–1 scenario, it also yields $M^t$3 dB over MiB through a $M^t$4 reduction in trainable parameters at almost identical mIoU (Muralidhara et al., 26 Jul 2025).

LAE adopts a different fixed-size strategy. It keeps a frozen pre-trained backbone and maintains two PET modules of the same type: an online module for plasticity and an offline module for stability, with the offline copy updated by EMA,

$M^t$5

At inference, predictions from the online and offline experts are combined by normalized max-fusion. LAE stores no raw past data and no separate PET copy per task, and it reports $M^t$6 on CIFAR-100 with Adapter5 and $M^t$7 on ImageNet-R, surpassing DualPrompt by $M^t$8 and $M^t$9, respectively (Gao et al., 2023).

SAFE also preserves a fixed parameter budget, but splits adaptation into a slow PET branch learned only in the first session and a fast PET branch updated in later sessions. The first session uses a transfer loss built from feature cross-correlation with the pre-trained model, while later sessions use cross-classification and cosine feature-alignment losses to limit semantic drift. During inference, an entropy-based aggregation strategy weights the slow and fast learners. SAFE reports $D^1,\dots,D^T$0 on ImageNet-A versus $D^1,\dots,D^T$1 for RanPAC, $D^1,\dots,D^T$2 average over six datasets versus $D^1,\dots,D^T$3, and memory overhead of about $D^1,\dots,D^T$4–$D^1,\dots,D^T$5 of the backbone, constant across sessions (Zhao et al., 2024).

Regularization-based constant-footprint PECL reappears in EWC-LoRA. Instead of allocating a new adapter per task, it trains a shared LoRA branch and penalizes changes in the full $D^1,\dots,D^T$6-space through a diagonal Fisher estimate, using

$D^1,\dots,D^T$7

After each task, the LoRA update is merged into the base model and the temporary branch is discarded. On CIFAR-100, DomainNet, ImageNet-R, and ImageNet-A, final-task accuracy rises from $D^1,\dots,D^T$8 to $D^1,\dots,D^T$9, from $t$0 to $t$1, from $t$2 to $t$3, and from $t$4 to $t$5, respectively, relative to Vanilla LoRA (Zheng et al., 19 Feb 2026).

PIECE removes PET modules altogether. It computes importance scores from either the diagonal Fisher (PIECE-F) or a second-order normalized score combining gradient and curvature (PIECE-S), selects the top $t$6 of parameters, and updates only that in-place subset. The method introduces no new parameters, uses no prior data, and is evaluated on language and multimodal foundation models. On Llama3-8B, PIECE-F attains OP $t$7 and BWT $t$8, while PIECE-S attains OP $t$9 and BWT $A_N$0; on LLaVA-1.5-7B, PIECE-S reaches Flickr30K $A_N$1, OP $A_N$2, and BWT $A_N$3 (Wang et al., 19 Nov 2025). The contrast between PIECE-F and PIECE-S makes explicit a recurrent PECL trade-off: higher transfer can come with weaker retention, and stronger retention can reduce adaptivity.

4. Task-modular, routing, and hierarchical methods

A second major regime allocates task-specific PET modules but controls their training or activation to keep continual complexity manageable. ConPET formalizes this in LLM continual learning. Static ConPET adapts replay-based methods such as EMR, EA-EMR, and CRL by replacing full fine-tuning with PET and a dynamic replay strategy that samples from all previous tasks under a fixed batch-count budget, so per-task cost remains $A_N$4. Dynamic ConPET instead assigns one PET module per task plus a PET-based selector, activating only top-$A_N$5 modules per example; with fixed $A_N$6, $A_N$7, $A_N$8, and $A_N$9, its per-task cost is $F_N$0. ConPET reports that PET reduces tunable parameters from $F_N$1 to $F_N$2; Static ConPET reduces the scale of tunable parameters by over $F_N$3 times and improves by at least $F_N$4 absolute points in A and W on all six knowledge-extraction benchmarks, while Dynamic ConPET reaches $F_N$5 on FewNERD versus $F_N$6 for EMR* (2309.14763).

SAPT also uses multiple PET blocks, but aligns learning and selection through shared attention. For task $F_N$7, an input-derived query $F_N$8 attends over keys $F_N$9, producing attention weights $$FS=\text{mIoU}_{\mathrm{joint}}-\text{mIoU}_{\mathrm{CL}}$$0 that both form the training-time mixture $$FS=\text{mIoU}_{\mathrm{joint}}-\text{mIoU}_{\mathrm{CL}}$$1 and perform inference-time soft selection without task-ID. To stabilize the selector, the Attentive Reflection Module trains a lightweight generative replay PET block per task and applies a KL loss so that current attention on pseudo-inputs remains close to stored historical attention. On T5-Large, SAPT-LoRA reaches AP $$FS=\text{mIoU}_{\mathrm{joint}}-\text{mIoU}_{\mathrm{CL}}$$2, F.Ra $$FS=\text{mIoU}_{\mathrm{joint}}-\text{mIoU}_{\mathrm{CL}}$$3, FWT $$FS=\text{mIoU}_{\mathrm{joint}}-\text{mIoU}_{\mathrm{CL}}$$4, and BWT $$FS=\text{mIoU}_{\mathrm{joint}}-\text{mIoU}_{\mathrm{CL}}$$5 on SuperNI, compared with AP $$FS=\text{mIoU}_{\mathrm{joint}}-\text{mIoU}_{\mathrm{CL}}$$6 and F.Ra $$FS=\text{mIoU}_{\mathrm{joint}}-\text{mIoU}_{\mathrm{CL}}$$7 for replay. On LongSeq it reaches AP $$FS=\text{mIoU}_{\mathrm{joint}}-\text{mIoU}_{\mathrm{CL}}$$8 and F.Ra $$FS=\text{mIoU}_{\mathrm{joint}}-\text{mIoU}_{\mathrm{CL}}$$9. Removing ARM drops AP to $\Omega_N$0 and raises F.Ra to $\Omega_N$1, showing that the attention-alignment mechanism is not a minor auxiliary term (Zhao et al., 2024).

HiDe-PET provides a more explicitly decomposed formulation. It models continual learning with pre-training as three subproblems: within-task prediction, task-identity inference, and task-adaptive prediction. The architecture therefore combines task-specific PET parameters $\Omega_N$2, task-shared PET parameters $\Omega_N$3, a TII head $\Omega_N$4, and a TAP head $\Omega_N$5, together with stored representation statistics $\Omega_N$6 and $\Omega_N$7. Across Split CIFAR-100, Split ImageNet-R, Split CUB-200, and Split Cars-196, HiDe-PET improves on recent prompt- and adapter-based baselines. Under Sup-21/1K on Split CIFAR-100, LAE-Adapter reports FAA $\Omega_N$8 and HiDe-Adapter $\Omega_N$9; gains are larger on fine-grained splits and weaker pre-training (Wang et al., 2024).

PEARL combines task-specific adapters with dynamic rank allocation. Starting from a shared reference model $\Delta\Theta^t$00, it fine-tunes a copy on the current task, forms the layerwise task vector $\Delta\Theta^t$01, computes an SVD, and chooses the smallest rank $\Delta\Theta^t$02 whose explained variance exceeds a proximity-based threshold $\Delta\Theta^t$03. New LoRA adapters are then initialized from the truncated factors and trained while the reference remains fixed. On Seq-CIFAR100 with 5 tasks under Class-IL, PEARL with ResNet-18 reaches $\Delta\Theta^t$04 versus $\Delta\Theta^t$05 for AGILE and $\Delta\Theta^t$06 for DER++; on ImageNet-R with 10 tasks and ViT-B/16, PEARL reaches $\Delta\Theta^t$07 versus $\Delta\Theta^t$08 for InfLoRA and $\Delta\Theta^t$09 for C-LoRA. Its limitations are equally explicit: parameter isolation causes model size to grow with $\Delta\Theta^t$10, Class-IL inference cost is $\Delta\Theta^t$11, and the method requires known task boundaries and per-task full-reference fine-tuning for SVD estimation (Bhat et al., 17 May 2025).

In few-shot class-incremental learning, CPE-CLIP extends prompt-based PECL to a multimodal setting. It freezes CLIP’s vision and language encoders and learns only language prompts, vision prompts derived through a projection $\Delta\Theta^t$12, and the projection itself, for about $\Delta\Theta^t$13K trainable parameters, less than $\Delta\Theta^t$14 of CLIP. Session-wise gradient scaling

$\Delta\Theta^t$15

reduces prompt plasticity as the class set grows. On miniImageNet, the method reports $\Delta\Theta^t$16 in session 0, $\Delta\Theta^t$17 in session 8, average $\Delta\Theta^t$18, and performance drop $\Delta\Theta^t$19, whereas the next best methods in the summary lie around average $\Delta\Theta^t$20–$\Delta\Theta^t$21 with drops around $\Delta\Theta^t$22–$\Delta\Theta^t$23 (D'Alessandro et al., 2023).

5. Spectral and representation-space formulations

Several recent PECL methods replace standard low-rank adaptation by explicitly structured subspaces. In ASR, CSSVD decomposes each pretrained weight matrix $\Delta\Theta^t$24 by SVD into head and tail singular directions, freezes the head $\Delta\Theta^t$25, and adapts only the low-energy tail by an approximate rotation

$\Delta\Theta^t$26

For later tasks, rotations are combined by convex weight averaging. On an OWSM v3.2 small model with $\Delta\Theta^t$27M parameters, only linear weights totaling about $\Delta\Theta^t$28M parameters are updated. CSSVD reaches average WER $\Delta\Theta^t$29 and BWT $\Delta\Theta^t$30 in Experiment 1, and $\Delta\Theta^t$31 and $\Delta\Theta^t$32 in Experiment 2, outperforming PECL baselines while staying near zero forgetting (Eeckt et al., 8 Jun 2026).

TailLoR also works in spectral coordinates, but fixes the singular bases $\Delta\Theta^t$33 and $\Delta\Theta^t$34 of a pretrained weight $\Delta\Theta^t$35 and learns only a low-rank perturbation of the singular-value matrix,

$\Delta\Theta^t$36

A soft spectral penalty weights updates more heavily in dominant directions, so optimization is routed into the singular-value tail. On Standard and Long Sequence benchmarks with T5-Large, TailLoR with head protection obtains the best combined OA, $\Delta\Theta^t$37, slightly above ELLA at $\Delta\Theta^t$38; on TRACE it reaches OA $\Delta\Theta^t$39 and BWT $\Delta\Theta^t$40, compared with $\Delta\Theta^t$41 and $\Delta\Theta^t$42 for ELLA (Dragoi et al., 4 Jun 2026). This suggests that not all low-rank coordinates are equally safe for continual adaptation.

CoRe moves the intervention from weight space to representation space. For each task and selected layer, it introduces $\Delta\Theta^t$43, $\Delta\Theta^t$44, and $\Delta\Theta^t$45, and edits a base representation $\Delta\Theta^t$46 by

$\Delta\Theta^t$47

while regularizing $\Delta\Theta^t$48. The paper gives an explicit drift bound,

$\Delta\Theta^t$49

so stability is controlled at the level of hidden representations rather than inferred from weight norms. Empirically, CoRe reports the highest accuracy on CIFAR-Inc10 while using the fewest parameters, about $\Delta\Theta^t$50 of backbone parameters versus $\Delta\Theta^t$51–$\Delta\Theta^t$52 for adapters, and reports Avg $\Delta\Theta^t$53 on CIFAR-Inc5 versus $\Delta\Theta^t$54 for Adapter in the summary (Luo et al., 11 Mar 2026).

These methods indicate a broader shift within PECL: instead of treating low-rank updates as purely engineering devices, recent work uses singular directions, orthogonal subspaces, or representation-level constraints as explicit stability mechanisms. That shift is visible in vision, language, and speech, although the concrete parameterizations remain different across modalities.

6. Theory, resource trade-offs, and open problems

Theoretical work has started to formalize why some PECL schemes forget less than others. NTK-CL recasts test-time forgetting as generalization gaps under Neural Tangent Kernel analysis and identifies three governing factors: training sample size, task-level feature orthogonality, and regularization. Its algorithm responds directly to these factors by tripling each sample’s representation through three branches, using adaptive EMA to preserve intra-task NTK structure, imposing dissimilarity and orthogonality losses to suppress inter-task NTK overlap, and regularizing parameter drift. On CIFAR-100 it reports $\Delta\Theta^t$55 and $\Delta\Theta^t$56, compared with $\Delta\Theta^t$57 and $\Delta\Theta^t$58 for the best prior baseline listed in the summary; on ImageNet-A it reports $\Delta\Theta^t$59 and $\Delta\Theta^t$60, compared with $\Delta\Theta^t$61 and $\Delta\Theta^t$62 (Liu et al., 2024).

Several recurring misconceptions can be stated precisely. First, PECL is not inherently rehearsal-free: ConPET stores all past data on disk and SAPT uses generative replay PET blocks, whereas LAE, SAFE, and PIECE are explicitly memory-free or zero-history in the summaries (2309.14763). Second, PECL is not synonymous with per-task parameter growth: Dynamic ConPET, SAPT, PEARL, and CPE-CLIP grow task modules, but CLoRA, EWC-LoRA, SAFE, LAE, and PIECE maintain constant or in-place parameter footprints across tasks (Muralidhara et al., 26 Jul 2025). Third, parameter efficiency is not exhausted by counting trainable parameters. CLoRA’s NetScore shows that mIoU alone can miss substantial compute savings; EWC-LoRA adds about $\Delta\Theta^t$63 GB Fisher overhead relative to Vanilla LoRA on ViT-B/16; and PEARL’s Class-IL inference requires routing each input through all task heads, so low adaptation cost does not imply low deployment cost (Zheng et al., 19 Feb 2026).

Open problems are repeatedly identified across the literature. The PECFT survey highlights subspace interference, task-agnostic inference, module merging, richer reasoning tasks, and more realistic online or unsupervised streams (Coleman et al., 18 Apr 2025). ConPET proposes hierarchical PET module trees, adaptive replay budgets, and meta-continual PET (2309.14763). PEARL leaves alternative distance metrics and adapter folding as future directions (Bhat et al., 17 May 2025). CoRe points to adaptive rank selection and compression across many task modules (Luo et al., 11 Mar 2026). A plausible implication is that future PECL research will be shaped less by a single dominant PET primitive than by how explicitly a method controls the stability–plasticity trade-off, resource overhead, and task inference under deployment constraints.