Prompt-based Continual Learning (PCL)

• Prompt-based continual learning is a paradigm that leverages a frozen pre-trained transformer backbone combined with small, learnable prompts to adapt to new tasks without catastrophic forgetting.
• It mitigates forgetting by using prompts as compact external memory that condition the model through dynamic prompt retrieval, routing, and consistency mechanisms.
• The approach is highly parameter-efficient and has been extended to diverse settings, including domain-incremental, open-world, and security-sensitive applications.

Prompt-based continual learning (PCL) is a rehearsal-free continual learning paradigm in which a pre-trained backbone—most often a Vision Transformer (ViT)—is kept frozen while small learnable prompts are updated across a sequence of tasks. In the standard class-incremental setting, tasks introduce disjoint classes and task identity is unknown at test time; prompts therefore function as a compact external memory that conditions the frozen backbone without storing past data. A widely used formulation appears in "Learning to Prompt for Continual Learning" (Wang et al., 2021), and subsequent work has extended the paradigm to domain-incremental, task-agnostic, medical, security-sensitive, and open-world representation-learning settings (Kim et al., 2024, Oh et al., 13 Mar 2026, Kim et al., 2024).

1. Problem formulation and continual-learning regimes

PCL is usually studied under continual learning protocols in which a single model observes a sequence of datasets or tasks $\{\mathcal{D}_1,\dots,\mathcal{D}_T\}$ and must avoid catastrophic forgetting while adapting to new data. In the class-incremental learning (CIL) regime emphasized by several works, each task introduces new classes, the class sets are disjoint across tasks, and the model must classify among all classes seen so far without task labels at inference (2405.14124, Gao et al., 2024). Rehearsal-free formulations further prohibit storing exemplars from earlier tasks, which makes prompt-based adaptation attractive because the backbone can remain frozen while prompts and a classifier absorb task-specific change (Gao et al., 2024, Kim et al., 2024).

Although CIL remains the dominant benchmark, the literature has broadened the scope of PCL. Domain-incremental formulations keep the label space fixed while the input distribution shifts across domains; task identifiers may remain unavailable and past data infeasible to store, which makes prompt selection and classifier stability central concerns (Oh et al., 13 Mar 2026). Open-world continual representation learning goes further: subsequent streams introduce novel training classes that are disjoint from one another and from unseen test classes, so the objective is not only retention but also generalization to unseen categories under retrieval metrics such as Recall@1 (Kim et al., 2024). A distinct line transfers the same design principles to recommender systems, where prompts become task-conditioned external memory over user behavior sequences rather than visual tokens (Yang et al., 26 Feb 2025).

A useful analytical decomposition, adopted in theoretical work on PCL, separates CIL into within-task prediction (WTP) and task-identity inference (TII). Under this view, prompt-based methods improve both components, while some architectural modifications—especially better gating or routing—are designed specifically to improve WTP by making prompt- or expert-selection more statistically efficient (2405.14124). This suggests that PCL is not merely a parameter-efficient finetuning recipe, but also a structured answer to the classical stability–plasticity problem.

2. Architectural foundations: prompts, prefix tuning, and prompt pools

The canonical PCL architecture combines a frozen pre-trained transformer with a small number of learnable prompt parameters. In vision, prompts are usually injected by prompt tuning or prefix tuning. Under prefix tuning, prompt tokens are appended to key and value streams of self-attention, so the adapted attention block can be written as

$$f_{\mathrm{PreT}}(x;Q,K,V)=\mathrm{MSA}\left(Q,[K\;P^K],[V\;P^V]\right),$$

where $P^K$ and $P^V$ are learned key and value prefixes. In continual learning, each task may have its own prompt $P_t$, or prompts may be selected or composed from a shared prompt pool $[2405.14124]$.

In "Learning to Prompt for Continual Learning" (Wang et al., 2021), prompts are maintained in a memory space as a prompt pool paired with keys. For an input $x$, a query feature $q(x)$ is extracted from the frozen backbone, matched against prompt keys, and a subset of prompts is concatenated to the token sequence before the transformer stack. The objective jointly optimizes prompts, keys, and a classifier while keeping the backbone frozen. This makes continual learning a problem of constructing and retrieving prompt memory rather than repeatedly overwriting shared backbone parameters (Wang et al., 2021).

Many later methods preserve this frozen-backbone, prompt-only adaptation pattern while changing how prompts are parameterized. Some learn task-specific prompts inserted at selected transformer layers; some decompose prompts into general and expert components; some replace per-task prompt sets with a unified pool that is softly composed, sparsely selected, quantized, or dynamically generated from the input (Kim et al., 2024, Dai et al., 2024, Jiao et al., 2024). Across these variants, the central architectural claim remains consistent: catastrophic forgetting is mitigated because the backbone is not overwritten, while plasticity is localized in prompt space (2405.14124).

Parameter efficiency follows directly from this design. Prompts typically add only $O(L_p\cdot d)$ parameters per task, where $L_p$ is prompt length and $$f_{\mathrm{PreT}}(x;Q,K,V)=\mathrm{MSA}\left(Q,[K\;P^K],[V\;P^V]\right),$$0 the embedding dimension, often far fewer than full backbone finetuning. This compactness explains why PCL has been repeatedly presented as a rehearsal-free alternative to replay-heavy methods and as a practical option for privacy-constrained settings such as medical AI (2405.14124, Oh et al., 14 Aug 2025).

3. Prompt retrieval, generation, and routing

A large fraction of the PCL literature concerns prompt routing: how to decide which prompt, or which combination of prompts, should condition the frozen backbone for a given input. Early formulations used single-query, single-key matching. In L2P, the query is typically the backbone’s $$f_{\mathrm{PreT}}(x;Q,K,V)=\mathrm{MSA}\left(Q,[K\;P^K],[V\;P^V]\right),$$1 feature without prompts, and prompt selection is performed by top-$$f_{\mathrm{PreT}}(x;Q,K,V)=\mathrm{MSA}\left(Q,[K\;P^K],[V\;P^V]\right),$$2 retrieval in key space (Wang et al., 2021). This design is effective, but later work identifies low prompt-selection accuracy as a persistent bottleneck because the identity-prediction step is not always optimized directly by task loss, or because a single key is too coarse a summary of a task’s data distribution (Tu et al., 22 Jan 2025, Jiao et al., 2024).

Several strands of work refine this routing mechanism. "Consistent Prompting for Rehearsal-Free Continual Learning" (Gao et al., 2024) identifies two mismatches in prior prompt-based systems: classifier inconsistency, where training updates only the current task classifier but testing aggregates logits from all classifiers, and prompt inconsistency, where the prompt used at training may differ from the prompt selected at inference. It introduces classifier consistency learning, prompt consistency learning, and a multi-key prompt-selection mechanism. "Semantic Prompting with Image-Token for Continual Learning" (Han et al., 2024) removes explicit task prediction altogether: prompt selection is based on semantic similarity between image-token keys and prompt keys inside intermediate transformer layers, yielding a task-agnostic and single-pass design. "One-stage Prompt-based Continual Learning" (Kim et al., 2024) takes a related efficiency perspective, replacing the separate query ViT with intermediate-layer token embeddings from the backbone itself, thereby removing one of the two ViT feed-forward stages that earlier PCL methods used.

A second trend replaces hard selection with differentiable composition or generation. "Prompt Customization for Continual Learning" (Dai et al., 2024) reformulates prompting around a prompt generation module and a prompt modulation module: prompts are generated as continuous combinations of a fixed codebook and then modulated according to correlations between input features and generated prompts. "Vector Quantization Prompting for Continual Learning" (Jiao et al., 2024) addresses the opposite problem—continuous prompts may lack sufficient abstraction—by using vector quantization so that prompt selection becomes end-to-end trainable while remaining discrete. "Multiple Queries with Multiple Keys" (Tu et al., 22 Jan 2025) argues that precise prompt matching requires both prompt-aware multiple queries and class-level multiple keys; it reports prompt matching-rate improvements of over $$f_{\mathrm{PreT}}(x;Q,K,V)=\mathrm{MSA}\left(Q,[K\;P^K],[V\;P^V]\right),$$3 in challenging scenarios. "Key-Value Pair-Free Continual Learner via Task-Specific Prompt-Prototype" (Luo et al., 8 Jan 2026) removes key–value retrieval entirely and binds each task-specific prompt to class prototypes, so inference compares prompt-conditioned features to prototype-based classifiers rather than retrieving prompts from a key space.

These developments clarify a recurring point of contention in PCL: much of the empirical variance across methods comes not from whether prompts are used, but from how the method estimates the relationship between an input and the prompt distribution that best matches it.

4. Theoretical perspectives, stability–plasticity, and controlled growth

A major theoretical advance interprets self-attention itself as a special mixture-of-experts (MoE) architecture with shared linear experts and quadratic gating. Under this view, prefix tuning adds new task-specific experts to the frozen backbone’s implicit expert set, and the effectiveness of prompt-based continual learning can be analyzed through expert expansion and gating estimation rather than only through empirical prompt heuristics (2405.14124). The same work argues that standard linear gating for prefix prompts is statistically suboptimal and proposes Non-linear Residual Gates (NoRGa), which add a non-linear residual term to prefix gating scores to improve sample efficiency while preserving parameter efficiency (2405.14124).

This MoE perspective aligns with broader attempts to separate stability from plasticity. "PromptFusion: Decoupling Stability and Plasticity for Continual Learning" (Chen et al., 2023) explicitly proposes to decouple the two with a $$f_{\mathrm{PreT}}(x;Q,K,V)=\mathrm{MSA}\left(Q,[K\;P^K],[V\;P^V]\right),$$4 module for catastrophic forgetting and a $$f_{\mathrm{PreT}}(x;Q,K,V)=\mathrm{MSA}\left(Q,[K\;P^K],[V\;P^V]\right),$$5 module for new knowledge, together with PromptFusion-Lite, which dynamically determines whether to activate both modules for each input image. The abstract reports that on Split-Imagenet-R the method exceeds state-of-the-art prompt-based methods by more than $$f_{\mathrm{PreT}}(x;Q,K,V)=\mathrm{MSA}\left(Q,[K\;P^K],[V\;P^V]\right),$$6 in accuracy, while PromptFusion-Lite uses $$f_{\mathrm{PreT}}(x;Q,K,V)=\mathrm{MSA}\left(Q,[K\;P^K],[V\;P^V]\right),$$7 less computational resources than PromptFusion (Chen et al., 2023).

Another route addresses prompt-pool growth itself. "Learning Whether to Grow for Prompt-based Continual Learning" (Feng et al., 2024) treats prompt expansion as a decision problem rather than a default operation. Its Hinder Forward Capability (HFC) metric measures how much orthogonal gradient projection onto the complement of old feature spaces hinders learning of a new task, and the Dynamic Growing Approach uses HFC with a dynamic threshold to decide whether a new prompt set should be added or an existing one reused. A plausible implication is that prompt growth in PCL is no longer viewed as merely adding capacity; it is increasingly treated as a controlled mechanism for managing cross-task similarity and prompt retrieval accuracy (Feng et al., 2024).

These theoretical and semi-theoretical formulations converge on a common interpretation: PCL works well not simply because prompts are small, but because prompt space can impose structure on how knowledge is added, isolated, fused, or routed.

5. Methodological diversification and representative families

The PCL literature now contains several recognizable families of methods, distinguished by how they represent prompt diversity, how they evolve prompts over tasks, and which continual-learning regime they target.

Family	Representative methods	Characteristic mechanism
Consistency and matching	CPrompt, MQMK, ProP	Better train–test alignment, finer prompt matching, or prototype binding
Generation and customization	PC, VQ-Prompt, DPaRL	Soft generation, quantization, or dynamic prompt generation
Diversity and probabilistic modeling	Probabilistic Prompt, RainbowPrompt	Probabilistic prompt distributions, prompt evolving, probabilistic gates
Domain-specific and medical PCL	Residual SODAP, distributed medical PCL	Sparse routing, classifier preservation, minimal prompt expansion

One important branch studies diversity explicitly. "Learning Probabilistic Prompt for Continual Learning" (Park et al., 6 Jul 2026) diagnoses prompt collapse, the observation that prompts in deterministic pools tend to become highly similar to one another and fail to capture diverse data distributions. It responds by modeling each prompt as a Gaussian distribution, constructing a query-conditioned mixture of these distributions, sampling multiple prompts, and adding a distribution regularization loss based on KL divergence between successive prompt distributions (Park et al., 6 Jul 2026). "RainbowPrompt: Diversity-Enhanced Prompt-Evolving for Continual Learning" (Hong et al., 30 Jul 2025) similarly argues that fixed learned prompts and prompts generated from an entangled shared task space both limit representational diversity. Its prompt-evolving mechanism transforms and aligns accumulated task-specific prompts into a unified prompt, and a learnable probabilistic gate determines which layers to activate during evolution. The abstract reports average gains of $$f_{\mathrm{PreT}}(x;Q,K,V)=\mathrm{MSA}\left(Q,[K\;P^K],[V\;P^V]\right),$$8 on image classification and $$f_{\mathrm{PreT}}(x;Q,K,V)=\mathrm{MSA}\left(Q,[K\;P^K],[V\;P^V]\right),$$9 on video action recognition across all scenarios (Hong et al., 30 Jul 2025).

Another branch emphasizes dynamic prompt generation rather than a static pool. In the open-world setting of "Open-World Dynamic Prompt and Continual Visual Representation Learning" (Kim et al., 2024), Dynamic Prompt and Representation Learner (DPaRL) generates prompts on the fly from stage tokens and image features, then jointly learns these dynamic prompts and the visual representation under ArcFace-style metric learning. The paper reports an average $P^K$0 improvement in Recall@1 over state-of-the-art methods on open-world retrieval benchmarks (Kim et al., 2024). This suggests that prompt pools are not obligatory; in representation-learning settings with unseen test classes, prompt synthesis may be better aligned with the objective than prompt retrieval.

Domain- and application-specific variants further broaden the paradigm. "Residual SODAP" (Oh et al., 13 Mar 2026) targets rehearsal-free domain-incremental learning and combines $P^K$1-entmax sparse prompt selection, residual aggregation of frozen and active prompts, data-free distillation with pseudo-feature replay, prompt-usage-based drift detection, and uncertainty-aware multi-loss weighting. On three domain-incremental benchmarks without task IDs or extra data storage, it reports AvgACC/AvgF of $P^K$2 on DR, $P^K$3 on Skin Cancer, and $P^K$4 on CORe50 (Oh et al., 13 Mar 2026). "Towards Efficient Prompt-based Continual Learning in Distributed Medical AI" (Oh et al., 14 Aug 2025) uses a unified prompt pool with a minimal expansion strategy and a regularization term that balances retention and adaptation; on diabetic retinopathy datasets it reports at least $P^K$5 improvement in final classification accuracy and $P^K$6 points in F1-score over state-of-the-art approaches while lowering inference cost (Oh et al., 14 Aug 2025).

6. Empirical behavior, limitations, and security considerations

Empirically, PCL has repeatedly shown that strong rehearsal-free performance is possible with frozen backbones and small trainable prompt sets. On the efficiency side, "One-stage Prompt-based Continual Learning" (Kim et al., 2024) reports approximately $P^K$7 computational cost reduction for both training and inference by eliminating the separate query ViT, with marginal accuracy drop below $P^K$8, and further reports that Query-Pool Regularization improves performance sufficiently to outperform prior two-stage PCL methods by about $P^K$9 on CIFAR-100, ImageNet-R, and DomainNet (Kim et al., 2024). "Semantic Prompting with Image-Token for Continual Learning" (Han et al., 2024) reports competitive performance on four benchmarks while significantly reducing training time, including CIFAR-100 B0-Inc10 training time of $P^V$0 ms/image versus $P^V$1 for CODA-Prompt, and inference time of $P^V$2 ms/image versus $P^V$3 (Han et al., 2024). These results reinforce a central empirical property of PCL: routing and prompt insertion strategy can matter as much as raw prompt capacity.

At the same time, several limitations recur across the literature. Prompt selection can be noisy when task counts are large or when a single query or key poorly represents data distributions; training and inference may be inconsistent if the model is trained with one prompt regime but tested with another; and prompt pools can collapse to highly similar vectors unless diversity is modeled explicitly (Gao et al., 2024, Park et al., 6 Jul 2026). A common misconception is that freezing the backbone alone resolves continual-learning difficulty. The accumulated work suggests otherwise: classifier instability, routing errors, insufficient prompt diversity, and entangled prompt-sharing mechanisms can still produce substantial degradation even when the backbone never changes (Oh et al., 13 Mar 2026, Han et al., 2024).

Security work adds a further caution. "Attack On Prompt: Backdoor Attack in Prompt-Based Continual Learning" (Nguyen et al., 2024) argues that the same remembering capability that makes PCL attractive for rehearsal-free learning can also preserve poisoned knowledge across tasks. Its continual backdoor framework reports up to $P^V$4 attack success rate across benchmark datasets and prompt-based continual learners (Nguyen et al., 2024). This does not invalidate PCL, but it establishes that prompt memory is an attack surface rather than a purely benign substitute for replay buffers.

Taken together, the current literature presents PCL as a mature and rapidly diversifying area rather than a single method. Its central design pattern—frozen pre-trained backbone plus continually adapted prompt space—has remained stable since L2P (Wang et al., 2021), but its active research frontier now lies in routing theory, prompt diversity, controlled growth, consistency between training and inference, classifier preservation, and robustness under open-world, domain-shifted, and adversarial conditions.