LDEPrompt: Layer-importance guided Dual Expandable Prompt Pool for Pre-trained Model-based Class-Incremental Learning

Abstract: Prompt-based class-incremental learning methods typically construct a prompt pool consisting of multiple trainable key-prompts and perform instance-level matching to select the most suitable prompt embeddings, which has shown promising results. However, existing approaches face several limitations, including fixed prompt pools, manual selection of prompt embeddings, and strong reliance on the pretrained backbone for prompt selection. To address these issues, we propose a \textbf{L}ayer-importance guided \textbf{D}ual \textbf{E}xpandable \textbf{P}rompt Pool (\textbf{LDEPrompt}), which enables adaptive layer selection as well as dynamic freezing and expansion of the prompt pool. Extensive experiments on widely used class-incremental learning benchmarks demonstrate that LDEPrompt achieves state-of-the-art performance, validating its effectiveness and scalability.

• The paper proposes a novel framework (LDEPrompt) that dynamically expands prompt pools based on layer importance to mitigate catastrophic forgetting.
• It employs a dual prompt pool design, using a frozen global pool for past tasks and a trainable pool for new ones, ensuring effective knowledge preservation.
• Experimental results on CIFAR100, CUB-200, and VTAB demonstrate significant accuracy improvements over SOTA methods.

Layer-Importance Guided Dual Expandable Prompt Pool for Pre-trained Model-based Class-Incremental Learning

Introduction

The paper "LDEPrompt: Layer-importance guided Dual Expandable Prompt Pool for Pre-trained Model-based Class-Incremental Learning" (2604.11091) introduces a novel framework for class-incremental learning (CIL) using pre-trained models. LDEPrompt addresses critical limitations of existing prompt-based CIL methods, such as rigid prompt pool capacities, lack of adaptive layer selection, and excessive reliance on the pretrained backbone for prompt selection. The framework leverages both a layer-importance criterion and a dual-pool architecture (global and training) to achieve dynamic prompt pool expansion, knowledge preservation, and improved cross-task adaptivity.

Methodology

LDEPrompt establishes a methodical, modular architecture to mitigate catastrophic forgetting and enhance class-incremental learning efficacy in PTM-based settings.

Layer Importance Evaluation

Prior to each incremental task, LDEPrompt computes layer-wise information gain via mutual information estimates between layer inputs and outputs, normalized with a softmax. Only those layers with above-average importance scores receive prompt pool insertions, which sharply contrasts with previous works employing a flat, layer-agnostic prompt allocation. This approach enables more parameter-efficient adaptation and targets prompts to the most informative representational subspaces.

Dual Expandable Prompt Pools

LDEPrompt introduces a strictly frozen global prompt pool for previous tasks and a dynamically constructed, task-specific training prompt pool. The training pool draws from the global pool using cosine similarity between the CLS token feature and prompt keys, selecting the top-$s$ most relevant prompts and initializing the remainder as new, trainable vectors. After each task, the newly acquired prompts are merged into the global pool, effectively expanding it and enabling cumulative knowledge accretion without mutual interference. The frozen nature of the global pool avoids catastrophic overwriting, a significant issue in conventional single-pool approaches.

Figure 1: Overview of LDEPrompt, depicting layer-importance evaluation, prompt retrieval, dynamic pool construction, and the merging protocol that distinguishes frozen and trainable prompt components.

Prompt Insertion and Optimization Protocols

Prompts are attached only at strategically chosen transformer layers (empirically shown to reduce redundancy and overfitting). During training, standard cross-entropy optimization is performed solely on the classifier and task prompts, while the backbone and the global prompts remain static throughout continual learning.

Experimental Results

LDEPrompt is evaluated under an exemplar-free CIL protocol on three canonical benchmarks: CIFAR100, CUB-200, and VTAB. Metrics include averaged accuracy across all incremental phases (Avg) and terminal-stage accuracy (Last).

Numerical Superiority

LDEPrompt demonstrates statistically significant improvements over previous SOTA methods, including DualPrompt, CODA-Prompt, EvoPrompt, ConvPrompt, and Aper-VPT. On CIFAR100 and VTAB, LDEPrompt achieves up to 0.50% and 1.37% absolute gains in Last accuracy, and as much as 2.91% on CUB-200 (final incremental accuracy), outperforming rigid-pool and non-adaptive baselines.

Figure 2: Accuracy evolution across incremental stages on CIFAR100, visualizing consistent performance lead and reduced degradation relative to baselines.

Ablation studies further quantify the contribution of each architectural component. Removing the dual-pool mechanism or forgoing prompt freezing yields substantial drops in performance. Universal prompt insertion (all layers) increases redundancy without performance benefits—underscoring the necessity of selective, importance-guided deployment.

Theoretical and Practical Implications

LDEPrompt's adaptive mechanism signals a shift toward architectures that reconcile scalability, plasticity, and stability in PTM-based CIL. The dual-pool structure inherently supports unbounded prompt pool growth, ensuring theoretical representational sufficiency as the task horizon expands. Layer-importance evaluation acts as an inductive bias for targeted parameterization, potentially generalizing to other domains and modalities.

Practically, LDEPrompt's capacity for plug-and-play integration with transformers makes it amenable for extension to NLP and multimodal CL scenarios—especially where exemplars are disallowed and catastrophic interference must be rigorously constrained. The retention of a frozen, expandable prompt memory also aligns with lifelong learning desiderata in privacy-sensitive or real-world deployments where task-specific data are strictly non-retained.

Future Directions

Future work should investigate automated criteria for prompt pool budgets, layer selection algorithms potentially informed by task context, and meta-learning extensions to further scale prompt reuse or enable cross-dataset transfer. Furthermore, probing the interaction of LDEPrompt with large vision-LLMs (VLMs) and vision transformers of increasing width/depth is warranted, as is adaptation to data stream and online CL protocols without explicit task boundaries.

Conclusion

LDEPrompt offers a principled, empirical advance for prompt-based class-incremental learning with transformers. By introducing a layer-importance-based prompt insertion strategy and a dual expandable prompt pool architecture, it enables dynamic capacity management and robust knowledge preservation, consistently achieving superior incremental learning performance compared to prior art. This framework lays the foundation for further research into scalable, interference-free continual adaptation in PTM-centric CL regimes.