Enhancing Continual Learning of Vision-Language Models via Dynamic Prefix Weighting

Abstract: We investigate recently introduced domain-class incremental learning scenarios for vision-LLMs (VLMs). Recent works address this challenge using parameter-efficient methods, such as prefix-tuning or adapters, which facilitate model adaptation to downstream tasks by incorporating task-specific information into input tokens through additive vectors. However, previous approaches often normalize the weights of these vectors, disregarding the fact that different input tokens require different degrees of adjustment. To overcome this issue, we propose Dynamic Prefix Weighting (DPW), a framework that dynamically assigns weights to prefixes, complemented by adapters. DPW consists of 1) a gating module that adjusts the weights of each prefix based on the importance of the corresponding input token, and 2) a weighting mechanism that derives adapter output weights as a residual of prefix-tuning weights, ensuring that adapters are utilized only when necessary. Experimental results demonstrate that our method achieves state-of-the-art performance in domain-class incremental learning scenarios for VLMs. The code is available at: https://github.com/YonseiML/dpw.

• The paper introduces Dynamic Prefix Weighting (DPW), dynamically modulating token-level prefix and adapter contributions to enhance continual learning in vision-language models.
• It employs a novel combination of RePA, CondAct, and RWM modules to improve task separation, reduce interference, and achieve state-of-the-art transfer performance.
• Empirical results on MTIL and ODCL benchmarks demonstrate that DPW outperforms existing methods while reducing parameter overhead and ensuring robust forward and backward transfer.

Dynamic Prefix Weighting for Parameter-Efficient Continual Learning in Vision-LLMs

Introduction

The paper "Enhancing Continual Learning of Vision-LLMs via Dynamic Prefix Weighting" (2604.18075) addresses the critical limitations of current parameter-efficient fine-tuning (PEFT) approaches in the context of continual learning (CL) of vision-LLMs (VLMs). Existing methods—such as prompt/prefix-tuning and adapter tuning—generally allocate task-specific parameters at the sample or task level, overlooking the token-level heterogeneity in adaptation needs. This work introduces Dynamic Prefix Weighting (DPW), a framework that orchestrates token-level modulation of prefix and adapter contributions for VLMs in incremental learning scenarios, enabling fine-grained adaptation without sacrificing forward- and backward-transfer performance.

Motivation and Analysis of Existing PEFT Strategies

Conventional PEFT approaches (e.g., MoE-Adapters, DIKI) design mechanisms where the additive vectors' contributions are normalized per sample or by a router determined from a summary token (often [CLS]), enforcing uniform influence for all tokens in a sample. This rigid strategy disregards the nuanced variation in task relevance across input tokens. Empirical evidence and recent literature highlight the performance gains by injecting task-specific information proportional to token relevance; however, a lack of suitable weighting mechanisms and the intrinsic entanglement induced by the pretrained backbones' attention projections hinders fine-grained token-level adaptation.

Figure 1: Token weighting mechanism comparison: only DPW enables token-level, non-uniform modulation of both prefixes and adapters.

Furthermore, the Transformer attention mechanism, when employed for prefix assignment, projects task-relevant and irrelevant tokens closer in the feature space, limiting the distinction necessary for effective downstream transfer in CL. The provided analysis highlights that query projection in Transformer layers destroys separability between task-relevant and task-irrelevant embeddings, aggravating interference and impeding robust knowledge retention and transfer.

The DPW Framework

The DPW architecture consists of three primary components:

1. Reparameterized Prefix Attention (RePA)

RePA substitutes the canonical attention-based prefix scoring with a learnable affine transformation applied directly to input tokens—sidestepping the query-key dot-product and thus detaching from the target distribution bias in the pretrained backbone. This enables learning task-specific, token-wise relevance scores while avoiding the substantial parameter overhead of end-to-end training new projection matrices.

Figure 2: DPW framework overview: RePA computes per-token prefix scores, CondAct assigns normalized weights, and residual weight is dynamically routed to the adapter.

Empirical analysis demonstrates that before projection, task-relevant and irrelevant tokens are well separated, but this structure collapses after query projection—a phenomenon directly remedied by RePA.

2. Conditional Activation (CondAct)

CondAct replaces softmax with a composite operation: scores undergo sigmoid gating, followed by conditional normalization (capping total prefix weight per token at 1), and finally conditional filtering (dynamic thresholding based on score likelihood under Gaussian fit to observed training statistics). This procedure allows token-level prefix utilization to flexibly range up to a bounded maximum, strictly limiting representation drift and mitigating catastrophic and forward forgetting.

Theoretical analysis clarifies that bounding prefix-induced drift per token via CondAct reduces worst-case perturbation by a factor of prefix length compared to sigmoid-only strategies.

Figure 3: Prefix score maps: DPW yields refined task-relevant distributions compared to standard attention.

3. Residual Weighting Mechanism (RWM) for Adapters

Any residual prefix demand (i.e., when the total desired adaptation exceeds the prefix bound) is dynamically allocated to the adapter output via RWM. The adapter is implemented with a LoRA-based module, refined for CL by constraining the down-projection matrix to the principal subspace of the pretrained value projection. The RWM ensures that prefixes handle token-level adaptation up to a stable maximum, delegating only excess per-token adaptation to the adapter—this suppresses interference and promotes orthogonality between prefix and adapter contributions.

Figure 4: Cosine similarity between prefix and adapter outputs: RWM yields near-orthogonal representations, confirming complementary roles.

Empirical Evaluation

Extensive experiments are conducted on two challenging benchmarks for domain- and class-incremental learning: Multi-domain Task Incremental Learning (MTIL) and Open-Domain Continual Learning with Class-Incremental Labeling (ODCL-CIL). The DPW model, with and without an aggressive parameter reduction variant, consistently exceeds prior state-of-the-art methods including MoE-Adapters, DIKI, and GIFT in all metrics (Transfer, Last, Avg.), even surpassing approaches that train all model parameters or use synthetic reference data.

Strong numerical results include:

• On MTIL, the main DPW model achieves up to 79.3% average and 88.3% last accuracy, outpacing GIFT and MoE-Adapter baselines.
• In ODCL-CIL (more challenging due to lack of task IDs at inference), DPW matches or surpasses prompt and adapter-based SOTA methods with 78.6% average and 86.6% last accuracy.

DPW also offers significant efficiency advantages: due to lightweight affine scoring (RePA) and elimination of routers, inference time is reduced compared to MoE-Adapter/DIKI baselines, and parameter count is minimized with negligible loss—even when scaling to fewer adaptative heads.

Figure 5: Mean performance versus parameter count: DPW delivers higher accuracy with fewer trainable parameters compared to DIKI.

Additional Analysis

Ablation studies confirm the necessity of each DPW module: RePA improves task separation; CondAct is essential to bound interference and adapt prefix utilization; and RWM is critical for effective adapter routing and complementarity. UMAP and cosine similarity visualizations robustly demonstrate enhanced disentanglement and token-wise modulation compared to both simple softmax/sigmoid and router-based approaches.

Figure 6: (Left) Cosine similarity: Query projection over-collapses token diversity. (Right) UMAP: RePA preserves inter-task separability post-projection.

Figure 7: Dynamic cutoff in CondAct versus fixed threshold: Only dynamic, data-driven filtering consistently achieves optimal forward and backward transfer.

Practical and Theoretical Implications

The modular DPW design (RePA+CondAct+RWM) provides a blueprint for token-level, bounded adaptation in parameter-efficient CL. The theoretical bound on representation drift induced by prefixes allows for reliability in knowledge retention, especially in zero-shot and forward-transfer settings. No explicit router or batch-voting is required, further reducing computational and memory overhead.

In practical terms, DPW offers a robust mechanism for deploying VLMs in continual/streaming environments with heterogeneous tasks and limited adaptation budgets—key for edge deployment scenarios or open-world vision-language tasks. Furthermore, the concept of residual token-level weighting can be extended to other domains where per-token or per-dimension adaptation needs are highly uneven.

Conclusion

The DPW framework introduces a rigorous and effective approach to parameter-efficient continual adaptation in VLMs, characterized by dynamic, token-level balancing of prefix and adapter utilization. Bypassing the limitations of attention-based query-key routing and sample-level normalizations, DPW achieves stronger transfer, better retention, and higher efficiency than all prior SOTA PEFT methods. Its general principles—the residual weighting mechanism, bounded conditional activation, and input-centric affine scoring—suggest broad applicability for future CL methods that must robustly trade off adaptation granularity, efficiency, and knowledge retention.