Is Prompt Selection Necessary for Task-Free Online Continual Learning?

Abstract: Task-free online continual learning has recently emerged as a realistic paradigm for addressing continual learning in dynamic, real-world environments, where data arrive in a non-stationary stream without clear task boundaries and can only be observed once. To consider such challenging scenarios, many recent approaches have employed prompt selection, an adaptive strategy that selects prompts from a pool based on input signals. However, we observe that such selection strategies often fail to select appropriate prompts, yielding suboptimal results despite additional training of key parameters. Motivated by this observation, we propose a simple yet effective SinglePrompt that eliminates the need for prompt selection and focuses on classifier optimization. Specifically, we simply (i) inject a single prompt into each self-attention block, (ii) employ a cosine similarity-based logit design to alleviate the forgetting effect inherent in the classifier weights, and (iii) mask logits for unexposed classes in the current minibatch. With this simple task-free design, our framework achieves state-of-the-art performance across various online continual learning benchmarks. Source code is available at https://github.com/efficient-learning-lab/SinglePrompt.

• The paper demonstrates that eliminating prompt selection with a single prompt leads to comparable or higher accuracy in task-free online continual learning.
• It introduces a novel architecture combining prefix tuning, cosine similarity-based logits, and minibatch logit masking to reduce parameter count and computational overhead.
• Empirical results on benchmarks like CIFAR100 show that SinglePrompt outperforms existing methods by maintaining high accuracy across both non-disjoint and disjoint task scenarios.

Eliminating Prompt Selection in Task-Free Online Continual Learning: An Expert Overview of SinglePrompt

Introduction: Questioning the Necessity of Prompt Selection in Task-Free OCL

Task-free online continual learning (OCL) confronts the most unconstrained and realistic distribution shift scenario, where the absence of explicit task boundaries and the requirement for single-pass learning preclude prior approaches based on task-anchored memorization. Parameter-efficient fine-tuning (PEFT) strategies—especially prompt-based tuning coupled with prompt selection procedures—have dominated recent research, motivated by the intuition that multiple, contextually chosen prompts support flexibility and mitigate catastrophic forgetting. However, empirical investigations in this paper demonstrate that prompt selection mechanisms exhibit weak or negligible correlation with input semantics and often fail to exploit prompt diversity effectively.

Analysis of Prompt Selection Failures

A comprehensive audit of both task-free and task-based prompt selection algorithms, including L2P, MVP, MISA, DualPrompt, and ConvPrompt, was conducted using various benchmarks such as CIFAR100. The analysis revealed two fundamental issues:

• Prompt-to-class mapping is highly irregular: Multiple prompts are selected for the same class without semantic consistency, fragmenting the representation and leading to interference (Figure 1, Figure 2).
• Underutilization of the prompt pool: Many prompts are rarely or never selected throughout the learning trajectory, revealing the selection mechanism's limited adaptivity and redundancy.

Figure 1: Distribution of prompt selection counts per class in L2P, illustrating the lack of class-prompt alignment and substantial selection irregularity.

Figure 2: Analogous histogram for DualPrompt; demonstrates low prompt selection accuracy and poor class-to-prompt association.

These inefficacies persist even in settings with access to task IDs, invalidating the core assumption of input-specific or task-specific specialization via prompt selection. Notably, ablation experiments show that random prompt selection yields comparable results to learned selection, and a configuration with a single prompt, i.e., eliminating selection altogether, achieves equal or higher accuracy with far fewer parameters.

The SinglePrompt Architecture

Against this empirical backdrop, the paper introduces SinglePrompt, a design that reframes prompt-based continual learning as prompt selection-free prefix tuning coupled with classifier and training objective modifications for robust adaptation. The method consists of three critical elements:

1. Prefix Tuning with a Single Prompt: Each self-attention block in the backbone Vision Transformer (ViT) is augmented with one learnable prompt, eschewing the need for a prompt pool or selection strategy.

   Figure 3: Operational schematic of SinglePrompt. At each self-attention block, a single prompt is injected prior to the attention computation, and logits for unexposed classes are masked out.

2. Cosine Similarity-Based Logits: Instead of standard linear classifier logits, classification scores are computed via normalized cosine similarity between the encoded feature and each class prototype. This counteracts the classifier weight norm imbalance that exacerbates forgetting in early-learned classes.

   Figure 4: Visualization evidencing L2 norm disparity in linear classifier weights across observed classes, motivating cosine similarity-based logits.

3. Minibatch Logit Masking: During training, logits of classes absent in the current minibatch are masked, ensuring updates only affect exposed prototypes and reducing spurious prototype drift.

Experimental Validation

The SinglePrompt framework was rigorously evaluated on canonical task-free OCL scenarios, with particular emphasis on CIFAR100, Tiny ImageNet, and ImageNet-R, under the Si-Blurry protocol. The bench-marking includes all major replay-, regularization-, and prompt-based baselines.

Key empirical results include:

• On CIFAR100 (disjoint class ratio 50%, no memory buffer), SinglePrompt achieves 87.53% $A_{\text{last}}$ and 85.58% $A_{\text{auc}}$, improving upon the previous state-of-the-art (MISA) by 6.55% and using 60% fewer learnable parameters.
• Performance gains are robust to task distribution severity: SinglePrompt dominates across the spectrum from non-disjoint (single-task) to fully disjoint (worst-case) settings.
• Logit masking and cosine similarity individually and jointly contribute to reducing measured forgetting ($F_{\text{last}}$) and improving anytime inference (Figure 5).

  Figure 5: Anytime inference accuracy curves on CIFAR100. SinglePrompt outperforms MISA throughout training, consistently maintaining higher accuracy.

Ablation studies further show that Prefix-Tuning is the optimal PEFT adapter under these settings (outperforming LoRA, AdaptMLP, and regular prompt tuning) and that the selection of prompt length and the number of tuned layers can be tuned for parameter budget versus accuracy trade-offs, with $M=20$ and $K=5$ yielding the best empirical balance.

Theoretical and Practical Implications

From a theoretical perspective, these findings challenge the foundational premise of prompt selection as a necessity for task-free OCL. The empirical evidence shows that model capacity and memory efficiency can be decoupled from prompt selection granularity, invalidating the assumed communicative role of instance-adaptive prompt routing. Practically, this yields a simpler, more robust architecture with lower computational overhead, amenable to real-world deployment scenarios with strict resource constraints.

Future Directions

The results open several avenues for further research:

• Revisiting specialization in modular architectures: The lack of semantic separation in prompts suggests a need to rethink modularity and specialization under distribution shift.
• Theoretical analysis of invariant adaptation: Analytical frameworks should account for the emergent sufficiency of universal versus instance-specific adaptation mechanisms in non-stationary streams.
• Reinforced emphasis on classifier/representation design: With prompt specialization debunked, future work should prioritize mechanisms that directly stabilize classifier prototypes and promote equitable allocation of plasticity.

Conclusion

SinglePrompt demonstrates compelling evidence that prompt selection, a centerpiece in recent OCL literature, fails to provide meaningful adaptivity in task-free scenarios. A prefix tuning architecture with a single, universally reused prompt, cosine similarity-based classifier, and minibatch masking achieves state-of-the-art accuracy with substantially reduced parameter count and computational cost. This work recharacterizes the design space for prompt-based continual learning and provides a new minimalist standard for practical deployment and subsequent algorithmic innovation (2604.04420).