---
title: CLS Token Attention Steering Prompts (CASP)
url: https://www.emergentmind.com/topics/cls-token-attention-steering-prompts-casp
type: topic
---

# CLS Token Attention Steering Prompts (CASP)

CLS Token Attention Steering Prompts (CASP) are algorithmic mechanisms for explicitly controlling and enhancing the contribution of the global [CLS] token in Transformer-based models, by introducing trainable steering biases or specialized prompt vectors in key positions of the network's attention structure. Originating in recent few-shot class-incremental learning (FSCIL), visual prompt tuning, and prompt-efficient learning, CASP techniques extend to image, language, and multimodal models. The aim is to directly modulate the self-attention dynamics surrounding aggregation tokens such as [CLS], thereby improving discrimination, transfer, and robustness while maintaining high parameter efficiency and mitigating catastrophic forgetting [2601.16773, 2510.16670, 2505.02406].

## 1. Architectural Principles of CASP

CASP methods architecturally center on manipulating the processing of the [CLS] or global aggregation token in Transformer layers. In canonical Vision Transformers (ViT), the [CLS] token is responsible for aggregating sequence-wide semantic information through self-attention. CASP augments this process via the following interventions:

- **Steering Bias Injection**: Introduces trainable D-dimensional prompts—denoted $S_Q$, $S_K$, $S_V$—added to the query, key, and value (Q/K/V) projections of the [CLS] token in every Transformer block. For a given block, with $x_{cls} \in\mathbb{R}^D$ and frozen $W_Q, W_K, W_V \in\mathbb{R}^{D\times D}$,
  \[
  Q'_{cls} = W_Q x_{cls} + S_Q,\quad K'_{cls} = W_K x_{cls} + S_K,\quad V'_{cls} = W_V x_{cls} + S_V
  \]
  These biases redirect the [CLS] token's attention filter towards task-relevant features, enabling explicit control over what information is emphasized or suppressed [2601.16773].

- **Instance- and Layer-Adaptive Capsule Prompts**: Certain implementations substitute the fixed CLS token by a dynamic "capsule prompt" at each layer, obtained as the sum of trainable task prompts and instance-adaptive means of layer representations:
  \[
  S^i = p^i + \text{Mean}(\text{inputs at layer } i)
  \]
  This forms an "attention anchor", concentrating bidirectional attention between the prompt and all sequence tokens [2510.16670].

- **CLS/Image Prompt Disentanglement**: Visual prompt coordination frameworks further split prompts into CLS prompts (steering only the [CLS]) and image prompts (dedicated to local patch tokens), running separate brief self-attention passes to specialize updates for each role [2505.02406].

- **Multi-CLS Token Approaches**: For weakly supervised segmentation, multiple class-specific CLS tokens are introduced, one for each possible class, and CASP mechanisms promote one-to-one assignments between tokens and semantic classes via random masking [2507.06848].

## 2. Attention Steering and Perturbation Strategies

CASP systems are augmented by perturbative and regularization mechanisms to ensure robust transfer and generalization:

- **Attention Perturbation**: During training, steering prompts $S_Q, S_K, S_V$ are stochastically perturbed with dropout or Gaussian noise, yielding perturbed prompts $S'_Q, S'_K, S'_V$:
  \[
  S'_Q = \text{Dropout}(S_Q; p),\quad S'_Q = S_Q + \epsilon, \quad \epsilon \sim \mathcal{N}(0, \sigma^2 I)
  \]
  This regularization requires the model to learn attention filters stable against perturbations, ensuring that representations are not overfitted to the base session classes [2601.16773].

- **Dynamic Inference-Time Steering**: In language models, attention steering can be applied at inference-time, enforcing a user-specified minimum attention mass $\psi_{target}$ on certain tokens (e.g., instruction spans or [CLS]) by adaptively biasing raw attention logits:
  \[
  A_{ij}' = \text{softmax}_j(L_{ij} + B_{ij}),\quad B_{ij} = \mathbb{I}(j \in S)\log\alpha_i,\quad \alpha_i = \max(1, {\psi_{target}}/{\psi_{current}(i)})
  \]
  This adaptive scheme enables plug-and-play emphasis of regions or instructions without retraining [2505.12025].

## 3. Feature Generalization via Token Mixup and Prompt Matching

CASP solutions integrate explicit feature augmentation strategies to simulate future-task variability:

- **Manifold Token Mixup**: Synthetic "virtual" classes are generated by convexly interpolating shallow token sets from the base session:
  \[
  \hat{Z} = \lambda Z + (1-\lambda)Z[\text{idx}]
  \]
  These mixed features are propagated through the transformer, and a mixed loss is applied to regularize the network toward smoother decision boundaries and reserved latent capacity [2601.16773].

- **Prompt Matching Functions**: For image tokens, CASP-linked frameworks can match tokens to prompt pools via cosine similarity, assigning prompt vectors to those most semantically aligned, resulting in more precise, diverse feature extraction [2505.02406].

## 4. Training Protocols, Efficiency, and Empirical Results

CASP is characterized by high efficiency with respect to trainable parameters and compute overhead:

- **Parameter Isolation**: Only the steering bias vectors (and any domain adaptation prompt) are updated during base training; all backbone parameters remain frozen, and no fine-tuning is performed during incremental phases [2601.16773].
- **Prototype-Based Incremental Learning**: In FSCIL, new class prototypes are constructed as means in CLS-embedding space and appended to an expanding classifier bank. The backbone is fixed for all future sessions, preventing catastrophic forgetting [2601.16773].
- **Efficiency Metrics**: CASP (ViT-B/16, CUB200) incurs only 0.1MB additional parameter cost—substantially lower than competing prompt, adapter, or privilege-based PEFT methods. It also avoids grid search for prompt length present in alternative approaches [2601.16773, 2510.16670].

| Method           | Trainable Param (MB) | CUB200 A_avg (%) | CIFAR100 A_avg (%) | ImageNet-R A_avg (%) |
|------------------|---------------------|------------------|--------------------|----------------------|
| CASP             | 0.1                 | 86.4             | 90.2               | 77.4                 |
| SEC-F            | 1.7                 | 84.9             | —                  | 76.7                 |
| ASP              | 3.0                 | —                | —                  | —                    |
| PriViLege        | 16.3                | —                | —                  | —                    |

CASP exhibits reduced base-to-last session forgetting (A_B – A_L), indicating strong preservation of previously learned representations [2601.16773].

## 5. Application Domains and Variants

CASP has been deployed across a diverse range of tasks:

- **Few-Shot Class-Incremental Learning (FSCIL)**: Attains state-of-the-art accuracy in few-shot increments on CUB200, CIFAR100, and ImageNet-R without incremental fine-tuning [2601.16773].
- **Prompt-Efficient Fine-Tuning**: Capsule prompt variants show that a single adaptive vector per layer achieves or surpasses full fine-tuning on several NLP benchmarks like SuperGLUE, using only 0.004% of model parameters—markedly outperforming other PEFT techniques [2510.16670].
- **Visual Prompt Specialization**: Token-Coordinated Prompt Attention demonstrates that separating CLS and patch prompts improves class discrimination and feature diversity for vision tasks [2505.02406].
- **Weakly Supervised Semantic Segmentation**: Multiple CLS token CASP instantiations enable pseudo-mask extraction with competitive mIoU, outperforming multi-stage and CAM-based WSSS baselines [2507.06848].
- **Instruction-Following in LLMs**: Dynamic inference-time CASP enables users to "spotlight" instructions or segments, improving syntactic and multi-turn instruction-following robustness [2505.12025].

## 6. Broader Context, Extensions, and Limitations

CASP conceptualizes the [CLS] token as a human-like attention filter: its steering prompts allow dynamic, task-aware re-weighting of the attention field in a model. This motivates several directions:

- **Multimodal and Task-Adaptive Extensions**: The primary results are in unimodal visual and language models; future CASP research may focus on cross-modal transformers, variable-way FSCIL, or patch-level manifold mix approaches [2601.16773].
- **CLS Token Specialization**: Disentangling CLS from local (patch or word) tokens and providing dedicated biasing has empirically been shown to yield +0.7–1.5 pp gains over single-prompt or undifferentiated prompt methods [2601.16773, 2505.02406].
- **Limitations**: All known CASP implementations depend on clear summary (CLS) tokens; performance can degrade if steering is applied too early/late in deep networks [2304.06212] or if prompts overfit to the base data distribution (mitigated by perturbations and mixup).

In summary, CLS Token Attention Steering Prompts provide a paradigm for principled, parameter-efficient global attention modulation within transformers. By focusing on explicit control of the [CLS] aggregation mechanism—and coupling this with prompt perturbation and mixup—they consistently outperform previous methods in transfer, adaptation, and robust continual learning benchmarks, with high computational efficiency and generalizability across modalities [2601.16773, 2510.16670, 2505.02406, 2505.12025, 2507.06848, 2304.06212, 2211.06083].

Source: https://www.emergentmind.com/topics/cls-token-attention-steering-prompts-casp