ARSP: Adaptive Residual Semantic Prompt in CRISP

• The paper introduces a one-level prompt-tuning mechanism that injects learnable residual prompts into the Mask2Former decoder via adaptive query matching.
• ARSP utilizes a CLIP-based residual semantic prompt generator to form a class-specific prompt pool, ensuring semantic coherence during incremental training.
• The module is optimized within CRISP using hierarchical prompt injection and contrastive semantic consistency loss, effectively reducing category confusion in continual video segmentation.

Adaptive Residual Semantic Prompt (ARSP) is a prompt-tuning module for continual video instance segmentation that injects class-level semantic knowledge into the Transformer decoder of Mask2Former. In the formulation introduced within CRISP, ARSP is a one-level prompt-tuning sub-module composed of a residual semantic prompt generator, an adaptive query–prompt matching stage, and hierarchical prompt injection into decoder self-attention. Its stated purpose is to address category-wise confusion during incremental training by constructing a learnable semantic residual prompt pool generated by category text and using an adjustive query-prompt matching mechanism to map current-task object queries to semantic residual prompts (Liu et al., 14 Aug 2025). A closely related antecedent is the two-level prompt-tuning scheme of "Semantic Residual Prompts for Continual Learning," which also uses residual semantic transfer, but for class-incremental continual learning in a frozen Vision Transformer rather than for video instance segmentation (Menabue et al., 2024).

1. Definition and research context

In CRISP, ARSP is defined as a one-level prompt-tuning mechanism tailored to the Transformer decoder of Mask2Former for video. The module has three named components: a Residual Semantic Prompt Generator, Adaptive Query–Prompt Matching, and Hierarchical Prompt Injection. The prompt generator learns a small pool of class-specific residual vectors $\{p_0,\ldots,p_{c_t-1}\}$ via a lightweight text-encoder head built on CLIP; the matching stage compares current-task object queries $Q_t \in \mathbb{R}^{N_q^t \times d}$ against the prompt pool using cosine similarity; and the injection stage adds the matched prompts into the value stream of decoder self-attention (Liu et al., 14 Aug 2025).

This formulation places ARSP within the broader family of prompt-based continual learning methods, but its role is narrower and more specific than the earlier semantic residual prompting framework of 2024. In "Semantic Residual Prompts for Continual Learning," prompt tuning is organized as a two-level adaptation mechanism for class-incremental continual learning: first-level CLIP text prompts produce stable class prototypes, and second-level prompts provide per-layer residuals that adapt a frozen ViT (Menabue et al., 2024). By contrast, the ARSP module in CRISP is explicitly described as one-level and decoder-centric.

A common misconception is to treat ARSP as a generic synonym for semantic residual prompting. The published descriptions distinguish them. The 2024 work studies Semantic Residual Prompts for continual learning with two prompt pools and generative replay, whereas the 2025 work names Adaptive Residual Semantic Prompt as a module inside CRISP for continual video instance segmentation, with adaptive query–prompt matching and a semantic consistency loss (Menabue et al., 2024).

2. Architectural composition

ARSP comprises three main blocks.

First, the Residual Semantic Prompt Generator learns the prompt pool from a set of learnable token sequences $X \in \mathbb{R}^{c_t \times d}$. A small MLP or adapter built on the frozen CLIP text encoder, denoted $G_\theta$, maps these tokens to the prompt pool:

$P_t = G_\theta(X) \in \mathbb{R}^{c_t \times d}.$

Each row $p_j$ is the residual semantic prompt for class $j$ (Liu et al., 14 Aug 2025).

Second, Adaptive Query–Prompt Matching compares the object queries of the current task with the prompt pool. Given $Q_t \in \mathbb{R}^{N_q^t \times d}$, ARSP forms a cosine-similarity matrix

$$S_{i,j} = \frac{q_i \cdot p_j}{\|q_i\|_2 \|p_j\|_2},$$

then selects, for each query $i$, the single best-matching class prompt index

$Q_t \in \mathbb{R}^{N_q^t \times d}$0

The matched prompt vectors are collected as

$Q_t \in \mathbb{R}^{N_q^t \times d}$1

Third, Hierarchical Prompt Injection inserts the matched prompts into each decoder layer’s self-attention value stream:

$Q_t \in \mathbb{R}^{N_q^t \times d}$2

The exposition states that injection occurs at every decoder layer’s self-attention value stream, and that decoder depth is $Q_t \in \mathbb{R}^{N_q^t \times d}$3 by default in Mask2Former (Liu et al., 14 Aug 2025).

These blocks together define ARSP as a semantic modulation mechanism over decoder queries rather than as a classifier head or retrieval-only module. A plausible implication is that the semantic prompts act as structured residuals aligned to query embeddings, rather than as static class tokens.

3. Mathematical formulation

The prompt-pool generation step follows the CoOp paradigm. Let $Q_t \in \mathbb{R}^{N_q^t \times d}$4 denote the number of classes in current task $Q_t \in \mathbb{R}^{N_q^t \times d}$5 and $Q_t \in \mathbb{R}^{N_q^t \times d}$6 the embedding dimension of the CLIP text encoder. ARSP defines a learnable token matrix

$Q_t \in \mathbb{R}^{N_q^t \times d}$7

and computes

$Q_t \in \mathbb{R}^{N_q^t \times d}$8

The exposition states that, in effect, $Q_t \in \mathbb{R}^{N_q^t \times d}$9 learns to map raw token embeddings into class prototypes in the same space as the object queries (Liu et al., 14 Aug 2025).

The query–prompt matching step is entirely similarity-based. For each query $X \in \mathbb{R}^{c_t \times d}$0 and prompt $X \in \mathbb{R}^{c_t \times d}$1, cosine similarity is used:

$X \in \mathbb{R}^{c_t \times d}$2

The matched prompt index is then

$X \in \mathbb{R}^{c_t \times d}$3

and the matched prompt set is

$X \in \mathbb{R}^{c_t \times d}$4

The semantic consistency term introduces an explicit contrastive objective. With indicator

$X \in \mathbb{R}^{c_t \times d}$5

the loss is

$X \in \mathbb{R}^{c_t \times d}$6

The positive pair is $X \in \mathbb{R}^{c_t \times d}$7 and the negative pairs are $X \in \mathbb{R}^{c_t \times d}$8 for $X \in \mathbb{R}^{c_t \times d}$9 (Liu et al., 14 Aug 2025).

This formalization is narrower than that of the 2024 continual-learning framework, where residual transfer is defined at the level of frozen ViT blocks as $G_\theta$0, inserted before the MLP residual path, and where two training stages separately optimize first-level text prompts and second-level residual prompts (Menabue et al., 2024). The distinction matters because the two systems use different carriers of semantic information: decoder self-attention values in CRISP, and per-layer ViT residuals in the earlier class-incremental setting.

4. Training procedure and optimization

Within the CRISP training loop, ARSP is invoked during each incremental-task training epoch. The documented sequence is:

1. Extract multi-scale features $G_\theta$1.
2. Decode object queries $G_\theta$2.
3. Generate the prompt pool $G_\theta$3.
4. Compute similarities $G_\theta$4.
5. For each query, select $G_\theta$5.
6. Form the matched prompt set $G_\theta$6.
7. Inject prompts into each decoder self-attention layer.
8. Compute the segmentation loss $G_\theta$7.
9. Compute the semantic consistency loss $G_\theta$8. 10. Compute the instance-correlation loss $G_\theta$9.
10. Optimize the total loss

$P_t = G_\theta(X) \in \mathbb{R}^{c_t \times d}.$0

1. Back-propagate only into $P_t = G_\theta(X) \in \mathbb{R}^{c_t \times d}.$1.
2. Update parameters with Adam/SGD (Liu et al., 14 Aug 2025).

The hyperparameters reported for ARSP inside CRISP are specific. The prompt pool size is $P_t = G_\theta(X) \in \mathbb{R}^{c_t \times d}.$2, equal to the number of new classes in task $P_t = G_\theta(X) \in \mathbb{R}^{c_t \times d}.$3; the embedding dimension is the CLIP text embedding size, reported as $P_t = G_\theta(X) \in \mathbb{R}^{c_t \times d}.$4 in the experiments; decoder depth is $P_t = G_\theta(X) \in \mathbb{R}^{c_t \times d}.$5 by default in Mask2Former; $P_t = G_\theta(X) \in \mathbb{R}^{c_t \times d}.$6 and $P_t = G_\theta(X) \in \mathbb{R}^{c_t \times d}.$7; no explicit temperature $P_t = G_\theta(X) \in \mathbb{R}^{c_t \times d}.$8 is introduced in $P_t = G_\theta(X) \in \mathbb{R}^{c_t \times d}.$9; and self-attention uses $p_j$0 scaling (Liu et al., 14 Aug 2025).

The design choices also clarify what ARSP does not include. The exposition states that the prompt dimension and CLIP adapter are frozen except for the small $p_j$1 and $p_j$2, and that there is no rehearsal or negative-sample mining beyond standard contrastive log-sum-exp (Liu et al., 14 Aug 2025). This contrasts with the 2024 semantic residual prompting framework, where generative replay for both training stages uses a Mixture-of-Gaussians fitted on past CLIP features or past ViT CLS features, and replay losses are added to the objective (Menabue et al., 2024).

5. Relation to semantic residual prompting in continual learning

The most direct precursor to ARSP is the 2024 work on semantic residual prompts for continual learning. That method begins from two frozen foundation models: a CLIP vision encoder and text encoder, and an ImageNet-pretrained ViT with $p_j$3 Transformer blocks. It augments them with two pools of learnable prompts: a first-level prompt pool $p_j$4, and a second-level prompt pool $p_j$5 (Menabue et al., 2024).

In that earlier framework, level 1 constructs stable class prototypes

$p_j$6

which serve as keys. Level 2 uses the input image embedding $p_j$7 to score similarities

$p_j$8

selects

$p_j$9

and forms per-layer residuals

$j$0

These residuals are injected into each ViT block before the MLP update (Menabue et al., 2024).

ARSP in CRISP preserves several conceptual motifs from this earlier line of work: semantic prompting, residual-style injection, and explicit matching between learned semantic prototypes and current representations. However, the mechanisms are materially different. The 2024 method is two-level, uses CLIP text prototypes as stable keys, and targets a frozen ViT for class-incremental classification; the 2025 ARSP module is one-level, matches object queries directly to class prompts via cosine similarity, and injects the matched prompts into decoder self-attention for continual video instance segmentation (Menabue et al., 2024).

This comparison helps delimit the term. ARSP is best understood not as the entire semantic residual prompting paradigm, but as a specific decoder-level realization of that paradigm in CRISP.

6. Empirical role, design rationale, and interpretation

For CRISP, the abstract attributes category-wise learning to ARSP: it constructs a learnable semantic residual prompt pool generated by category text and uses an adjustive query-prompt matching mechanism to build a mapping relationship between the query of the current task and the semantic residual prompt. The abstract further states that a semantic consistency loss based on contrastive learning is introduced to maintain semantic coherence between object queries and residual prompts during incremental training, and that experiments on YouTube-VIS-2019 and YouTube-VIS-2021 demonstrate that CRISP significantly outperforms existing continual segmentation methods in the long-term continual video instance segmentation task, avoiding catastrophic forgetting and effectively improving segmentation and classification performance (Liu et al., 14 Aug 2025).

For the earlier class-incremental continual-learning framework, the empirical pattern is reported in more detail. On nine benchmarks—Split ImageNet-R, CIFAR-100, Cars-196, CUB-200, EuroSAT, RESISC45, CropDiseases, ISIC, and ChestX—the method outperforms zero-shot CLIP and prior prompt-tuning continual-learning methods by large margins, including $j$1 points over CLIP on EuroSAT and $j$2 on Cars-196. The average final accuracy gain over the best rehearsal-free prior is reported as $j$3, and the gap to a joint-training upper bound as only $j$4. Ablations show that removing Level 2 prompts, replacing additive residuals with prefix-tuning, dropping replay or using a single Gaussian per class, and omitting confidence-weighting each degrade performance (Menabue et al., 2024).

Those earlier ablations do not constitute direct evidence about CRISP’s ARSP module, but they illuminate the design logic behind residual semantic prompting. They suggest that residual prompt injection can function as a mechanism for balancing stability and plasticity, especially when prompt selection is tied to semantically meaningful representations rather than learned task-specific keys. A plausible implication is that ARSP’s adaptive matching and semantic consistency loss play an analogous stabilizing role for decoder query space in continual video instance segmentation.

7. Related methods and conceptual boundaries

ARSP in CRISP is situated among prompt-learning and segmentation architectures explicitly named in the exposition. The prompt generator is said to follow the CoOp paradigm, and the overall decoder context is Mask2Former. The references listed alongside the ARSP exposition include Bowen Cheng et al., “Masked-attention Mask Transformer for Universal Image Segmentation,” Kaiyang Zhou et al., “Learning to Prompt for Vision–LLMs,” Seunghun Lee et al., “Context-Aware Video Instance Segmentation,” and Kim et al., “ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning” (Liu et al., 14 Aug 2025).

The term “adaptive” in ARSP refers concretely to the query–prompt matching rule $j$5 rather than to dynamic prompt generation conditioned on each query. Likewise, the term “residual” refers concretely to adding the matched prompts into the self-attention value stream, not to residual adapters distributed throughout the backbone (Liu et al., 14 Aug 2025). Clarifying these boundaries helps avoid conflation with neighboring prompt-based methods.

Another boundary concerns memory mechanisms. The CRISP exposition states that ARSP uses no rehearsal and no negative-sample mining beyond standard contrastive log-sum-exp, whereas the earlier semantic residual prompting method relies on Mixture-of-Gaussians replay for both CLIP features and ViT CLS features (Menabue et al., 2024). Consequently, although both methods address catastrophic forgetting, they do so through distinct optimization regimes.

Taken together, these distinctions indicate that ARSP is best characterized as a semantic prompt injection mechanism for continual video instance segmentation that inherits the broader intuition of semantic residual transfer while instantiating it with decoder-level query matching, contrastive semantic consistency, and incremental-task optimization specific to CRISP (Liu et al., 14 Aug 2025).