---
title: 'ARSP: Adaptive Residual Semantic Prompt in CRISP'
url: https://www.emergentmind.com/topics/adaptive-residual-semantic-prompt-arsp
type: topic
---

# ARSP: Adaptive Residual Semantic Prompt in CRISP

Adaptive Residual Semantic Prompt (ARSP) is a prompt-tuning module for continual video instance segmentation that injects class-level semantic knowledge into the Transformer decoder of Mask2Former. In the formulation introduced within CRISP, ARSP is a one-level prompt-tuning sub-module composed of a residual semantic prompt generator, an adaptive query–prompt matching stage, and hierarchical prompt injection into decoder self-attention. Its stated purpose is to address category-wise confusion during incremental training by constructing a learnable semantic residual prompt pool generated by category text and using an adjustive query-prompt matching mechanism to map current-task object queries to semantic residual prompts [2508.10432]. A closely related antecedent is the two-level prompt-tuning scheme of "Semantic Residual Prompts for Continual Learning," which also uses residual semantic transfer, but for class-incremental continual learning in a frozen Vision Transformer rather than for video instance segmentation [2403.06870].

## 1. Definition and research context

In CRISP, ARSP is defined as a one-level prompt-tuning mechanism tailored to the Transformer decoder of Mask2Former for video. The module has three named components: a **Residual Semantic Prompt Generator**, **Adaptive Query–Prompt Matching**, and **Hierarchical Prompt Injection**. The prompt generator learns a small pool of class-specific residual vectors $\{p_0,\ldots,p_{c_t-1}\}$ via a lightweight text-encoder head built on CLIP; the matching stage compares current-task object queries $Q_t \in \mathbb{R}^{N_q^t \times d}$ against the prompt pool using cosine similarity; and the injection stage adds the matched prompts into the value stream of decoder self-attention [2508.10432].

This formulation places ARSP within the broader family of prompt-based continual learning methods, but its role is narrower and more specific than the earlier semantic residual prompting framework of 2024. In "Semantic Residual Prompts for Continual Learning," prompt tuning is organized as a two-level adaptation mechanism for class-incremental continual learning: first-level CLIP text prompts produce stable class prototypes, and second-level prompts provide per-layer residuals that adapt a frozen ViT [2403.06870]. By contrast, the ARSP module in CRISP is explicitly described as one-level and decoder-centric.

A common misconception is to treat ARSP as a generic synonym for semantic residual prompting. The published descriptions distinguish them. The 2024 work studies **Semantic Residual Prompts** for continual learning with two prompt pools and generative replay, whereas the 2025 work names **Adaptive Residual Semantic Prompt** as a module inside CRISP for continual video instance segmentation, with adaptive query–prompt matching and a semantic consistency loss [2403.06870].

## 2. Architectural composition

ARSP comprises three main blocks.

First, the **Residual Semantic Prompt Generator** learns the prompt pool from a set of learnable token sequences $X \in \mathbb{R}^{c_t \times d}$. A small MLP or adapter built on the frozen CLIP text encoder, denoted $G_\theta$, maps these tokens to the prompt pool:
$$
P_t = G_\theta(X) \in \mathbb{R}^{c_t \times d}.
$$
Each row $p_j$ is the residual semantic prompt for class $j$ [2508.10432].

Second, **Adaptive Query–Prompt Matching** compares the object queries of the current task with the prompt pool. Given $Q_t \in \mathbb{R}^{N_q^t \times d}$, ARSP forms a cosine-similarity matrix
$$
S_{i,j} =
\frac{q_i \cdot p_j}{\|q_i\|_2 \|p_j\|_2},
$$
then selects, for each query $i$, the single best-matching class prompt index
$$
a_i = \arg\max_{0 \le j < c_t} S_{i,j}.
$$
The matched prompt vectors are collected as
$$
P_m = [p_{a_1}, p_{a_2}, \ldots, p_{a_{N_q^t}}] \in \mathbb{R}^{N_q^t \times d}.
$$

Third, **Hierarchical Prompt Injection** inserts the matched prompts into each decoder layer’s self-attention value stream:
$$
O^l = \mathrm{Softmax}\!\left(\frac{Q^l(K^l)^T}{\sqrt{d_k}}\right)(V^l + P_m^l).
$$
The exposition states that injection occurs at every decoder layer’s self-attention value stream, and that decoder depth is $L=6$ by default in Mask2Former [2508.10432].

These blocks together define ARSP as a semantic modulation mechanism over decoder queries rather than as a classifier head or retrieval-only module. A plausible implication is that the semantic prompts act as structured residuals aligned to query embeddings, rather than as static class tokens.

## 3. Mathematical formulation

The prompt-pool generation step follows the CoOp paradigm. Let $c_t$ denote the number of classes in current task $t$ and $d$ the embedding dimension of the CLIP text encoder. ARSP defines a learnable token matrix
$$
X \in \mathbb{R}^{c_t \times d}
$$
and computes
$$
P_t = G_\theta(X) \in \mathbb{R}^{c_t \times d}.
$$
The exposition states that, in effect, $G_\theta$ learns to map raw token embeddings into class prototypes in the same space as the object queries [2508.10432].

The query–prompt matching step is entirely similarity-based. For each query $q_i$ and prompt $p_j$, cosine similarity is used:
$$
S_{i,j} =
\frac{q_i \cdot p_j}{\|q_i\|_2 \|p_j\|_2}.
$$
The matched prompt index is then
$$
a_i = \arg\max_{0 \le j < c_t} S_{i,j},
$$
and the matched prompt set is
$$
P_m = [p_{a_1}, p_{a_2}, \ldots, p_{a_{N_q^t}}].
$$

The semantic consistency term introduces an explicit contrastive objective. With indicator
$$
I_{i,j} =
\begin{cases}
0 & \text{if } j = a_i,\\
1 & \text{otherwise,}
\end{cases}
$$
the loss is
$$
\mathcal{L}_{ISC}
= \frac{1}{N_q^t}
\sum_{i=1}^{N_q^t}
\log\left[
1 +
\frac{\sum_{j=1}^{c_t} I_{i,j}\exp(S_{i,j})}
{\exp(S_{i,a_i})}
\right].
$$
The positive pair is $(q_i, p_{a_i})$ and the negative pairs are $(q_i, p_j)$ for $j \ne a_i$ [2508.10432].

This formalization is narrower than that of the 2024 continual-learning framework, where residual transfer is defined at the level of frozen ViT blocks as $R[\ell] = \mathrm{sim}_{c^*} \cdot Q_{c^*}[\ell]$, inserted before the MLP residual path, and where two training stages separately optimize first-level text prompts and second-level residual prompts [2403.06870]. The distinction matters because the two systems use different carriers of semantic information: decoder self-attention values in CRISP, and per-layer ViT residuals in the earlier class-incremental setting.

## 4. Training procedure and optimization

Within the CRISP training loop, ARSP is invoked during each incremental-task training epoch. The documented sequence is:

1. Extract multi-scale features $F = \mathrm{Backbone}(B)$.
2. Decode object queries $Q_t = \mathrm{DecoderQueries}(F)$.
3. Generate the prompt pool $P_t = G_\theta(X)$.
4. Compute similarities $S_{i,j} = \cos(q_i, p_j)$.
5. For each query, select $a_i = \arg\max_j S_{i,j}$.
6. Form the matched prompt set $P_m = [p_{a_1},\ldots,p_{a_{N_q^t}}]$.
7. Inject prompts into each decoder self-attention layer.
8. Compute the segmentation loss $\mathcal{L}_{Seg}$.
9. Compute the semantic consistency loss $\mathcal{L}_{ISC}$.
10. Compute the instance-correlation loss $\mathcal{L}_{IC}$.
11. Optimize the total loss
   $$
   \mathcal{L} = \mathcal{L}_{Seg} + \lambda_{ISC}\mathcal{L}_{ISC} + \lambda_{IC}\mathcal{L}_{IC}.
   $$
12. Back-propagate only into $\{G_\theta, X, \text{decoder-layer prompts/queries}\}$.
13. Update parameters with Adam/SGD [2508.10432].

The hyperparameters reported for ARSP inside CRISP are specific. The prompt pool size is $c_t$, equal to the number of new classes in task $t$; the embedding dimension is the CLIP text embedding size, reported as $512$ in the experiments; decoder depth is $6$ by default in Mask2Former; $\lambda_{ISC} = 3$ and $\lambda_{IC} = 3$; no explicit temperature $\tau$ is introduced in $\mathcal{L}_{ISC}$; and self-attention uses $\sqrt{d_k}$ scaling [2508.10432].

The design choices also clarify what ARSP does **not** include. The exposition states that the prompt dimension and CLIP adapter are frozen except for the small $G_\theta$ and $X$, and that there is **no rehearsal or negative-sample mining beyond standard contrastive log-sum-exp** [2508.10432]. This contrasts with the 2024 semantic residual prompting framework, where generative replay for both training stages uses a Mixture-of-Gaussians fitted on past CLIP features or past ViT CLS features, and replay losses are added to the objective [2403.06870].

## 5. Relation to semantic residual prompting in continual learning

The most direct precursor to ARSP is the 2024 work on semantic residual prompts for continual learning. That method begins from two frozen foundation models: a CLIP vision encoder and text encoder, and an ImageNet-pretrained ViT with $L$ Transformer blocks. It augments them with two pools of learnable prompts: a first-level prompt pool $P = \{p_c \in \mathbb{R}^d \mid c \in \text{seen classes}\}$, and a second-level prompt pool $Q = \{Q_c[\ell] \in \mathbb{R}^{d'} \mid c \in \text{seen classes}, \ell=1\ldots L\}$ [2403.06870].

In that earlier framework, level 1 constructs stable class prototypes
$$
w_c = E_{txt}([p_c;\,"\![CL\text{-}NAME]\!"]) \in \mathbb{R}^d,
$$
which serve as keys. Level 2 uses the input image embedding $z = E_{vis}(x)$ to score similarities
$$
\mathrm{sim}_c = \langle z, w_c \rangle,
$$
selects
$$
c^* = \arg\max_c \mathrm{sim}_c,
$$
and forms per-layer residuals
$$
R[\ell] = \mathrm{sim}_{c^*} \cdot Q_{c^*}[\ell].
$$
These residuals are injected into each ViT block before the MLP update [2403.06870].

ARSP in CRISP preserves several conceptual motifs from this earlier line of work: semantic prompting, residual-style injection, and explicit matching between learned semantic prototypes and current representations. However, the mechanisms are materially different. The 2024 method is **two-level**, uses CLIP text prototypes as stable keys, and targets a frozen ViT for class-incremental classification; the 2025 ARSP module is **one-level**, matches object queries directly to class prompts via cosine similarity, and injects the matched prompts into decoder self-attention for continual video instance segmentation [2403.06870].

This comparison helps delimit the term. ARSP is best understood not as the entire semantic residual prompting paradigm, but as a specific decoder-level realization of that paradigm in CRISP.

## 6. Empirical role, design rationale, and interpretation

For CRISP, the abstract attributes category-wise learning to ARSP: it constructs a learnable semantic residual prompt pool generated by category text and uses an adjustive query-prompt matching mechanism to build a mapping relationship between the query of the current task and the semantic residual prompt. The abstract further states that a semantic consistency loss based on contrastive learning is introduced to maintain semantic coherence between object queries and residual prompts during incremental training, and that experiments on YouTube-VIS-2019 and YouTube-VIS-2021 demonstrate that CRISP significantly outperforms existing continual segmentation methods in the long-term continual video instance segmentation task, avoiding catastrophic forgetting and effectively improving segmentation and classification performance [2508.10432].

For the earlier class-incremental continual-learning framework, the empirical pattern is reported in more detail. On nine benchmarks—Split ImageNet-R, CIFAR-100, Cars-196, CUB-200, EuroSAT, RESISC45, CropDiseases, ISIC, and ChestX—the method outperforms zero-shot CLIP and prior prompt-tuning continual-learning methods by large margins, including $+36.7$ points over CLIP on EuroSAT and $+17$ on Cars-196. The average final accuracy gain over the best rehearsal-free prior is reported as $+5.84\%$, and the gap to a joint-training upper bound as only $-3.96$. Ablations show that removing Level 2 prompts, replacing additive residuals with prefix-tuning, dropping replay or using a single Gaussian per class, and omitting confidence-weighting each degrade performance [2403.06870].

Those earlier ablations do not constitute direct evidence about CRISP’s ARSP module, but they illuminate the design logic behind residual semantic prompting. They suggest that residual prompt injection can function as a mechanism for balancing stability and plasticity, especially when prompt selection is tied to semantically meaningful representations rather than learned task-specific keys. A plausible implication is that ARSP’s adaptive matching and semantic consistency loss play an analogous stabilizing role for decoder query space in continual video instance segmentation.

## 7. Related methods and conceptual boundaries

ARSP in CRISP is situated among prompt-learning and segmentation architectures explicitly named in the exposition. The prompt generator is said to follow the CoOp paradigm, and the overall decoder context is Mask2Former. The references listed alongside the ARSP exposition include Bowen Cheng et al., “Masked-attention Mask Transformer for Universal Image Segmentation,” Kaiyang Zhou et al., “Learning to Prompt for Vision–Language Models,” Seunghun Lee et al., “Context-Aware Video Instance Segmentation,” and Kim et al., “ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning” [2508.10432].

The term “adaptive” in ARSP refers concretely to the query–prompt matching rule $a_i = \arg\max_j S_{i,j}$ rather than to dynamic prompt generation conditioned on each query. Likewise, the term “residual” refers concretely to adding the matched prompts into the self-attention value stream, not to residual adapters distributed throughout the backbone [2508.10432]. Clarifying these boundaries helps avoid conflation with neighboring prompt-based methods.

Another boundary concerns memory mechanisms. The CRISP exposition states that ARSP uses no rehearsal and no negative-sample mining beyond standard contrastive log-sum-exp, whereas the earlier semantic residual prompting method relies on Mixture-of-Gaussians replay for both CLIP features and ViT CLS features [2403.06870]. Consequently, although both methods address catastrophic forgetting, they do so through distinct optimization regimes.

Taken together, these distinctions indicate that ARSP is best characterized as a semantic prompt injection mechanism for continual video instance segmentation that inherits the broader intuition of semantic residual transfer while instantiating it with decoder-level query matching, contrastive semantic consistency, and incremental-task optimization specific to CRISP [2508.10432].

Source: https://www.emergentmind.com/topics/adaptive-residual-semantic-prompt-arsp