RSRefSeg 2: Decoupled Remote Sensing Segmentation

• The paper proposes a decoupled two-stage framework that separates coarse localization from fine segmentation to address semantic ambiguity in remote sensing.
• It integrates CLIP-family cross-modal alignment with SAM-family promptable segmentation using a cascaded second-order prompter for refined mask generation.
• The method achieves state-of-the-art performance on benchmarks like RefSegRS, RRSIS-D, and RISBench, improving gIoU and segmentation precision.

RSRefSeg 2 is a foundation-model-based method for referring remote sensing image segmentation that reformulates the conventional dual-modal encoding, cross-modal interaction, and pixel decoding pipeline into a decoupled dual-stage framework: coarse localization followed by fine segmentation. It combines CLIP-family cross-modal alignment with SAM-family promptable segmentation, using CLIP to activate target features in a pre-aligned semantic space and a cascaded second-order prompter to generate localization prompts that guide SAM toward pixel-level mask refinement. The method is positioned as a response to semantic ambiguity, error propagation, and weak interpretability in earlier coupled RRSIS systems, and it reports state-of-the-art results on RefSegRS, RRSIS-D, and RISBench (Chen et al., 8 Jul 2025).

1. Task setting and conceptual motivation

Referring Remote Sensing Image Segmentation (RRSIS) takes a remote sensing image and a natural-language referring expression and predicts the binary mask of the referred region or object. The task was introduced for remote sensing as a counterpart of referring image segmentation in natural images, with RefSegRS emphasizing that top-down imagery contains small and scattered targets, large scale variation, dense clutter, and strong dependence on spatial relations such as “in the parking area” or “along the road” (Yuan et al., 2023). RSRefSeg 2 adopts this same problem setting, but argues that the dominant architectural paradigm in RRSIS remains overly coupled (Chen et al., 8 Jul 2025).

The paper’s central diagnosis is that most prior remote-sensing referring segmentation methods are built around a three-stage workflow comprising dual-modal encoding, cross-modal interaction, and pixel decoding. In that formulation, target localization and boundary delineation are solved together inside a single multimodal feature stream. RSRefSeg 2 argues that this coupling conflates two distinct subproblems: determining where the text-referred target is, and determining the precise extent of its mask. According to the paper, such coupling amplifies error propagation under semantic ambiguity and restricts both interpretability and generalizability (Chen et al., 8 Jul 2025).

This argument places RSRefSeg 2 in direct continuity with earlier foundation-model RRSIS systems while also differentiating it from them. RSRefSeg already used CLIP/SigLIP and SAM with an AttnPrompter that translated text-conditioned visual activations into prompts (Chen et al., 12 Jan 2025). RS2-SAM 2 instead adapted SAM 2 through a union encoder, a bidirectional hierarchical fusion module, a mask prompt generator, and a text-guided boundary loss (Rong et al., 10 Mar 2025). RSRefSeg 2 keeps the CLIP-to-SAM collaboration premise, but explicitly rejects the standard coupled fusion-decoding formulation in favor of a decoupled localization-to-segmentation pipeline (Chen et al., 8 Jul 2025).

The motivation is especially acute in remote sensing because the referred target may be tiny, visually similar to distractors, embedded in cluttered backgrounds, or specified through multi-entity spatial descriptions such as “the vehicle on the right of the airplane on the upper left.” The paper’s analysis of CLIP misactivation treats these cases as failures of semantic selection rather than failures of pixel-level mask generation. This suggests that a dedicated localization stage can isolate the text-grounding problem before handing a cleaner prompt to a strong segmentation prior (Chen et al., 8 Jul 2025).

2. Decoupled architecture and semantic transmission

RSRefSeg 2 is organized around three modules: a dual-modality aligned semantic feature encoder, a cascaded second-order referring prompter, and a prompt-guided refinement mask generator (Chen et al., 8 Jul 2025). The overall computation is written as

$$\begin{aligned} \mathcal{F} &= \Phi_{\text{enc}}(\mathcal{I}_1, \mathcal{T}) \ \mathcal{P} &= \Phi_{\text{prompter}}(\mathcal{F}) \ \hat{\mathcal{M}} &= \Phi_{\text{dec}}(\mathcal{I}_2, \mathcal{P}) . \end{aligned}$$

Here, $\mathcal{I}_1$ is a lower-resolution image used for localization, $\mathcal{I}_2$ is a higher-resolution image used for segmentation, $\mathcal{T}$ is the referring expression, $\mathcal{F}=\{v,t\}$ are visual and textual features, $\mathcal{P}$ are prompts, and $\hat{\mathcal{M}}$ is the final mask (Chen et al., 8 Jul 2025).

The encoder stage uses a CLIP-family model; in the reported implementation the default visual-language backbone is siglip2-so400m-patch16-512 (Chen et al., 8 Jul 2025). Visual and textual encoding are defined as

$$\begin{aligned} t &= \Phi_{\text{enc-t}}(\mathcal{T}, W_t, W_t^\star) \ v &= \Phi_{\text{enc-v}}(\mathcal{I}_1, W_v, W_v^\star) \ W^\star &= AB^T , \end{aligned}$$

where $W^\star$ denotes LoRA low-rank adaptation. The original pooling layer of CLIP is removed so that the visual stream remains spatially dense, producing $v \in \mathbb{R}^{h_1 \times w_1 \times d_1}$. The text stream is split into word-level features $\mathcal{I}_1$0 and a sentence-level feature $\mathcal{I}_1$1 derived from the EOS token (Chen et al., 8 Jul 2025).

The key architectural notion is the paper’s “semantic transmission pipeline.” Language semantics are first encoded in CLIP/SigLIP space, then converted into activated visual features by the prompter, and finally transmitted to SAM as sparse and dense prompts. This differs from a standard multimodal decoder: the CLIP-family model is responsible for coarse cross-modal grounding, while the SAM-family model is responsible for prompt-conditioned mask generation. A plausible implication is that interpretability improves because localization quality can be examined separately from segmentation quality; the paper explicitly illustrates cases of good localization with poor segmentation and poor localization with plausible segmentation (Chen et al., 8 Jul 2025).

3. Cascaded second-order referring prompter

The cascaded second-order referring prompter is the distinctive mechanism that addresses CLIP misactivation in complex descriptions (Chen et al., 8 Jul 2025). The paper argues that native CLIP often over-activates dominant nouns or multiple same-category entities, while under-weighting relational and positional tokens. To mitigate this, the prompter decomposes the text embedding into complementary semantic subspaces and performs two cascaded rounds of text-visual interaction:

$\mathcal{I}_1$2

The first stage, Referring Semantic Decomposition, produces two complementary semantic subspaces rather than a single sentence representation. For one branch, the paper gives

$\mathcal{I}_1$3

with $\mathcal{I}_1$4 computed symmetrically. The purpose is to prevent semantic collapse and enable what the paper calls implicit cascaded reasoning (Chen et al., 8 Jul 2025).

The second stage, Text-Visual Interaction, uses these decomposed subspaces to refine the visual activation progressively. For the first interaction,

$\mathcal{I}_1$5

A second interaction repeats the same logic using $\mathcal{I}_1$6 and the already refined visual feature $\mathcal{I}_1$7 (Chen et al., 8 Jul 2025). This is the sense in which the prompter is “second-order”: the second semantic subspace does not see the original visual map alone, but a first-pass activated map.

Prompt generation is hybrid. Sparse prompts are produced by learnable sparse prompt queries that attend first to refined text and then to refined visual features:

$\mathcal{I}_1$8

Dense prompts are produced as a text-filtered activation map:

$\mathcal{I}_1$9

The dense prompt therefore acts as a coarse localization map, while the sparse prompt provides SAM-compatible sparse positional information (Chen et al., 8 Jul 2025).

4. SAM-guided fine segmentation and optimization

The fine segmentation stage uses SAM 2; the default implementation is sam2.1-hiera-large, with ablations over sam2.1-t, sam2.1-s, sam2.1-b, and sam2.1-l (Chen et al., 8 Jul 2025). The decoder stage is written as

$\mathcal{I}_2$0

The dense CLIP-derived prompt is encoded and added to SAM image features, while the sparse prompt is concatenated with learnable mask-filtering and IoU tokens (Chen et al., 8 Jul 2025). SAM outputs four masks, but only the first mask is selected as the final prediction.

The optimization is parameter-efficient rather than fully end-to-end. The CLIP/SigLIP encoders and the SAM image encoder are mostly frozen and adapted with LoRA; the SAM mask decoder is fine-tuned, and the cascaded second-order prompter is trained from scratch (Chen et al., 8 Jul 2025). The total objective is

$\mathcal{I}_2$1

with

$\mathcal{I}_2$2

$\mathcal{I}_2$3

and

$\mathcal{I}_2$4

The orthogonal term enforces complementarity between the two semantic subspaces. The alignment term is formulated with dense-prompt, spatial, and sample-wise contrastive components (Chen et al., 8 Jul 2025).

An important implementation detail is that the final reported configuration sets $\mathcal{I}_2$5 and $\mathcal{I}_2$6, while retaining $\mathcal{I}_2$7 together with $\mathcal{I}_2$8 and $\mathcal{I}_2$9 (Chen et al., 8 Jul 2025). This means the final training recipe does not actually activate every auxiliary loss written in the general formulation. The optimizer is AdamW with learning rate $\mathcal{T}$0, batch size 64, 300 epochs, cosine annealing with linear warm-up, BF16 precision, no data augmentation, CLIP localization input at $\mathcal{T}$1, and SAM segmentation input at $\mathcal{T}$2 (Chen et al., 8 Jul 2025).

5. Benchmarks, datasets, and reported performance

RSRefSeg 2 is evaluated on three remote-sensing referring segmentation benchmarks (Chen et al., 8 Jul 2025). RefSegRS, introduced for RRSIS on top of SkyScapes, contains 4,420 image-expression-mask triplets at $\mathcal{T}$3 resolution and was originally motivated by small and scattered foregrounds in remote sensing imagery (Yuan et al., 2023). RRSIS-D contains 17,402 triplets at $\mathcal{T}$4 resolution with 20 categories and large scale variation (Chen et al., 12 Jan 2025). RISBench, built from VRSBench, contains 52,472 triplets, 26 categories, average expression length 14.31 words, and vocabulary size 4,431 (Chen et al., 8 Jul 2025).

The paper reports gIoU, cIoU, and Precision@X. It emphasizes gIoU because cIoU is biased toward larger objects (Chen et al., 8 Jul 2025).

Dataset	RSRefSeg 2 result	Reported comparison
RefSegRS	cIoU 81.24, gIoU 77.39	+3.49 gIoU over RS2-SAM 2
RRSIS-D	cIoU 79.45, gIoU 69.17	+2.45 gIoU over RS2-SAM 2
RISBench	cIoU 74.77, gIoU 72.57	+4.04 gIoU over LSCF

On RefSegRS, RSRefSeg 2 reaches [email protected]/0.6/0.7/0.8/0.9 of 88.22/82.99/73.97/60.92/34.40, with cIoU 81.24 and gIoU 77.39. On RRSIS-D, it achieves 80.23/75.78/65.41/50.65/31.05, with cIoU 79.45 and gIoU 69.17. On RISBench, it reaches 79.08/75.36/70.62/62.75/48.75, with cIoU 74.77 and gIoU 72.57 (Chen et al., 8 Jul 2025).

The RRSIS-D category analysis is also specific. RSRefSeg 2 reports category-wise mean IoU of 72.06, performs best on 15 of 20 categories, and is especially strong on golf field, baseball field, stadium, chimney, bridge, vehicle, windmill, and airplane. It underperforms FIANet on dam and harbor, which the paper associates with ambiguous boundaries (Chen et al., 8 Jul 2025). This distribution of gains is consistent with the method’s stated strengths in complex semantic interpretation, small-object segmentation, and high-IoU mask quality.

6. Ablations, interpretability, and limitations

The ablations concentrate on the cascaded prompter, prompt construction, LoRA strategy, and backbone choices (Chen et al., 8 Jul 2025). A central result is the effect of decoupled prompting on RRSIS-D: without the cascaded prompter and without dense prompt, the model reports gIoU 60.06; with the cascaded prompter but no dense prompt, gIoU rises to 64.13; with both cascaded prompter and dense prompt, it reaches 66.01. This directly supports the claim that the decoupled CLIP-to-SAM pipeline depends on both the second-order prompter and hybrid prompting.

The best structural settings reported in the ablations are $\mathcal{T}$5, $\mathcal{T}$6, $\mathcal{T}$7, semantic query count $\mathcal{T}$8, and sparse prompt count $\mathcal{T}$9 (Chen et al., 8 Jul 2025). LoRA also matters: vision-side adaptation contributes more than text-side adaptation, and the best small-model ablation uses clip-t rank 16, clip-v rank 16, and sam-v rank 32. The best backbone combination in the main ablation is siglip2-p16-512 + sam2.1-l, which yields gIoU 69.17 on RRSIS-D (Chen et al., 8 Jul 2025).

The loss ablation is unusually revealing. Orthogonal subspace regularization helps, dense prompt supervision helps little, spatial alignment supervision helps little, and sample-wise contrastive alignment helps substantially. The best ablation row uses CE + orthogonal loss + NCE-$\mathcal{F}=\{v,t\}$0-$\mathcal{F}=\{v,t\}$1 + Dice, reaching gIoU 67.06 (Chen et al., 8 Jul 2025). This suggests that the method’s main gains come less from dense prompt map supervision and more from semantic decomposition, sample-wise alignment, and prompt-mediated transmission into SAM.

Interpretability is one of the paper’s stated advantages. Because localization and segmentation are decoupled, dense prompt visualizations expose whether a failure originates in CLIP-stage semantic grounding or in SAM-stage boundary refinement. The qualitative analysis explicitly presents good-localization/good-segmentation, good-localization/poor-segmentation, and poor-localization/plausible-segmentation cases (Chen et al., 8 Jul 2025). This is a structural distinction from earlier coupled RRSIS pipelines, where such diagnosis is less direct.

The authors also state several limitations. Complex logical or compound semantics remain difficult, and they give the example “a bright airplane located at the bottom edge of the image with its wings extending to the right.” Ambiguous-boundary categories such as dam and harbor remain challenging because SAM’s priors are largely shaped by natural-image object boundaries (Chen et al., 8 Jul 2025). The paper does not provide explicit inference-speed or FLOPs tables. A plausible implication is that deployment cost remains nontrivial because the framework uses large foundation models and a $\mathcal{F}=\{v,t\}$2 segmentation input, even though the localization stage runs at lower resolution.

In the broader development of remote-sensing referring segmentation, RSRefSeg 2 can therefore be understood as a decoupled CLIP-to-SAM system that retains the foundation-model collaboration idea of RSRefSeg (Chen et al., 12 Jan 2025), differs from the coupled fusion-oriented customization of RS2-SAM 2 (Rong et al., 10 Mar 2025), and operationalizes a more explicit separation between semantic localization and boundary delineation. Its main encyclopedic significance lies in turning prompt-mediated collaboration between a vision-language encoder and a segmentation foundation model into the central architectural principle of RRSIS (Chen et al., 8 Jul 2025).