RSKT-Seg: Remote Sensing Segmentation

• The paper introduces RSKT-Seg, a framework that addresses rotation variance, multi-scale context, and domain gaps in remote sensing imagery.
• It integrates RS-CMA, RS-Fusion, and RS-Transfer modules to aggregate multi-directional visual features and remote sensing priors efficiently.
• Empirical results on OVRSISBench show notable improvements in mIoU and mACC over baselines, coupled with faster inference times.

Searching arXiv for the papers on arXiv and related OVRSIS context. RSKT-Seg is an open-vocabulary remote sensing image segmentation framework designed for the specific conditions of aerial and satellite imagery, where rotation variance, multi-scale context, spectral diversity, and a pronounced domain gap relative to natural-image pretraining complicate transfer from standard vision-LLMs. Introduced together with the standardized benchmark OVRSISBench, RSKT-Seg combines multi-directional cost aggregation, efficient spatial-and-semantic fusion, and remote-sensing-aware knowledge transfer to segment both seen and unseen categories from text queries without retraining at inference time (Li et al., 15 Sep 2025).

1. Problem setting and conceptual scope

Open-Vocabulary Remote Sensing Image Segmentation (OVRSIS) adapts open-vocabulary segmentation to top-down remote sensing imagery. In this setting, a model is trained on seen categories in source remote sensing datasets but must segment both seen and unseen categories at test time using category text queries. Under the cross-dataset transfer protocol, $C_{\text{test}}$ may partially overlap with $C_{\text{train}}$ but contains disjoint categories unseen during training. At inference, the model receives a set of text prompts $T=\{c_k\}$ for $k=1,\dots,N_t$ and produces pixel-wise masks for the queried categories without retraining. In RSKT-Seg, multiple prompt templates are used per category, and the fused cost map is organized as $H_f \times W_f \times N_t \times P$ with $P$ prompt templates (Li et al., 15 Sep 2025).

The task is shaped by three stated difficulties. The first is the domain gap: vision-LLMs such as CLIP, trained on natural images, struggle to generalize to remote sensing imagery with different textures, scales, and spectral characteristics. The second is rotation variance: objects such as bridges and airplanes are arbitrarily oriented in top-down imagery, so robust segmentation requires rotation-invariant cues. The third is benchmark fragmentation: inconsistent datasets and protocols impede fair comparison.

Evaluation in OVRSISBench uses mean Intersection-over-Union, mean Accuracy, and frequency-weighted IoU. For $n$ classes, with per-class true positives $TP_i$, false positives $FP_i$, and false negatives $FN_i$:

$C_{\text{train}}$0

$C_{\text{train}}$1

$C_{\text{train}}$2

These metrics are used to characterize both category-level fidelity and aggregate transfer performance.

2. OVRSISBench and the standardized evaluation protocol

OVRSISBench reformulates eight representative remote sensing segmentation datasets under an open-vocabulary setting: DLRSD, iSAID, Potsdam, Vaihingen, UAVid, UDD5, LoveDA, and VDD. The benchmark trains on DLRSD and iSAID, described as large-scale and diverse, and evaluates across all eight datasets under standardized splits and metrics. This cross-dataset transfer protocol uses partially disjoint taxonomies so that unseen categories are present during evaluation, thereby enforcing open-vocabulary generalization rather than closed-set interpolation (Li et al., 15 Sep 2025).

The benchmark also specifies the prompt and backbone protocol. Category names are encoded by the CLIP text encoder, and multiple prompt templates are used per category. For evaluation, the paper uses CLIP ViT-B/16 and CLIP ViT-L/14@336 as image and text encoders, together with an additional remote-sensing knowledge encoder, DINO ViT-B/32 pretrained on large-scale remote sensing data as rsDINO. This standardized setup enables consistent comparison across representative OVS and OVRSIS models.

A central empirical observation associated with the benchmark is that natural-image OVS models degrade markedly when applied directly to remote sensing segmentation. OVRSIS-specific methods improve upon these baselines, but the benchmark analysis reports that methods with limited remote-sensing priors still do not fully capture rotation invariance and large-scale spatial context. This motivates the design of RSKT-Seg as a framework tailored to the structural peculiarities of remote sensing imagery.

3. Architecture and algorithmic components

RSKT-Seg is organized around three modules: Multi-Directional Cost Map Aggregation (RS-CMA), Efficient Cost Map Fusion (RS-Fusion), and Remote Sensing Knowledge Transfer (RS-Transfer) (Li et al., 15 Sep 2025).

The end-to-end pipeline begins with a remote sensing image $C_{\text{train}}$3 and a set of category queries $C_{\text{train}}$4. The CLIP text encoder produces text embeddings

$C_{\text{train}}$5

The framework then constructs vision-language cost maps, fuses them through spatial and class-aware transformers, and upsamples the resulting aggregated representation to output final logits

$C_{\text{train}}$6

from which segmentation masks are obtained by $C_{\text{train}}$7 over $C_{\text{train}}$8.

RS-CMA

RS-CMA explicitly addresses rotation variance. The image is rotated by $C_{\text{train}}$9, $T=\{c_k\}$0, $T=\{c_k\}$1, and $T=\{c_k\}$2 to form rotated variants $T=\{c_k\}$3, $T=\{c_k\}$4. CLIP visual features are extracted from each rotated image, while rsDINO features are extracted from the unrotated view. For a spatial index $T=\{c_k\}$5 and class index $T=\{c_k\}$6, directional CLIP cost maps are defined by cosine similarity, and a DINO cost map is computed in the same manner:

$T=\{c_k\}$7

Aggregation fuses the four CLIP cost maps with the DINO cost map:

$T=\{c_k\}$8

and projects the result to

$T=\{c_k\}$9

The module is notable for introducing no learnable parameters in directional encoding and for using shared encoders across rotations, which permits efficient batched computation. Ablations consider mean aggregation, concatenation, and separate-path fusion; performance is reported as similar across these variants, with concatenation slightly better.

RS-Fusion

RS-Fusion jointly enhances spatial discriminability and class discriminability while reducing computational burden. It consists of a Spatial Enhancement Transformer (SET) and a Class Enhancement Transformer (CET).

SET concatenates $k=1,\dots,N_t$0 with mid-level CLIP and DINO features, applies a 2D convolution $k=1,\dots,N_t$1 to reduce spatial size from $k=1,\dots,N_t$2 to $k=1,\dots,N_t$3, and performs cross-attention with $k=1,\dots,N_t$4 as queries and the reduced features as keys and values. The formulation is

$k=1,\dots,N_t$5

$k=1,\dots,N_t$6

Because the key/value sequence length is reduced by $k=1,\dots,N_t$7, the attention score computation is reduced by a factor of $k=1,\dots,N_t$8.

CET then performs spatial average pooling and self-attention to model inter-category dependencies. Its attention takes the form

$k=1,\dots,N_t$9

$H_f \times W_f \times N_t \times P$0

A similar reduction factor of $H_f \times W_f \times N_t \times P$1 holds when average pooling uses ratio $H_f \times W_f \times N_t \times P$2. SET and CET are stacked for $H_f \times W_f \times N_t \times P$3 layers, yielding the aggregated cost map $H_f \times W_f \times N_t \times P$4.

RS-Transfer

RS-Transfer addresses low-resolution aggregation by injecting remote-sensing priors during upsampling. It concatenates $H_f \times W_f \times N_t \times P$5 with intermediate features from RemoteCLIP, CLIP, and rsDINO at specified layers and projects them through a learned module:

$H_f \times W_f \times N_t \times P$6

After $H_f \times W_f \times N_t \times P$7 upsampling-refinement layers, the method produces final logits at image resolution. The stated role of this module is feature alignment and detail recovery, particularly for boundary fidelity and domain adaptation.

4. Training and inference procedure

The implementation uses PyTorch and Detectron2. The encoders are CLIP ViT-B/16 or ViT-L/14@336 for vision and text, rsDINO ViT-B/32 as the remote sensing visual encoder, and RemoteCLIP ViT-B/16 for RS-Transfer feature injection. Feature taps are specified as $H_f \times W_f \times N_t \times P$8 for ViT-B/16, $H_f \times W_f \times N_t \times P$9 for rsDINO, $P$0 for RemoteCLIP, and $P$1 for ViT-L/14. Optimization uses AdamW with learning rate $P$2, weight decay $P$3, batch size $P$4, and hardware of $P$5 NVIDIA RTX 4090 (24 GB) (Li et al., 15 Sep 2025).

The number of RS-Fusion layers is dataset-dependent: $P$6 for DLRSD training and $P$7 for iSAID training. Upsampling uses $P$8. The training objective is a single cross-entropy segmentation loss,

$P$9

where $n$0 is the one-hot label tensor. No additional contrastive or distillation terms are used.

Rotation handling is built directly into RS-CMA. The method computes CLIP features on four rotated copies of the input to build rotation-invariant cost maps, but these rotations are not treated as additional training samples; they function instead as an efficient multi-directional encoding strategy. At inference, unseen classes are handled by supplying their text prompts, and no retraining is required.

The reported fine-tuning study indicates that the best performance is obtained with CLIP “attention” trainable and DINO frozen. The paper’s interpretation is that freezing DINO preserves remote sensing priors while allowing CLIP to adapt.

5. Empirical performance and comparative position

On OVRSISBench, RSKT-Seg is reported to outperform strong OVS baselines by $n$1 mIoU and $n$2 mACC on average, while achieving $n$3 faster inference through efficient aggregation. The benchmark compares RSKT-Seg with OVS baselines such as SCAN, SAN, SED, and Cat-Seg, and with OVRSIS methods such as OVRS and GSNet. The paper attributes the gap to explicit rotation modeling, efficient spatial/class fusion, and remote-sensing-aware feature transfer (Li et al., 15 Sep 2025).

Training setting	RSKT-Seg result	Comparison examples
DLRSD training, ViT-B	43.90 m-mIoU, 59.51 m-mACC	GSNet 40.08/55.87; OVRS 39.57/54.51; Cat-Seg 36.62/52.21
DLRSD training, ViT-L	46.21 m-mIoU, 62.69 m-mACC	GSNet 44.56/60.89; OVRS 44.39/60.24; Cat-Seg 44.23/57.58
iSAID training, ViT-B	30.03 m-mIoU, 52.73 m-mACC	GSNet 27.87/47.14; OVRS 27.59/47.05; Cat-Seg 26.82/46.45
iSAID training, ViT-L	36.76 m-mIoU, 57.91 m-mACC	GSNet 35.71/56.48; OVRS 36.18/56.05; Cat-Seg 33.11/52.80

Efficiency is a prominent part of the reported characterization. Mean inference time is 65.11 ms for RSKT-Seg, compared with 114.88 for Cat-Seg, 285.37 for OVRS, 145.83 for SCAN, 53.79 for SAN, and 60.47 for SED; the corresponding throughput is approximately 15.36 FPS. The model has 296.15M total parameters, 59.89M trainable parameters, and a training time of 7.96 ms per iteration, which the paper reports as faster than SED at 8.60, Cat-Seg at 9.60, and OVRS at 18.53.

Per-dataset and per-class results further illustrate the method’s behavior. When trained on DLRSD and evaluated on iSAID, the paper reports class-level mIoU values including ship 62.37, storage tank 82.74, and plane 83.99. It also reports improved boundary accuracy on Vaihingen classes, including tree at 46.71 mIoU. This suggests that the RS-Transfer stage and the use of remote-sensing priors contribute not only to category recognition but also to spatial delineation.

Ablation studies isolate the contribution of each major component. Adding multi-directional cost maps or a DINO cost map improves performance, and combining them yields further gains; one cited example raises DLRSD m-mIoU from 46.42 to 48.72. Fusion choices are comparable, with concatenation slightly better. The optimal number of aggregation layers is $n$4 for DLRSD and $n$5 for iSAID. The use of rsDINO, pretrained on remote sensing data, outperforms natureDINO, confirming the value of remote sensing priors.

6. Limitations, failure modes, and later benchmark context

The paper identifies several failure modes. Shadows can induce misclassification among impervious surfaces, trees, and cars. Height-related categories, especially low vegetation versus trees, remain difficult to distinguish because the model lacks depth cues. Performance on unseen classes is also sensitive to prompt quality: ambiguous or overly generic prompts can degrade results. These issues delimit the current scope of text-conditioned transfer and indicate that geometric or multimodal cues may be necessary for further progress (Li et al., 15 Sep 2025).

The stated future directions include hierarchical or category-attribute prompts, depth or DSM modalities, shadow-invariant cues, broader remote sensing datasets with richer taxonomies and spectral bands, multi-modal fusion such as RGB+NIR+DSM, more advanced rotation-equivariant networks, and adaptive prompt learning for remote sensing categories. These directions follow directly from the identified weaknesses in prompt dependence, height ambiguity, and domain variation.

A later paper situates RSKT-Seg as the earlier OVRSISBenchV1 model and characterizes it as effective but relatively heavy because it relies on multiple external encoders and sliding-window inference for high-resolution imagery. In that subsequent benchmark expansion, OVRSISBenchV2 and the Pi-Seg baseline are introduced, and Pi-Seg is reported to surpass RSKT-Seg on the expanded benchmark and on downstream tasks. In that framing, RSKT-Seg remains the V1-era method emphasizing remote-sensing-aware cost-map construction and refinement, whereas Pi-Seg is presented as the lighter baseline recommended for realistic open-world evaluation (Li et al., 17 Apr 2026).

Within the development of OVRSIS, RSKT-Seg therefore occupies a specific position: it establishes a standardized evaluation setting through OVRSISBench and proposes a rotation-aware, remote-sensing-prior-driven segmentation architecture that materially improves over direct transfers of natural-image OVS models, while also exposing the computational and benchmark-scope limitations that motivated later benchmark and model revisions (Li et al., 15 Sep 2025).