OVRSISBenchV2: Remote Sensing Segmentation Benchmark

• OVRSISBenchV2 is an open-vocabulary remote sensing segmentation benchmark that unifies fragmented datasets and standardizes evaluation protocols.
• It integrates OVRSIS95K—with about 95K image–mask pairs—and ten downstream datasets to assess transfer across 128 semantic categories.
• The baseline model, Pi-Seg, uses CLIP-inspired perturbation modules to refine text-image alignment, achieving robust mIoU and mACC scores.

Searching arXiv for the benchmark paper and closely related OVRSIS work to ground the article and citations. OVRSISBenchV2 is a large-scale, application-oriented benchmark for open-vocabulary remote sensing image segmentation (OVRSIS), introduced in “Towards Realistic Open-Vocabulary Remote Sensing Segmentation: Benchmark and Baseline” (Li et al., 17 Apr 2026). OVRSIS seeks to segment arbitrary geospatial categories specified by text, without retraining for a fixed label set. The benchmark was designed to address fragmented datasets, limited training diversity, and the absence of evaluation protocols that reflect realistic geospatial application demands. It combines a new training foundation, OVRSIS95K, with ten downstream datasets, yielding 170,036 images and 128 categories, and extends evaluation beyond standard open-vocabulary segmentation to building extraction, road extraction, and flood detection (Li et al., 17 Apr 2026).

1. Scope and motivation

OVRSISBenchV2 was proposed in response to limitations identified in OVRSISBenchV1. The earlier benchmark unified several remote sensing datasets under a single cross-dataset protocol and revealed stark domain gaps for natural-image open-vocabulary segmentation methods, but it trained on a single dataset with limited scale and scene diversity. The V2 benchmark was therefore intended to assess more realistic open-world transfer rather than only diagnose domain shift (Li et al., 17 Apr 2026).

The underlying problem setting is technically demanding because remote sensing data are distributed across small, isolated datasets with heterogeneous sensors, resolutions, and taxonomies. Training diversity is limited, which exacerbates class imbalance and overfitting, while real deployments require generalization to unseen categories and domains, including disaster scenes, UAV versus satellite viewpoints, and varied object orientations. OVRSISBenchV2 addresses these conditions by broadening both the training substrate and the evaluation regime (Li et al., 17 Apr 2026).

A central feature of the benchmark is that it is neither restricted to semantic segmentation in the narrow sense nor organized around a closed label set. Instead, it adopts a unified taxonomy and consistent text prompting so that models are trained on OVRSIS95K and then evaluated under zero-shot or open-vocabulary settings on heterogeneous downstream datasets. This suggests a shift from dataset-specific benchmarking toward a more deployment-oriented stress test for transfer across platforms, resolutions, and semantic inventories.

2. OVRSIS95K and benchmark composition

OVRSISBenchV2 is built on OVRSIS95K, a balanced dataset of about 95K image–mask pairs with 35 common semantic categories spanning five representative remote sensing scene domains: town, industrial, forest, waterfront, and wasteland (Li et al., 17 Apr 2026). In the benchmark description, the training set is listed as 94,620 images in Table I, while the overall training foundation is described as about 95K image–mask pairs.

OVRSIS95K was constructed with a scalable, semi-automated pipeline comprising three stages: caption-driven category generation, label harmonization against a unified taxonomy with filtering and matching, and mask proposal extraction with human verification. Representative categories include airplane, airport, bridge, dam, ground track field, harbor, overpass, ship, stadium, storage tank, tennis court, train station, vehicle, windmill, roundabout, container crane, helipad, building, road, water, tree, grass, bareland, rangeland, developed space, agriculture land, intersection, and background, for a total of 35 categories (Li et al., 17 Apr 2026).

The full benchmark aggregates OVRSIS95K with ten downstream datasets for evaluation, totaling 170,036 images and 128 categories. The testing sets collectively contain 75,416 images across OVRSIS datasets—DLRSD, FLAIR, iSAID, LoveDA, OpenEarthMap, Potsdam, UAVid, UDD5, Vaihingen, and VDD—and across downstream tasks: WHU Aerial, WHU Sat II, Inria, and xBDpre for buildings; CHN6-CUG, DeepGlobe, Massachusetts, and SpaceNet for roads; and WBS-SI for flood detection (Li et al., 17 Apr 2026).

The benchmark also covers heterogeneous sensing platforms, specifically satellite and UAV imagery, varied spatial resolutions, and broad scene distributions. This composition is explicitly intended to expand scene diversity, semantic coverage, and evaluation difficulty relative to the earlier version (Li et al., 17 Apr 2026).

3. Taxonomy, splits, and evaluation protocol

OVRSISBenchV2 adopts a unified taxonomy that harmonizes labels across datasets. Base classes are those present in OVRSIS95K, whereas novel classes appear only in downstream tests. Dataset-specific labels are mapped onto the unified label set, and text encoding uses a consistent CLIP-style template: “a photo of {class}” (Li et al., 17 Apr 2026).

The evaluation protocol trains models on OVRSIS95K and tests them on downstream datasets under zero-shot or open-vocabulary settings. It includes in-domain versus out-of-domain evaluation across shared and unshared categories, domains such as satellite and UAV, and differing resolutions. In some analyses, the benchmark separately reports seen and unseen subsets. The paper gives UAVid as an example, with Building, Road, and Tree treated as seen classes and Low vegetation, Car, and Human treated as unseen classes (Li et al., 17 Apr 2026).

For multi-class semantic segmentation, class-wise intersection-over-union and accuracy are computed and then averaged over $K$ classes to obtain mean IoU and mean ACC:

$IoU_k = \frac{TP_k}{TP_k + FP_k + FN_k}$

$ACC_k = \frac{TP_k}{TP_k + FN_k}$

$$mIoU = \frac{1}{K}\sum_{k=1}^{K} IoU_k,\qquad mACC = \frac{1}{K}\sum_{k=1}^{K} ACC_k$$

For binary tasks such as building extraction, road extraction, and flood detection, the benchmark also gives standard definitions for precision, recall, F1 score, and Dice coefficient:

$P = \frac{TP}{TP + FP}$

$R = \frac{TP}{TP + FN}$

$F1 = \frac{2PR}{P + R}$

$Dice = \frac{2TP}{2TP + FP + FN}$

The reported downstream task results use mIoU and mACC, while F1 and Dice are included for completeness (Li et al., 17 Apr 2026).

A notable design point is that each test set retains substantial unseen semantics while preserving shared classes for meaningful cross-dataset transfer. The paper states that this makes the protocol “neither trivially overlapped nor excessively disjoint” (Li et al., 17 Apr 2026). A plausible implication is that the benchmark aims to measure transfer under controlled semantic partial overlap rather than under either near-identity or total mismatch.

4. Pi-Seg baseline

The benchmark introduces Pi-Seg as a baseline for OVRSIS. Pi-Seg builds on a CLIP-like vision–language backbone with a frozen image encoder producing dense visual features $V \in \mathbb{R}^{H \times W \times C}$ and a frozen text encoder producing category embeddings $T \in \mathbb{R}^{N \times C}$ from prompts of the form “a photo of {class}” (Li et al., 17 Apr 2026).

Its distinctive element is a positive-incentive noise mechanism implemented through two lightweight perturbation modules: Text-SPM and Image-SPM. Text-SPM injects a learnable stochastic residual into the text embeddings:

$IoU_k = \frac{TP_k}{TP_k + FP_k + FN_k}$0

$IoU_k = \frac{TP_k}{TP_k + FP_k + FN_k}$1

with $IoU_k = \frac{TP_k}{TP_k + FP_k + FN_k}$2, Gaussian by default, and with Laplace, Uniform, and Student-$IoU_k = \frac{TP_k}{TP_k + FP_k + FN_k}$3 also used. The parameters $IoU_k = \frac{TP_k}{TP_k + FP_k + FN_k}$4 are learnable, and the initialization scale of the text noise is controlled by $IoU_k = \frac{TP_k}{TP_k + FP_k + FN_k}$5, with $IoU_k = \frac{TP_k}{TP_k + FP_k + FN_k}$6 reported as yielding strong results (Li et al., 17 Apr 2026).

Image-SPM generates spatially varying perturbations conditioned on text so as not to break remote sensing spatial structures. A text-guided cross-attention compresses text cues to dimension $IoU_k = \frac{TP_k}{TP_k + FP_k + FN_k}$7, with $IoU_k = \frac{TP_k}{TP_k + FP_k + FN_k}$8 reported as effective, and predicts per-pixel perturbation parameters $IoU_k = \frac{TP_k}{TP_k + FP_k + FN_k}$9. The perturbed visual features are

$ACC_k = \frac{TP_k}{TP_k + FN_k}$0

$ACC_k = \frac{TP_k}{TP_k + FN_k}$1

with $ACC_k = \frac{TP_k}{TP_k + FN_k}$2 for each pixel $ACC_k = \frac{TP_k}{TP_k + FN_k}$3 (Li et al., 17 Apr 2026).

Using the perturbed features, Pi-Seg constructs a dense cosine-similarity cost map:

$ACC_k = \frac{TP_k}{TP_k + FN_k}$4

This map is refined through spatial and class-wise aggregation, denoted $ACC_k = \frac{TP_k}{TP_k + FN_k}$5, and then decoded with a lightweight upsampling head into segmentation logits. Training uses a standard per-pixel cross-entropy loss on the softmax of the refined cost map:

$ACC_k = \frac{TP_k}{TP_k + FN_k}$6

No extra contrastive or CLIP-style alignment loss is required; the paper attributes the positive-incentive effect to perturbation learning coupled with aggregation (Li et al., 17 Apr 2026).

5. Empirical characteristics and benchmark difficulty

The paper reports that OVRSISBenchV2 is significantly more challenging than OVRSISBenchV1. It attributes this to expanded scale, broader semantic coverage, multi-platform satellite/UAV evaluation, multi-resolution testing, and more extensive unseen-category coverage. It also notes that training trajectories in V2 are noisier and more conservative than in V1, which is presented as evidence of harder optimization (Li et al., 17 Apr 2026).

For standard OVRSIS across ten test datasets, Pi-Seg achieves the best mean results under both backbone scales reported. With ViT-B, it records 39.48 m-mIoU and 56.90 m-mACC, compared with RSKT-Seg at 35.62 and 53.09 and CAT-Seg at 37.88 and 55.72. Representative ViT-B dataset scores include DLRSD 39.69 and 58.33, LoveDA 45.92 and 63.56, UAVid 40.22 and 57.76, UDD5+Vaihingen 59.61 and 77.62, and VDD 31.70 and 46.31 (Li et al., 17 Apr 2026).

With ViT-L, Pi-Seg reaches 44.40 m-mIoU and 63.16 m-mACC, compared with RSKT-Seg at 40.10 and 59.94 and CAT-Seg at 42.49 and 61.01. Representative ViT-L scores include LoveDA 43.26 and 66.74, OpenEarthMap 36.23 and 57.46, UDD5+Vaihingen 59.60 and 74.74, and VDD 44.94 and 66.11 (Li et al., 17 Apr 2026).

The paper also reports seen-versus-unseen analysis on UAVid. When Building, Road, and Tree are treated as seen classes and Low vegetation, Car, and Human as unseen classes, Pi-Seg achieves 66.37 seen mIoU and 78.70 seen mACC, together with 24.30 unseen mIoU and 49.83 unseen mACC, which the authors describe as the best overall among compared baselines (Li et al., 17 Apr 2026). This suggests that the method’s gains are not limited to fitting base classes but extend to transfer on novel classes.

On OVRSISBenchV1, Pi-Seg remains competitive. Under DLRSD training, ViT-B yields 37.28 m-mIoU and 55.99 m-mACC, while ViT-L yields 44.63 and 61.85. Under iSAID training, Pi-Seg is described as competitive but not uniformly best, and the paper states that its advantages are more evident on the broader and harder V2 benchmark (Li et al., 17 Apr 2026).

6. Downstream tasks, ablations, and efficiency

A defining difference from OVRSISBenchV1 is the inclusion of downstream protocols for building extraction, road extraction, and flood detection. These tasks are framed as reflecting critical, application-driven deployment scenarios and safety-critical GIS applications (Li et al., 17 Apr 2026).

For building extraction with ViT-B, Pi-Seg reports mIoU scores of 85.88 on WHU Aerial, 66.16 on WHU Sat II, 78.61 on Inria, and 74.51 on xBDpre. For road extraction with ViT-B, it reports 62.12 on CHN6-CUG, 62.44 on DeepGlobe, 55.00 on Massachusetts, and 59.40 on SpaceNet. For flood detection with ViT-B, it reports 78.49 mIoU on WBS-SI. The mean over downstream tasks is 69.18 mIoU and 81.79 mACC for ViT-B, and ViT-L achieves 79.90 mIoU (Li et al., 17 Apr 2026).

Ablation studies indicate complementary contributions from the perturbation modules. On OVRSISBenchV2 with ViT-B, the baseline without the full perturbation scheme yields 36.73 m-mIoU and 56.36 m-mACC. Image-SPM only gives 38.04 and 57.00. Text-SPM only gives 36.41 and 54.21. Using both modules yields 39.65 and 57.63, the best result in the reported comparison (Li et al., 17 Apr 2026).

Sensitivity analyses also report that for the reduction ratio $ACC_k = \frac{TP_k}{TP_k + FN_k}$7, values of 1, 2, 4, and 8 produce, respectively, 37.25/55.60, 38.91/56.72, 38.39/55.34, and 39.23/57.25 in m-mIoU/m-mACC on ViT-B, with $ACC_k = \frac{TP_k}{TP_k + FN_k}$8 described as effective. For text noise standard deviation $ACC_k = \frac{TP_k}{TP_k + FN_k}$9, values 0.005, 0.01, 0.02, and 0.05 yield 37.46/55.93, 38.61/56.44, 38.91/56.72, and 38.63/57.69, with $$mIoU = \frac{1}{K}\sum_{k=1}^{K} IoU_k,\qquad mACC = \frac{1}{K}\sum_{k=1}^{K} ACC_k$$0 giving strong results and 0.05 giving the best m-mACC (Li et al., 17 Apr 2026).

Across ten random seeds, the paper reports 39.25 ± 0.56 m-mIoU and 57.53 ± 0.29 m-mACC for ViT-B, and 44.24 ± 0.48 and 63.01 ± 0.28 for ViT-L. It also states that Pi-Seg is largely distribution-agnostic with respect to perturbation sampling. Example ViT-B results include Gaussian image plus Laplace text at 39.22/55.99, Laplace image plus Uniform text at 39.23/55.89, Uniform image plus Laplace text at 39.02/57.66, and Student-$$mIoU = \frac{1}{K}\sum_{k=1}^{K} IoU_k,\qquad mACC = \frac{1}{K}\sum_{k=1}^{K} ACC_k$$1 image plus Uniform text at 39.67/57.27. The paper notes a sweet spot for moderate heavy tails in the Student-$$mIoU = \frac{1}{K}\sum_{k=1}^{K} IoU_k,\qquad mACC = \frac{1}{K}\sum_{k=1}^{K} ACC_k$$2 case (Li et al., 17 Apr 2026).

Under identical hardware and $$mIoU = \frac{1}{K}\sum_{k=1}^{K} IoU_k,\qquad mACC = \frac{1}{K}\sum_{k=1}^{K} ACC_k$$3 input, Pi-Seg is reported as lighter than RSKT-Seg: 155.60M versus 398.07M parameters for ViT-B, and 436.68M versus 678.29M for ViT-L. Its inference GFLOPs are identical to CAT-Seg, with no added evaluation-time cost from perturbation, and far lower than RSKT-Seg sliding-window inference variants. Reported throughput is 66.89 FPS for ViT-B and 57.92 FPS for ViT-L (Li et al., 17 Apr 2026). The benchmark also reports robustness across resolution regimes, with Pi-Seg obtaining 44.50 m-mIoU and 62.93 m-mACC in high-resolution settings and 34.46 and 50.87 in low-resolution settings (Li et al., 17 Apr 2026).

7. Practical use, limitations, and research significance

The code and datasets for OVRSISBenchV2 and Pi-Seg are publicly available through the project repository and linked Hugging Face dataset pages. The recommended training setup is 40K iterations on 4 NVIDIA H100 GPUs using AdamW with base learning rate $$mIoU = \frac{1}{K}\sum_{k=1}^{K} IoU_k,\qquad mACC = \frac{1}{K}\sum_{k=1}^{K} ACC_k$$4, cosine schedule, and total batch size 8. The input resolutions are 384×384 for ViT-B/16 and 336×336 for ViT-L/14, the prompt template is “a photo of {class}”, and the default hyperparameters are text perturbation $$mIoU = \frac{1}{K}\sum_{k=1}^{K} IoU_k,\qquad mACC = \frac{1}{K}\sum_{k=1}^{K} ACC_k$$5 and image reduction ratio $$mIoU = \frac{1}{K}\sum_{k=1}^{K} IoU_k,\qquad mACC = \frac{1}{K}\sum_{k=1}^{K} ACC_k$$6 (Li et al., 17 Apr 2026).

The benchmark and baseline are presented with several limitations. Fine-grained semantics remain difficult: Pi-Seg can confuse visually similar categories such as ship, airplane, and vehicle when short prompts lead to coarse prototypes. Rotation invariance is not explicitly modeled, since robustness is improved through feature-space smoothing rather than rotation-equivariant design. Domain and sensor coverage are broader than in earlier benchmarks but still incomplete, with SAR, multispectral data, and extreme scenes identified as open areas. Although OVRSIS95K reduces imbalance, rare classes may remain underrepresented. Complex backgrounds and small objects can also create alignment brittleness, motivating stronger context modeling and multi-scale reasoning (Li et al., 17 Apr 2026).

Taken together, OVRSISBenchV2 defines a more realistic evaluation regime for open-vocabulary remote sensing segmentation by combining a stronger training foundation, unified open-vocabulary protocols, and application-oriented downstream tasks (Li et al., 17 Apr 2026). Pi-Seg, as its baseline, demonstrates that semantically guided perturbations can broaden the alignment space between visual and text features while remaining efficient and easy to train. A plausible implication is that OVRSISBenchV2 may serve not only as a benchmark for model ranking but also as a controlled testbed for studying transfer under heterogeneous remote sensing conditions, partial semantic overlap, and deployment-oriented task variation.