Pi-Seg: Open-Vocabulary RS Segmentation

• Pi-Seg is a compact baseline for open-vocabulary remote sensing segmentation that uses semantically guided perturbations to reshape local visual-text feature alignment.
• It employs lightweight Semantic-aware Perturbation Modules (SPMs) on both text and image features to broaden the semantic neighborhood without heavy external encoders.
• Evaluation on OVRSISBenchV2 shows Pi-Seg achieving improved mIoU and mACC across diverse datasets, demonstrating robust cross-dataset performance for real-world applications.

Searching arXiv for the Pi-Seg source paper and closely related context. Pi-Seg is a baseline for open-vocabulary remote sensing image segmentation (OVRSIS) proposed in “Towards Realistic Open-Vocabulary Remote Sensing Segmentation: Benchmark and Baseline” (Li et al., 17 Apr 2026). OVRSIS aims to segment remote sensing images into pixel-level semantic regions conditioned on arbitrary text labels provided at inference, including classes unseen during training. Within that setting, Pi-Seg is designed to improve transferability without stacking multiple heavy external encoders, instead broadening the visual-text feature neighborhood during training through a semantically guided perturbation mechanism described as “positive-incentive noise” (Li et al., 17 Apr 2026).

1. Problem setting and motivation

Pi-Seg is situated in the OVRSIS setting, where an image $I \in \mathbb{R}^{H \times W \times 3}$ and a set of textual class names $C=\{t_1,\dots,t_N\}$ are given, and the model predicts a segmentation map $S \in \mathbb{R}^{H \times W}$ with entries in $C$. A defining condition is that $C_{\text{test}}$ can be partially or largely disjoint from $C_{\text{train}}$, so the task is not restricted to closed-set semantic segmentation (Li et al., 17 Apr 2026).

The motivation for Pi-Seg follows from three difficulties identified for OVRSIS. First, prior datasets are described as fragmented and limited in training diversity, often built from single sources such as iSAID or Potsdam, with insufficient class balance and scene diversity. Second, remote sensing imagery exhibits top-down viewpoints, rotation-agnostic appearance, scale diversity, and clutter, which violate assumptions of natural-image vision-LLMs such as CLIP and make dense pixel-text alignment brittle, especially for small or thin structures. Third, realistic evaluation requires cross-dataset transfer and downstream application protocols, including building extraction, road extraction, and flood detection, which earlier benchmark settings did not fully capture (Li et al., 17 Apr 2026).

Against that background, Pi-Seg is positioned as a compact transfer-oriented baseline. Its central claim is that semantically guided, learnable perturbations can regularize and broaden the visual-text feature neighborhood during training, thereby making pixel-text cost maps less brittle under domain shifts. This suggests that Pi-Seg treats open-vocabulary transfer not primarily as a problem of adding larger auxiliary priors, but as a problem of reshaping the local alignment geometry between dense visual features and textual prototypes (Li et al., 17 Apr 2026).

2. Architectural composition

Pi-Seg follows the CLIP paradigm with separate frozen encoders. A frozen CLIP image encoder, instantiated as ViT-B/16 or ViT-L/14, produces dense visual features after removing pooling in order to preserve spatial granularity. A frozen CLIP text encoder produces normalized class prototypes from prompts of the form “a photo of {class}.” The resulting feature tensors are visual features $V \in \mathbb{R}^{H' \times W' \times C}$ and text features $T \in \mathbb{R}^{N \times C}$ (Li et al., 17 Apr 2026).

The model introduces two lightweight Semantic-aware Perturbation Modules (SPMs). Text-SPM learns class-wise stochastic residuals intended to enlarge the semantic neighborhoods of textual prototypes while retaining discriminability. Image-SPM generates spatially varying, text-guided perturbations for dense visual features so that perturbation remains compatible with object boundaries and spatial structure. Visual-text alignment is then performed through normalized CLIP cosine similarity to form a dense pixel-class cost volume, followed by learnable aggregation and decoding on that cost volume (Li et al., 17 Apr 2026).

The dense alignment step is defined by cosine similarity between perturbed visual and text embeddings:

$$C(i, j) \;=\; \frac{\hat{V}(i)^{\top}\hat{T}(j)}{\|\hat{V}(i)\|_2\,\|\hat{T}(j)\|_2},$$

where $i$ indexes spatial positions and $C=\{t_1,\dots,t_N\}$0 indexes class prompts. The cost volume is refined by aggregation,

$C=\{t_1,\dots,t_N\}$1

and an upsampling decoder converts $C=\{t_1,\dots,t_N\}$2 into full-resolution segmentation logits (Li et al., 17 Apr 2026).

A notable design feature is the absence of heavy external prior encoders or adapters. The frozen CLIP encoders remain unchanged, while the SPM branches, aggregation module, and decoder are trained end-to-end with the segmentation objective. In comparative terms, this distinguishes Pi-Seg from transfer strategies that rely on more elaborate multi-encoder constructions (Li et al., 17 Apr 2026).

3. Positive-incentive noise mechanism

The defining mechanism of Pi-Seg is the perturbation of both textual and visual embeddings before dense alignment. For text, the perturbed embeddings are

$C=\{t_1,\dots,t_N\}$3

Here $C=\{t_1,\dots,t_N\}$4 and $C=\{t_1,\dots,t_N\}$5 are learnable per-class residual mean and scale parameters. The paper also denotes these learnable parameters by $C=\{t_1,\dots,t_N\}$6 and $C=\{t_1,\dots,t_N\}$7, with initialization controlled by the Text-SPM standard deviation hyperparameter $C=\{t_1,\dots,t_N\}$8 (Li et al., 17 Apr 2026).

For image features, Pi-Seg uses a text-guided cross-attention module with reduction ratio $C=\{t_1,\dots,t_N\}$9 for efficiency to predict spatially varying perturbation parameters $S \in \mathbb{R}^{H \times W}$0 and $S \in \mathbb{R}^{H \times W}$1 conditioned on $S \in \mathbb{R}^{H \times W}$2. The perturbed visual features are

$S \in \mathbb{R}^{H \times W}$3

with $S \in \mathbb{R}^{H \times W}$4 broadcast to spatial-channel dimensions (Li et al., 17 Apr 2026).

The perturbations are not optimized through a separate explicit contrastive or alignment loss. Instead, the learnable perturbation parameters are driven solely by the segmentation objective so that correlations are selectively enhanced within ground-truth regions and suppressed in non-target regions. The paper characterizes this through training dynamics including gt_in_mean, non_gt_mean, gap, and align_ratio, and interprets the effect as a positive semantic incentive rather than unstructured noise (Li et al., 17 Apr 2026).

Several clarifications are important. Pi-Seg does not introduce arbitrary stochastic corruption; the image perturbation branch is explicitly text-guided, and the perturbation is aligned with semantics to avoid boundary destruction. It also does not depend on a specific stochastic family: Gaussian, Laplace, Uniform, and Student-$S \in \mathbb{R}^{H \times W}$5 are reported as viable, with moderate heavy tails described as beneficial while overly aggressive tails degrade alignment. This suggests that the operative factor is controlled broadening of the feature neighborhood rather than commitment to a single perturbation law (Li et al., 17 Apr 2026).

4. Training and inference procedure

Pi-Seg is implemented on CAT-Seg and trained end-to-end on OVRSIS95K. The prompt template is unified as “a photo of {class},” and taxonomy cleaning with category mapping is used to unify class names across datasets. During training, the procedure consists of constructing the prompt list for a dataset taxonomy, computing normalized text embeddings through the frozen CLIP text encoder, computing normalized dense visual features through the frozen CLIP image encoder, perturbing text through Text-SPM, perturbing visual features through Image-SPM, constructing the cosine-similarity cost volume, refining it through spatial and class-wise aggregation, decoding and upsampling to full-resolution logits, and updating the SPM, aggregation, and decoder parameters with AdamW while keeping the encoders frozen (Li et al., 17 Apr 2026).

The optimization objective is standard pixel-wise cross-entropy over refined cost logits:

$S \in \mathbb{R}^{H \times W}$6

The total loss is reported as

$S \in \mathbb{R}^{H \times W}$7

with no additional explicit $S \in \mathbb{R}^{H \times W}$8 or $S \in \mathbb{R}^{H \times W}$9 terms, corresponding to $C$0 in the formulation described in the paper. The perturbation parameters $C$1 are therefore learned solely through gradients from $C$2 (Li et al., 17 Apr 2026).

At inference, arbitrary class lists $C$3 may contain seen and unseen categories. Text and visual features are computed, the learned SPM parameters are applied deterministically by using the learned $C$4 and expected perturbation, and segmentation is obtained by direct $C$5 over decoded logits. The reported evaluation complexity is identical to CAT-Seg, with no extra inference GFLOPs relative to that baseline (Li et al., 17 Apr 2026).

This training and inference configuration addresses a common misconception that open-vocabulary transfer in remote sensing necessarily requires heavier inference pipelines. In Pi-Seg, the extra machinery is concentrated in training-time regularization and lightweight learned perturbation modules rather than evaluation-time expansion of the backbone stack (Li et al., 17 Apr 2026).

5. Benchmark context and empirical performance

Pi-Seg is introduced together with OVRSISBenchV2, a large-scale and application-oriented benchmark for OVRSIS. Its training foundation is OVRSIS95K, a balanced dataset of about 95,000 annotated remote sensing images with 35 semantic categories across five scene domains: town, industrial, forest, waterfront, and wasteland. On 20,000 sampled images from the semi-automated construction pipeline, the reported audit accepts 97.25% of category proposals and 91.66% of masks, with 0.53% false positives (Li et al., 17 Apr 2026).

OVRSISBenchV2 aggregates approximately 170,036 annotated images spanning 128 semantic categories and covering satellite and UAV perspectives with heterogeneous resolutions. Training uses 94,620 images from OVRSIS95K, while testing uses 75,416 images across DLRSD, FLAIR, iSAID, LoveDA, OpenEarthMap, Potsdam, UAVid, UDD5, Vaihingen, and VDD. Beyond standard OVRSIS evaluation, the benchmark includes downstream protocols for building extraction on WHU Aerial, WHUSat-II, Inria, and xBDpre; road extraction on CHN6-CUG, DeepGlobe, Massachusetts, and SpaceNet; and flood detection on WBS-SI (Li et al., 17 Apr 2026).

Evaluation uses class-wise IoU and ACC, with dataset-level means

$C$6

and

$C$7

Mean-across-dataset scores are reported as m-mIoU and m-mACC (Li et al., 17 Apr 2026).

On OVRSISBenchV2, Pi-Seg delivers the best mean results under both CLIP backbones. With ViT-B, it attains m-mIoU = 39.48 and m-mACC = 56.90, surpassing CAT-Seg at 37.88/55.72, OVRS at 37.46/54.50, GSNet at 36.25/53.57, RSKT-Seg at 35.62/53.09, and training-free baselines including ProxyCLIP at m-mIoU = 26.68. With ViT-L, it reaches m-mIoU = 44.40 and m-mACC = 63.16, exceeding CAT-Seg at 42.49/61.01, OVRS at 42.77/59.31, GSNet at 41.89/57.42, and RSKT-Seg at 40.10/59.94. Reported ViT-L dataset highlights include UAVid at 41.25/56.89, UDD5 at 59.60/74.74, Potsdam at 39.07/53.75, and DLRSD at 45.64/64.31 (Li et al., 17 Apr 2026).

On OVRSISBenchV1, Pi-Seg also improves or remains competitive depending on training source and dataset. Under DLRSD training with ViT-B, it reports m-mIoU/m-mACC = 37.28/55.99 versus RSKT-Seg at 37.22/54.45, with clear gains on UAVid at 52.47/67.30 and VDD at 44.15/63.72. Under DLRSD training with ViT-L, it reports 44.63/61.85 versus RSKT-Seg at 39.46/57.86, with large improvements on UAVid at 59.08/72.58 and VDD at 47.73/67.10. Under iSAID training, the paper describes Pi-Seg as competitive, with improvements depending on dataset and consistent trends on the harder UAVid and VDD settings (Li et al., 17 Apr 2026).

For downstream tasks trained on OVRSIS95K, Pi-Seg with ViT-B reports mean m-mIoU/m-mACC = 69.18/81.79 across building, road, and flood tasks, outperforming CAT-Seg at 61.70/83.41 and SED at 64.90/80.25. The ViT-L variant is described as consistently strong across datasets and robust in transfer to safety-critical GIS applications (Li et al., 17 Apr 2026).

These results are presented as evidence that perturbation-injected training broadens the alignment neighborhood and stabilizes cost volumes under orientation, scale, and clutter variability. A plausible implication is that Pi-Seg’s empirical gains are strongest when benchmark design emphasizes realistic cross-dataset transfer rather than narrowly matched training and testing distributions (Li et al., 17 Apr 2026).

6. Ablations, efficiency, limitations, and future directions

The ablation studies isolate the contribution of the two perturbation branches. On OVRSISBenchV2 with ViT-B, the baseline without SPM reports 36.73/56.36. Using Image-SPM only yields 38.04/57.00, while Text-SPM only gives 36.41/54.21, indicating that text perturbation in isolation can introduce semantic drift. Using both modules, corresponding to Pi-Seg, gives 39.65/57.63, which the paper interprets as a complementary effect in which Text-SPM increases semantic flexibility and Image-SPM synchronizes visual adaptation (Li et al., 17 Apr 2026).

Hyperparameter analyses show that the Image-SPM reduction ratio achieves the best trade-off at $C$8, while overly low or high values harm capacity or stability. For Text-SPM, controlled stochasticity is beneficial: m-mIoU peaks around $C$9, and m-mACC around $C_{\text{test}}$0. Random-seed experiments report small standard deviations in m-mIoU and m-mACC, which the paper uses to support stability claims. Resolution experiments report best performance in high-resolution regimes, with m-mIoU/m-mACC = 44.50/62.93 for ViT-B, while remaining competitive on low-resolution inputs (Li et al., 17 Apr 2026).

Efficiency results emphasize the compactness of the design. Pi-Seg reports 155.60M parameters for ViT-B and 436.68M for ViT-L, compared with RSKT-Seg at 398.07M and 678.29M, respectively. Training GFLOPs are modestly higher because of SPM, but evaluation GFLOPs are identical to CAT-Seg, and FPS is comparable; for example, ViT-B runs at 66.89 FPS versus CAT-Seg at 66.30. The framework also avoids the sliding-window inference overhead required by heavy baselines (Li et al., 17 Apr 2026).

Qualitative analyses are described as showing cleaner masks, sharper boundaries, better background suppression, and cost maps that are more concentrated on targets while suppressing spurious activations. The training dynamics further support the “positive semantic incentive” interpretation through strengthened target-region correlations and suppressed non-target correlations (Li et al., 17 Apr 2026).

The limitations are explicit. Fine-grained category confusions persist, including classes such as ship or airplane versus vehicle, because the framework still relies on short prompts and coarse prototypes. Rotation invariance is not explicitly modeled; robustness is attributed to smoother alignment spaces rather than explicit equivariance. Future directions identified in the paper include enriching semantics with attribute-enhanced prompts or hierarchical category definitions, strengthening context modeling for visually similar classes, exploring distribution-adaptive perturbations tailored to sensing conditions, and integrating rotation-aware modules to complement feature-space smoothing (Li et al., 17 Apr 2026).

Taken together, Pi-Seg is best understood as a compact OVRSIS baseline that replaces heavy multi-encoder transfer with semantically guided perturbation of visual and textual embeddings. Within the benchmark regime defined by OVRSIS95K and OVRSISBenchV2, its contribution is not only a specific architecture, but also a concrete empirical argument that feature-space broadening can improve open-vocabulary pixel-text alignment under realistic remote sensing domain shift (Li et al., 17 Apr 2026).