SegEarth-OV: Open-Vocabulary Segmentation

• SegEarth-OV is a framework for open-vocabulary semantic segmentation that uses text prompts to generate pixel-level masks without manual annotation.
• It combines SimFeatUp for restoring spatial details with global bias alleviation to improve local semantic fidelity in remote sensing images.
• AlignEarth extends the framework to SAR imagery, enabling cross-modal, annotation-free segmentation across diverse sensor modalities.

Searching arXiv for SegEarth-OV and related papers to ground the article in current literature. SegEarth-OV is a framework for open-vocabulary semantic segmentation in remote sensing images that aims to produce pixel-level masks for arbitrary text-specified categories without manual semantic annotation. In its original formulation, it is presented as a training-free remote-sensing OVSS method built around two components—SimFeatUp and a patch-token debiasing step—to address the failure modes of CLIP-style vision-LLMs on Earth observation imagery, especially loss of spatial detail, distorted target shapes, and ill-fitting boundaries (Li et al., 2024). A later formulation recasts the same line of work as the first annotation-free open-vocabulary segmentation framework for remote sensing, extends it beyond optical imagery through AlignEarth, and emphasizes universality across sensor modalities, including SAR (Li et al., 25 Aug 2025).

1. Problem setting and motivation

SegEarth-OV is situated in remote-sensing open-vocabulary semantic segmentation, where the objective is to assign a semantic label to each pixel using text prompts rather than a fixed closed taxonomy. The motivating constraint is the high cost of manual annotation in remote sensing, combined with the need to interpret novel categories in applications such as agriculture, water resources, military, disaster relief, urban planning, and environmental monitoring (Li et al., 2024).

The framework is motivated by a specific domain mismatch between natural-image vision-LLMs and remote-sensing imagery. The papers identify three central difficulties. First, VLM image encoders such as CLIP downsample aggressively, so pixel-level predictions become coarse and small or thin structures such as roads and buildings are poorly localized. Second, patch-level features exhibit contamination from global semantics associated with the [CLS] token, which is useful for image-level classification but harmful for dense prediction. Third, for modalities such as SAR, large-scale VLMs are unavailable, which blocks straightforward transfer of open-vocabulary segmentation methods developed for optical imagery (Li et al., 2024, Li et al., 25 Aug 2025).

This problem framing is technically significant because remote-sensing scenes contain large scale variation, fine-grained details, strong orientation variation, and a high prevalence of “stuff” classes such as cropland and forest. The SegEarth-OV line treats these as failures of feature resolution and local semantic fidelity rather than primarily failures of downstream classifier design.

2. Framework structure and inference pipeline

The SegEarth-OV pipeline combines a frozen vision-language encoder with feature restoration and feature debiasing, then performs pixel-wise similarity against text embeddings. In the later extension, a distillation module is added so that the same open-vocabulary pipeline can operate on SAR imagery (Li et al., 25 Aug 2025).

Component	Function	Scope
SimFeatUp	Restores high-resolution spatial detail from coarse VLM features	Optical and, after alignment, SAR
Global Bias Alleviation	Subtracts global context from patch features	Dense prediction refinement
AlignEarth	Distills optical VLM semantics into an SAR encoder	Cross-modal extension

In the original optical inference workflow, features are extracted from CLIP, upsampled by SimFeatUp, corrected by subtracting a scaled [CLS] embedding, and then classified by similarity to prompt text embeddings (Li et al., 2024). The original paper describes the resulting setup as training-free because no dataset-specific retraining or adaptation is performed at downstream inference time; the later paper describes it as annotation-free because the framework avoids manual semantic annotation and task-specific post-training (Li et al., 25 Aug 2025). This suggests that the two descriptions emphasize different aspects of the same design principle: avoidance of label-dependent supervision in the target OVSS task.

The implementation described for the original version uses standard CLIP ViT architectures, ViT-B/16 in most experiments, with images resized to a long side of 448 and sliding-window inference (Li et al., 2024). For semantic segmentation the evaluation metric is mIoU, and for single-class tasks the metric is foreground IoU (Li et al., 2024).

3. SimFeatUp: universal feature upsampling

SimFeatUp is the core mechanism introduced to mitigate spatial information loss in deep vision-language features. It is a simple and general upsampler intended to restore lost spatial information in deep features in a training-free style (Li et al., 2024). The later paper describes it as a universal, annotation-free upsampler that robustly restores high-resolution spatial details from coarse features and corrects distorted target shapes without any task-specific post-training (Li et al., 25 Aug 2025).

Its design is explicitly motivated by the limitations of FeatUp. SegEarth-OV retains the idea of feature upsampling via joint bilateral upsampling, but modifies it in several ways. The selected feature to upsample is taken from the input to the last CLIP transformer block and projected into the multimodal space:

$$\mathcal{O}^{\prime} = \operatorname{Proj}(X[1:hw+1]).$$

The original reconstruction objective is

$$\mathcal{L}_{rec} = \left\| \mathcal{O}[1:hw+1] - \sigma_{\downarrow}(\sigma_{\uparrow}(\mathcal{O}[1:hw+1])) \right\|_2^2.$$

SegEarth-OV adds an image reconstruction term through a Content Retention Network:

$$\mathcal{L}_{img} = \left\| I - \operatorname{CRN}(\sigma_{\uparrow}(\mathcal{O}[1:hw+1])) \right\|_2^2,$$

and optimizes the combined loss

$$\mathcal{L} = \mathcal{L}_{rec} + \gamma \mathcal{L}_{img}.$$

The upsampler also departs from prior formulations by using a single parameterized JBU module, denoted “JBU-One,” recursively for larger upsampling ratios, rather than stacking independently parameterized modules. In the later paper, SimFeatUp further adopts an $11 \times 11$ window instead of a traditional $7 \times 7$ window to better handle the extreme scale variation characteristic of remote-sensing objects (Li et al., 25 Aug 2025).

The technical rationale is that remote-sensing OVSS is unusually sensitive to feature resolution. According to the papers, the CRN-driven loss improves content retention, while the earlier-stage feature selection and larger JBU neighborhood make the upsampler more compatible with fine structures and multi-scale targets (Li et al., 2024, Li et al., 25 Aug 2025). In practical terms, SimFeatUp is trained once on unlabeled remote-sensing imagery and then used as a plug-and-play module across datasets and categories.

4. Global Bias Alleviation and local semantic fidelity

The second defining component of SegEarth-OV is a post-processing operation that addresses “global bias” in local patch features. The papers attribute this bias to the abnormal response of local patch tokens to the [CLS] token in CLIP, arising from image-level pretraining objectives that entangle local and global semantics (Li et al., 2024).

The operation is intentionally simple:

$$\hat{\mathcal{O}} = \mathcal{O}[1:hw+1] - \lambda \mathcal{O}[0].$$

Here, $\mathcal{O}[0]$ is the repeated global token and $\lambda$ controls the subtraction strength. In the original paper, $\lambda$ is empirically fixed to 0.3 across datasets (Li et al., 2024). The later paper presents the same operation as Global Bias Alleviation and emphasizes that it is a simple, efficient post-processing technique requiring no retraining (Li et al., 25 Aug 2025).

This component is conceptually important because SegEarth-OV does not treat localization failure as solely a resolution problem. It also treats it as a representational contamination problem: patch features optimized for imagewise classification carry global scene context that can activate non-target regions during pixel-wise similarity matching. The reported visualization evidence shows that subtraction of the global token restores local specificity and improves mIoU (Li et al., 25 Aug 2025).

Taken together, SimFeatUp and Global Bias Alleviation form a two-step correction of frozen VLM features: first recovering spatial detail, then increasing local semantic discriminability. That pairing is the essential technical identity of SegEarth-OV.

5. AlignEarth and extension beyond optical imagery

The later SegEarth-OV paper extends the framework to SAR through AlignEarth, a distillation-based strategy that transfers semantic knowledge from an optical VLM encoder to an SAR encoder using paired optical-SAR images, without manual or text annotations (Li et al., 25 Aug 2025). This extension addresses a major practical limitation of open-vocabulary segmentation in remote sensing: the absence of large-scale SAR foundation VLMs.

AlignEarth uses three losses. The global contrastive loss aligns optical and SAR [CLS] tokens contrastively within a batch:

$$\mathcal{L}_{rec} = \left\| \mathcal{O}[1:hw+1] - \sigma_{\downarrow}(\sigma_{\uparrow}(\mathcal{O}[1:hw+1])) \right\|_2^2.$$0

The global distillation term is

$$\mathcal{L}_{rec} = \left\| \mathcal{O}[1:hw+1] - \sigma_{\downarrow}(\sigma_{\uparrow}(\mathcal{O}[1:hw+1])) \right\|_2^2.$$1

and the local distillation term averages features over $$\mathcal{L}_{rec} = \left\| \mathcal{O}[1:hw+1] - \sigma_{\downarrow}(\sigma_{\uparrow}(\mathcal{O}[1:hw+1])) \right\|_2^2.$$2 regions before aligning them:

$$\mathcal{L}_{rec} = \left\| \mathcal{O}[1:hw+1] - \sigma_{\downarrow}(\sigma_{\uparrow}(\mathcal{O}[1:hw+1])) \right\|_2^2.$$3

The use of region-level rather than one-to-one patch-level local distillation is motivated by imperfect alignment and acquisition-time mismatches in paired optical-SAR data (Li et al., 25 Aug 2025). Once aligned, the SAR encoder exports CLIP-aligned features and [CLS] tokens that are compatible with SimFeatUp and Global Bias Alleviation, allowing the original SegEarth-OV inference logic to be reused for SAR.

This extension changes the scope of SegEarth-OV from a remote-sensing optical OVSS method into a modality-agnostic annotation-free framework. A plausible implication is that the line of work treats open-vocabulary segmentation less as a property of a particular encoder and more as a property of an aligned feature space plus a dense-prediction interface.

6. Experimental results and empirical trajectory

The original SegEarth-OV paper evaluates on 17 remote-sensing datasets spanning semantic segmentation, building extraction, road detection, and flood detection. Specifically, it uses 8 semantic segmentation datasets—OpenEarthMap, LoveDA, iSAID, Potsdam, Vaihingen, UAVid, UDD5, and VDD—4 building extraction datasets, 4 road detection datasets, and 1 flood detection dataset (Li et al., 2024). Against training-free baselines including CLIP, MaskCLIP, SCLIP, GEM, and ClearCLIP, SegEarth-OV reports an average mIoU of 39.2% on the 8 semantic segmentation datasets, improving the previous best average from 33.4% to 39.2%, that is, by 5.8% (Li et al., 2024).

For the single-class tasks, the same paper reports an average improvement of 8.2% on building extraction, 4.0% on road detection, and 15.3% on flood detection over state-of-the-art methods (Li et al., 2024). Concrete examples include 49.2% IoU on WHU Aerial building extraction at $$\mathcal{L}_{rec} = \left\| \mathcal{O}[1:hw+1] - \sigma_{\downarrow}(\sigma_{\uparrow}(\mathcal{O}[1:hw+1])) \right\|_2^2.$$4, 35.4% IoU on CHN6-CUG road detection, and 60.2% IoU on WBS-SI flood detection (Li et al., 2024). These results support the paper’s central claim that feature restoration and debiasing are sufficient to make training-free OVSS practical in remote sensing.

The annotation-free extension preserves the 39.2% average mIoU on 8 optical remote-sensing datasets and adds SAR benchmarks. On 8 SAR datasets, direct application of optical VLM methods yields very low mIoU, below 16%, while SegEarth-OV with AlignEarth raises PIE-SAR from 12.1% to 51.1% and DDHR-Korea from 16.5% to 48.6% (Li et al., 25 Aug 2025). The same paper also reports that SimFeatUp and Global Bias Alleviation improve other OVSS models, including MaskCLIP, SCLIP, and ClearCLIP, by 2–9% mIoU, and that the framework shows modest improvements on Cityscapes and COCO-Stuff (Li et al., 25 Aug 2025).

A later follow-up, “SegEarth-OV3,” explores replacing the CLIP-centered pipeline with SAM 3 in a promptable, training-free remote-sensing OVSS setting. That work introduces dual-head mask fusion and presence-guided category filtering, and reports 53.4% mIoU averaged over 8 remote-sensing benchmarks, outperforming the best prior training-free method, CorrCLIP, by 12.7% mIoU (Li et al., 9 Dec 2025). In historical terms, this indicates that the SegEarth-OV program evolved from CLIP feature correction toward promptable unified segmentation-recognition architectures while preserving the training-free OVSS objective.

7. Interpretation, terminology, and relation to adjacent SegEarth systems

A recurrent source of confusion is the phrase “training-free.” In the original SegEarth-OV paper, SimFeatUp is trained on 16,000 unlabeled images randomly selected from Million-AID, but no semantic or dense annotation is used, and no further training or adaptation is performed for new datasets or unseen classes (Li et al., 2024). The later paper makes this distinction explicit by describing the framework as annotation-free rather than simply training-free (Li et al., 25 Aug 2025). The terminology therefore refers to the absence of task-specific semantic supervision and downstream retraining, not to the complete absence of learned components.

Within the broader SegEarth line, SegEarth-OV occupies the open-vocabulary, text-prompted, category-centric branch. SegEarth-R1 introduces geospatial pixel reasoning, a different task in which implicit natural-language queries are parsed and converted into masks; it is trained end-to-end, integrates a hierarchical visual encoder with an LLM, and is positioned as handling implicit queries rather than the open-vocabulary category prompting emphasized by SegEarth-OV (Li et al., 13 Apr 2025). SegEarth-R2 further expands language-guided segmentation toward hierarchical granularity, target multiplicity, reasoning requirements, and linguistic variability, again moving beyond the category-prompt formulation that defines SegEarth-OV (Xin et al., 23 Dec 2025).

In that broader context, SegEarth-OV is best understood not merely as a single algorithm but as the foundational open-vocabulary branch of the SegEarth research program. Its defining contribution is the claim—supported by results across optical and SAR settings—that remote-sensing OVSS can be made practical through annotation-free feature restoration, local semantic debiasing, and cross-modal semantic alignment, without relying on dense manual labeling (Li et al., 2024, Li et al., 25 Aug 2025).