SimFeatUp: Training-Free Upsampler for Remote Sensing

• The paper introduces SimFeatUp as a training-free feature upsampler that restores lost spatial details from low-resolution deep features in remote sensing images.
• It employs a recycled Joint Bilateral Upsampling module with an enlarged 11x11 kernel to enhance boundary accuracy and preserve object shape fidelity.
• SimFeatUp integrates seamlessly within segmentation frameworks, offering plug-and-play spatial restoration without the need for task-specific retraining.

Searching arXiv for SimFeatUp and related papers. SimFeatUp is a training-free feature upsampler introduced for open-vocabulary semantic segmentation of remote sensing images, where low-resolution deep features from visual encoders such as CLIP can produce distorted target shapes and ill-fitting boundaries. It is presented as a simple and general upsampler intended to restore lost spatial information in deep features in a training-free style, and it functions within the SegEarth-OV framework as a universal upsampler that restores high-resolution spatial details from coarse features without task-specific post-training (Li et al., 2024). Subsequent work characterizes it as a universal upsampler that robustly restores high-resolution spatial details from coarse features and as a central component in annotation-free open-vocabulary segmentation for remote-sensing images, including extension to broader remote sensing settings through associated modules such as Global Bias Alleviation and AlignEarth (Li et al., 25 Aug 2025).

1. Problem setting and motivation

SimFeatUp is motivated by a specific failure mode of open-vocabulary semantic segmentation in remote sensing. Remote sensing images are described as being sensitive to low-resolution features, and the resulting prediction masks exhibit distorted target shapes and ill-fitting boundaries when standard CLIP-based pipelines operate on coarse deep representations (Li et al., 2024). The underlying issue is that CLIP-style encoders generate low-resolution feature maps, often at $1/16$ of the input resolution, and this coarse representation is poorly matched to pixel-level interpretation tasks requiring accurate boundary placement and shape preservation (Li et al., 2024).

The method is therefore positioned around restoration of spatial detail rather than semantic re-learning. The stated goal is to improve boundary accuracy and object shape fidelity by recovering high-frequency, content-consistent details lost during feature downsampling (Li et al., 2024). In later formulation, the same objective is expressed as robust restoration of high-resolution spatial details from coarse VLM features, correcting distorted target shapes without any task-specific post-training (Li et al., 25 Aug 2025).

This framing places SimFeatUp within the broader remote-sensing OVSS problem rather than as a general segmentation backbone. A plausible implication is that its contribution is infrastructural: it intervenes between frozen visual-language encoders and dense prediction heads to compensate for the spatial inadequacy of encoder outputs while preserving the annotation-free character of the overall pipeline.

2. Core design and training paradigm

SimFeatUp is described as an evolution and redesign of FeatUp toward training-free inference for OVSS, with several explicit modifications (Li et al., 2024). It is characterized as a training-free feature upsampler that can be trained on a small, unrelated set of unlabeled images; no label or explicit supervision is needed; once trained, it is universally applicable as an external module within training-free OVSS pipelines; and it introduces no new trainable parameters at inference time (Li et al., 2024).

The design choices emphasized in the source material are consistent across the two SegEarth-OV papers. SimFeatUp adds an image reconstruction loss to ensure content in high-resolution features remains trustworthy for pixel-level inference, upsamples features from an earlier stage in CLIP rather than the final output tokens, employs one parameterized JBU block recycled multiple times for arbitrary upsampling ratios, and enlarges the upsampling kernel from $7 \times 7$ to $11 \times 11$ to handle the multi-scale nature of remote sensing targets (Li et al., 2024). The later paper restates these ideas as content retention, optimal feature selection, larger adaptive upsampling kernels, structural simplification, and universal training on generic remote-sensing images (Li et al., 25 Aug 2025).

The training paradigm is unusual relative to conventional segmentation systems because SimFeatUp itself is trained, but the downstream segmentation usage is described as training-free. It is trained only on images, with no segmentation labels, on a generic remote sensing image dataset such as Million-AID with 16k images, and once trained it is reused as a parameter-frozen module at test time for arbitrary remote sensing images without retraining or fine-tuning (Li et al., 2024). This suggests that “training-free” in this context refers to task-time deployment rather than absence of any pretraining.

3. Mathematical formulation and feature selection

The method’s optimization objective combines feature-space consistency and image-space content retention. The feature reconstruction loss is given as

$$\mathcal{L}_{rec}=\left\|\mathcal{O}[1:hw+1] - \sigma_{\downarrow}(\sigma_{\uparrow}(\mathcal{O}[1:hw+1])) \right\|_{2}^{2},$$

where $\mathcal{O}[1:hw+1]$ denotes non-$[CLS]$ output tokens from CLIP, $\sigma_{\uparrow}$ is the learnable upsampler, and $\sigma_{\downarrow}$ is the learnable downsampler (Li et al., 2024). In the later formulation, the same loss is written over selected CLIP image features $\mathcal{O}'$ as

$$\mathcal{L}_{rec} = \Big \| \mathcal{O}' - \sigma_{\downarrow} (\sigma_{\uparrow} (\mathcal{O}')) \Big \|^2_2 .$$

The content retention term is the distinguishing addition of SimFeatUp:

$7 \times 7$0

with $7 \times 7$1 the original input image and $7 \times 7$2 a lightweight Content Retention Net composed of 2 conv layers and Tanh (Li et al., 2024). The total training loss is

$7 \times 7$3

with $7 \times 7$4 set to $7 \times 7$5 in experiments (Li et al., 2024). During inference, only SimFeatUp is kept, while the CRN is discarded (Li et al., 2024).

A central architectural choice is what feature representation is upsampled. Instead of upsampling the final output tokens $7 \times 7$6, SimFeatUp upsamples feature tokens before the last transformer block but post-projection:

$7 \times 7$7

where $7 \times 7$8 is the input to CLIP’s last transformer block and $7 \times 7$9 is the subsequent linear projection from the CLIP encoder (Li et al., 2024). The later paper describes this as upsampling earlier outputs $11 \times 11$0, projected to match token dimension, rather than the final encoder output (Li et al., 25 Aug 2025). This suggests that SimFeatUp treats representation stage selection as a substantive factor in dense-prediction fidelity, not merely an implementation detail.

4. Joint Bilateral Upsampling mechanism

The upsampling engine in SimFeatUp is a learnable, parameterized Joint Bilateral Upsampling module. The method uses stacked JBU modules, each upsampling by $11 \times 11$1 spatially, and for a typical $11 \times 11$2 to $11 \times 11$3 scaling, four applications are used; in the JBU_One configuration, the parameters are shared across these stages (Li et al., 25 Aug 2025). The earlier paper describes this as employing one parameterized JBU block recycled multiple times for arbitrary upsampling ratios, reducing parameters (Li et al., 2024).

The JBU weights combine a spatial component and a range component. The spatial component is

$11 \times 11$4

while the range component is expressed as

$11 \times 11$5

where $11 \times 11$6 is guidance feature extracted from the high-resolution RGB image, $11 \times 11$7 is the local window, and $11 \times 11$8 is a small multilayer perceptron (Li et al., 2024). The later paper gives an equivalent description, stating that JBU computes each upsampled pixel as a weighted average of nearby low-resolution pixels, with weights based on spatial proximity and feature similarity (Li et al., 25 Aug 2025).

A distinctive remote-sensing-oriented modification is the larger window size. SimFeatUp increases the kernel from $11 \times 11$9 to $$\mathcal{L}_{rec}=\left\|\mathcal{O}[1:hw+1] - \sigma_{\downarrow}(\sigma_{\uparrow}(\mathcal{O}[1:hw+1])) \right\|_{2}^{2},$$0, with the stated motivation of handling the multi-scale nature of remote sensing targets and capturing sharply varying and large-scale targets (Li et al., 2024). In ablation, this enlarged kernel is reported to give a uniform additional gain, up to $$\mathcal{L}_{rec}=\left\|\mathcal{O}[1:hw+1] - \sigma_{\downarrow}(\sigma_{\uparrow}(\mathcal{O}[1:hw+1])) \right\|_{2}^{2},$$1 in one account and $$\mathcal{L}_{rec}=\left\|\mathcal{O}[1:hw+1] - \sigma_{\downarrow}(\sigma_{\uparrow}(\mathcal{O}[1:hw+1])) \right\|_{2}^{2},$$2 mIoU in the later one (Li et al., 2024, Li et al., 25 Aug 2025).

5. Integration within SegEarth-OV

Within the SegEarth-OV framework, SimFeatUp sits between image encoding and dense prediction. The pipeline is described as: remote sensing image input; patchification and CLIP image encoding with a ViT backbone; extraction of non-$$\mathcal{L}_{rec}=\left\|\mathcal{O}[1:hw+1] - \sigma_{\downarrow}(\sigma_{\uparrow}(\mathcal{O}[1:hw+1])) \right\|_{2}^{2},$$3 features $$\mathcal{L}_{rec}=\left\|\mathcal{O}[1:hw+1] - \sigma_{\downarrow}(\sigma_{\uparrow}(\mathcal{O}[1:hw+1])) \right\|_{2}^{2},$$4; feature upsampling through SimFeatUp to a high-resolution grid; Global Bias Alleviation; text encoding using prompts such as “a photo of a {class}”; and dense prediction by similarity between upsampled, de-biased visual features and text features (Li et al., 2024).

Global Bias Alleviation is a complementary operation motivated by the observation of the abnormal response of local patch tokens to the $$\mathcal{L}_{rec}=\left\|\mathcal{O}[1:hw+1] - \sigma_{\downarrow}(\sigma_{\uparrow}(\mathcal{O}[1:hw+1])) \right\|_{2}^{2},$$5 token in CLIP. The operation subtracts the global signal using scalar $$\mathcal{L}_{rec}=\left\|\mathcal{O}[1:hw+1] - \sigma_{\downarrow}(\sigma_{\uparrow}(\mathcal{O}[1:hw+1])) \right\|_{2}^{2},$$6:

$$\mathcal{L}_{rec}=\left\|\mathcal{O}[1:hw+1] - \sigma_{\downarrow}(\sigma_{\uparrow}(\mathcal{O}[1:hw+1])) \right\|_{2}^{2},$$7

where $$\mathcal{L}_{rec}=\left\|\mathcal{O}[1:hw+1] - \sigma_{\downarrow}(\sigma_{\uparrow}(\mathcal{O}[1:hw+1])) \right\|_{2}^{2},$$8 is the repeated $$\mathcal{L}_{rec}=\left\|\mathcal{O}[1:hw+1] - \sigma_{\downarrow}(\sigma_{\uparrow}(\mathcal{O}[1:hw+1])) \right\|_{2}^{2},$$9 token vector (Li et al., 2024). The later paper describes this as subtracting the inherent global context from patch features to enhance local semantic fidelity (Li et al., 25 Aug 2025).

In the extended annotation-free formulation, SimFeatUp remains the central upsampling backbone, and all modalities, including optical and SAR via AlignEarth transfer, rely on it to convert CLIP or aligned encoder features to high-resolution semantic maps (Li et al., 25 Aug 2025). A plausible implication is that SimFeatUp functions as a modality-agnostic spatial restoration layer as long as the upstream encoder yields semantically aligned low-resolution features.

6. Empirical behavior, ablations, and reported performance

The most detailed quantitative description comes from the 2024 SegEarth-OV paper. Across 17 remote sensing datasets spanning semantic segmentation, building extraction, road detection, and flood detection tasks, the method achieves an average of $\mathcal{O}[1:hw+1]$0, $\mathcal{O}[1:hw+1]$1, $\mathcal{O}[1:hw+1]$2, and $\mathcal{O}[1:hw+1]$3 improvement over state-of-the-art methods on the four tasks, respectively (Li et al., 2024). For open-vocabulary semantic segmentation benchmarks, SegEarth-OV with SimFeatUp is reported to outperform baselines including MaskCLIP, SCLIP, GEM, and ClearCLIP across 8 remote sensing benchmarks, with an average mIoU increase of $\mathcal{O}[1:hw+1]$4 over the previous best (Li et al., 2024). For single-class extraction tasks, SimFeatUp-enabled SegEarth-OV delivers $\mathcal{O}[1:hw+1]$5 IoU improvement on building extraction, $\mathcal{O}[1:hw+1]$6 on road segmentation, and $\mathcal{O}[1:hw+1]$7 on flood detection (Li et al., 2024).

The paper also reports a plug-and-play property: SimFeatUp plus global subtraction can be bolted on to other OVSS variants, and adding SimFeatUp to MaskCLIP, SCLIP, or ClearCLIP consistently yields significant improvements. One example given is ClearCLIP+Ours, with $\mathcal{O}[1:hw+1]$8 on OpenEarthMap and $\mathcal{O}[1:hw+1]$9 on WHU-Aerial (Li et al., 2024). The later paper similarly states that adding SimFeatUp plus Global Bias Alleviation to MaskCLIP, SCLIP, and ClearCLIP raises their scores by $[CLS]$0–$[CLS]$1, including a change on WHU(Aerial) from $[CLS]$2 to $[CLS]$3 for ClearCLIP (Li et al., 25 Aug 2025).

The ablation studies identify the functional contribution of each design element. The following components are explicitly reported:

Component	Reported effect
FeatUp (original, on natural images)	Only minor benefits and sometimes negative effects
Switching to earlier features $[CLS]$4	Significant mIoU improvements, e.g. $[CLS]$5 on WBS-SI
Training on remote sensing images	Further gains over natural-image training
JBU_One	Slight further improvement and drastic parameter reduction
Adding $[CLS]$6	Another $[CLS]$7–$[CLS]$8 absolute boost
Alleviating global bias	Boosts up to $[CLS]$9
Expanding the kernel to $\sigma_{\uparrow}$0	Uniform additional gain, up to $\sigma_{\uparrow}$1

These findings are stated in the 2024 paper’s Table 4 summary (Li et al., 2024). The 2025 paper restates similar ablation outcomes and adds that JBU_One yields similar or better accuracy with $\sigma_{\uparrow}$2 fewer parameters, remote sensing-specific training provides $\sigma_{\uparrow}$3–$\sigma_{\uparrow}$4 for remote-sensing images over training the upsampler on natural images, and “Rec. Image” improves preservation of small objects (Li et al., 25 Aug 2025).

Qualitatively, the method is reported to produce masks with better delineated boundaries and more spatially accurate object shapes, and the omission of the image reconstruction loss is associated with small objects vanishing in the reconstructed representation (Li et al., 2024, Li et al., 25 Aug 2025). Since these are visual-comparison claims rather than scalar metrics, they are best interpreted as evidence of the intended geometric correction behavior.

7. Scope, universality, and relation to surrounding methods

SimFeatUp is consistently described as universal, plug-and-play, and annotation-free in deployment. It can be trained once on generic remote sensing images and then used for semantic segmentation, building extraction, road extraction, and flood mapping without retraining (Li et al., 2024). The later paper extends this framing to “any remote sensing dataset/purpose” and reports additional experiments on natural images, where SimFeatUp boosts MaskCLIP, SCLIP, and ClearCLIP by $\sigma_{\uparrow}$5 mIoU without image-specific adaptation (Li et al., 25 Aug 2025).

Its relationship to FeatUp is important but narrowly defined in the source material. SimFeatUp is described as inspired by FeatUp and as an evolution and redesign of FeatUp toward training-free inference for remote sensing OVSS (Li et al., 2024). The main differences explicitly stated are the image reconstruction loss, earlier feature selection, a single recycled JBU block, and the enlarged $\sigma_{\uparrow}$6 kernel (Li et al., 2024). In ablation, FeatUp trained on natural images provides only minor benefits and can even degrade performance on some remote sensing datasets, which is used to motivate SimFeatUp’s domain-specific adjustments (Li et al., 2024).

A common misunderstanding would be to read “training-free” as meaning that SimFeatUp is never trained. The source material does not support that interpretation. Instead, SimFeatUp is trained on unlabeled images, with no labels or explicit supervision, and then used as a fixed module at inference time with no gradients, no retraining, and no fine-tuning (Li et al., 2024). Another potential misconception is that it is itself a full segmentation framework. The papers instead place it as one component within SegEarth-OV, operating after image encoding and before dense mask prediction (Li et al., 2024).

In this sense, SimFeatUp can be understood as a remote-sensing-specific feature restoration module for open-vocabulary dense prediction. The available evidence suggests that its principal contribution lies in reconciling frozen VLM semantics with the spatial precision demands of Earth observation, especially where distorted target shapes, boundary errors, and multi-scale objects make low-resolution feature maps inadequate (Li et al., 2024, Li et al., 25 Aug 2025).