Referring Uncertainty Scorer (RUS) in Remote Sensing

• Referring Uncertainty Scorer (RUS) is a module that defines spatial uncertainty by generating pixel-level uncertainty maps in remote sensing image segmentation.
• It employs cross-modal mechanisms such as linear projections and scaled dot-product attention to fuse visual and text features for error-aware segmentation.
• Integrated with adaptive modules like Uncertainty-Gated Fusion and Uncertainty-Driven Local Refinement, RUS improves segmentation accuracy and boundary precision.

The Referring Uncertainty Scorer (RUS) is a plug-and-play cross-modal module for explicit spatial modeling of referential ambiguity in referring remote sensing image segmentation. RUS estimates a dense uncertainty map that guides adaptive cross-modal fusion and localized refinement without introducing changes to the visual-language backbone architecture. In the context of complex overhead imagery, RUS captures spatial non-uniformity in semantic alignment, serving as an interpretable spatial prior for error-aware segmentation and modular downstream adaptation (Sun et al., 7 Jan 2026).

1. Formal Definition and Output Construction

Given a remote-sensing image $I$ and a referring text $S$, feature encoding proceeds via two-tower architectures: a visual encoder (e.g., ConvNeXt) and a text encoder (e.g., BERT). The coarsest-level visual tokens ($\mathbf{V}\in\mathbb{R}^{B\times N_u\times C}$, $N_u=H_u W_u$) and text tokens ($\mathbf{T}\in\mathbb{R}^{B\times L\times C}$) constitute the joint cross-modal input. The RUS produces a pixel-wise uncertainty logit map $\mathbf{u}\in\mathbb{R}^{B\times N_u\times 1}$ and reshapes it to a spatial map $U\in\mathbb{R}^{B\times 1\times H_u\times W_u}$. $U$ can be converted to a probabilistic uncertainty map $U^p = \sigma(U)$ via elementwise sigmoid. The logit-based $U$ serves as a high-dimensional spatial prior during inference.

2. Architectural Design and Attention Mechanisms

RUS is formulated as a shallow cross-modal network designed for effective and interpretable uncertainty estimation. The module architecture is summarized below:

1. Linear projections: Both visual and text features undergo linear projections ($V' = V W_v$, $T' = T W_t$), where $W_v, W_t\in\mathbb{R}^{C\times C}$ are initialized near identity.
2. Scaled dot-product attention: Attention coefficients $A = \mathrm{Softmax}(\gamma V'(T')^\top)$ (with $\gamma$ a learnable scalar, default $1/\sqrt{C}$) weight text embeddings for each visual position, producing aggregated $\tilde{T}$.
3. Concatenation and MLP: The projected visual features and attended text features are concatenated along channel dimension and passed through a two-layer MLP to produce scalar logits per spatial position.
4. Reshaping: The logits are reshaped to the spatial layout of uncertainty $U$.

This lightweight structure allows RUS to be efficiently trained and seamlessly inserted into existing multimodal segmentation models.

3. Online Error-Consistency Supervision

RUS training employs an online error-consistency supervision paradigm to ensure that its uncertainty estimates correlate with the model’s current segmentation errors. The supervision pipeline comprises:

• Coarse prediction ($\hat{Y}$): Foreground probability from the main decoder.
• Binary prediction: Thresholded $$\hat{y}(x) = \mathrm{StopGrad}(\mathbb{I}[\hat{Y}(x)>0.5])$$.
• Error map ($e(x)$): Comparison to ground truth $y(x)$ produces the online error map $e(x)=\mathbb{I}(\hat{y}(x)\neq y(x))$, optionally blurred to obtain $z(x)$.
• Spatial alignment: RUS uncertainty logits $U$ are upsampled (bilinear, logit space) to match the segmentation resolution.
• Supervision loss: Weighted binary cross-entropy

$$\mathcal{L}_{\mathrm{unc}} = \frac{1}{BHW} \sum_{b=1}^B \sum_{x=1}^{H\times W} w^{\mathrm{sample}}_b w^{\mathrm{pixel}}_{b,x} \; \mathrm{BCEWithLogits}(\tilde U_b(x), z_b(x))$$

with sample-level weighting to emphasize hard images (low IoU), and pixel-level reweighting balancing error sparsity.

The total training objective is $$\mathcal{L} = \mathcal{L}_{\mathrm{seg}} + \lambda_{\mathrm{unc}} \mathcal{L}_{\mathrm{unc}}$$, with $\lambda_{\mathrm{unc}}\approx 1$. This supervision aligns RUS outputs with the spatiotemporal structure of segmentation errors, yielding a meaningful uncertainty prior.

4. Inference Procedure and Functionality

In deployment, the RUS module processes encoder outputs to yield $U$ without access to error supervision. The resulting $U$ is interpreted as a spatial heatmap of referential ambiguity and local segmentation risk. RUS output is upsampled as necessary for integration into adaptation modules (see next section). No feedback from segmentation errors is required at test time, establishing RUS as an independent prior estimator.

5. Integration with Adaptive Inference Modules

RUS uncertainty maps act as external spatial priors for two downstream plug-and-play modules: Uncertainty-Gated Fusion (UGF) and Uncertainty-Driven Local Refinement (UDLR).

a) Uncertainty-Gated Fusion (UGF):

• Inserted prior to the multi-scale decoder.
• Given decoder-level tokens, computes cross-attention visual-language updates.
• RUS uncertainty $U$ (after resolution alignment) determines position-wise gating coefficients $\mathbf{g} = \sigma(\alpha\,\mathbf{u}+\beta)$, with learnable affine $\alpha,\beta$.
• Visual features are updated as

$$\mathbf{V}^+ = \mathrm{LayerNorm}\left(\mathbf{V} + \mathbf{g}\odot\Delta\mathbf{V}\right)$$

enhancing language injection in ambiguous regions (high $U$) and suppressing it where prediction is confident.

b) Uncertainty-Driven Local Refinement (UDLR):

• Operating at decoder's highest resolution, combines feature maps, coarse logits, and upsampled RUS uncertainty.
• Uncertainty-derived soft mask $\mathbf{M}$, generated from blurred and thresholded $U^p$, selects regions for refinement.
• A lightweight residual predictor $\mathcal{R}$ produces corrections $\Delta$, applied as

$$P_{\mathrm{fg}}^{\mathrm{ref}} = P_{\mathrm{fg}} + \mathbf{M}\odot\Delta$$

focusing learning and correction on error-prone boundaries and fine details.

Together, these modules leverage the spatially explicit uncertainty prior for adaptive, context-aware segmentation, without additional compute burden on backbone models.

6. Experimental Outcomes and Empirical Significance

Comprehensive ablation studies and benchmarks demonstrate the efficacy of RUS and its integration into UGF and UDLR. Experimental results on the RISBench test split are summarized:

Setting	[email protected]	[email protected]	[email protected]	oIoU	mIoU
Base (CroBIM)	77.55	66.38	34.07	73.04	69.33
+ UGF	77.68	66.72	34.45	73.03	69.44
+ UDLR	77.60	66.93	35.10	73.02	69.53
Full	77.73	67.05	35.52	73.04	69.62

• UGF most benefits cross-modal disambiguation ([email protected]/0.7).
• UDLR favors geometric fidelity and boundary refinement ([email protected], mIoU).
• The full RUS pipeline delivers complementary gains.

On RRSIS-D, [email protected] improves by +1.37 pp and mIoU by +1.02 pp; on RefSegRS, [email protected] and [email protected] both increase, with mIoU rising by +0.31 pp. These results validate that RUS, when plugged into modular inference scaffolds, yields consistent advances in both segmentation precision and contour accuracy.

7. Significance and Interpretive Implications

RUS, as constructed and validated in “CroBIM-U: Uncertainty-Driven Referring Remote Sensing Image Segmentation,” enables explicit modeling of spatial uncertainty in cross-modal remote sensing tasks (Sun et al., 7 Jan 2026). The approach distinctively addresses the spatial non-uniformity of referential alignment, enabling targeted adaptation without architectural modifications to the main model. A plausible implication is that such uncertainty-guided modularity could generalize to other cross-modal dense prediction tasks where local ambiguity is non-uniform and error supervision is available. The empirical improvements confirm the effectiveness of the RUS framework in both robustness and geometric detail preservation, establishing it as a robust spatial prior solution within plug-and-play cross-modal segmentation pipelines.