CroBIM-U: Uncertainty-Driven Remote Sensing

• CroBIM-U is an uncertainty-driven framework for pixel-wise segmentation in remote sensing imagery that uses a dynamic spatial risk prior to guide language-vision fusion.
• It features a specialized Referring Uncertainty Scorer which generates per-pixel uncertainty maps to effectively address referential ambiguities and fine-grained boundary errors.
• Empirical evaluations on public benchmarks show that CroBIM-U significantly improves mean IoU and precision metrics, ensuring robust performance in challenging dense and variable-scale scenes.

CroBIM-U (Cross-modal BIM with Uncertainty) is an uncertainty-driven framework for referring remote sensing image segmentation, targeting the problem of pixel-wise localization of objects described by natural language within complex aerial or satellite imagery. CroBIM-U departs from prior uniform fusion architectures by leveraging a pixel-wise spatial “risk prior”—an uncertainty map—constructed online and used to modulate both language–vision fusion and local refinement. This approach explicitly addresses challenges of spatial non-uniformity in referential ambiguity, which is prevalent due to extreme scale variations, dense similar distractors, and fine-grained boundary structures in overhead imagery. CroBIM-U demonstrates substantial improvements in both robustness and geometric fidelity over previous state-of-the-art methods, while preserving architectural flexibility by acting as a plug-and-play extension atop standard remote sensing segmentation stacks (Sun et al., 7 Jan 2026).

1. Remote Sensing Referring Segmentation and Spatial Non-Uniformity

Referring remote sensing image segmentation (RRSIS) is defined as the task of, given an aerial image $I$ and a natural language query $S$, producing a binary mask $\hat{Y}$ marking the pixels corresponding to the described object. RRSIS is demanding due to three main factors: (a) frequent co-occurrence of visually similar objects (e.g., buildings, ships), (b) extreme variations in scale, orientation, and appearance among targets and distractors, and (c) complex, low-contrast, or thin boundaries.

Traditional solutions for this task employ two-tower cross-modal architectures with uniform language–vision fusion and mask refinement, treating all pixels as equally uncertain. Such uniformity inadvertently introduces unnecessary linguistic perturbations to visually unambiguous regions, while failing to provide adequate disambiguation or correction in regions that are ambiguous or error-prone. This failure to model the substantial spatial non-uniformity of cross-modal alignment reliability is a primary motivation for CroBIM-U.

2. Pixel-Wise Referring Uncertainty Map

The central innovation of CroBIM-U is the explicit, pixel-level referring uncertainty map $U(x, y)$, which encodes (for each pixel) the degree of current model uncertainty or risk regarding referential alignment. This map is produced at stride-32 resolution as a logit tensor $$U \in \mathbb{R}^{B \times 1 \times H_u \times W_u}$$, with a probabilistic version $$U^p = \sigma(U) \in [0, 1]^{B \times 1 \times H_u \times W_u}$$, where $\sigma$ denotes the sigmoid function.

$U$ is resized via bilinear interpolation to match the spatial resolution of any backbone feature map $H \times W$, yielding $\tilde{U} = \mathrm{Resize}(U; H, W)$ and $\tilde{U}^p = \sigma(\tilde{U})$. For each pixel $(x, y)$, the uncertainty score is thus $U^p(x, y) = \sigma\big(\tilde{U}(x, y)\big)$.

This uncertainty map serves as a dynamic spatial prior throughout the rest of the CroBIM-U pipeline: it directs inference to modulate the influence of language (in relatively ambiguous regions) and to concentrate boundary correction where the risk of error is high.

3. Architecture: Referring Uncertainty Scorer (RUS)

The Referring Uncertainty Scorer (RUS) is a plug-and-play module that estimates the uncertainty map $U$ given the coarsest visual feature tokens $\mathbf{V} \in \mathbb{R}^{B \times N_u \times C}$ and text tokens $\mathbf{T} \in \mathbb{R}^{B \times L \times C}$. The process consists of linear projections,

$$\mathbf{V}' = \mathbf{V} W_v, \quad \mathbf{T}' = \mathbf{T} W_t,$$

followed by scaled cross-attention,

$$\mathbf{A} = \mathrm{Softmax}\big(\gamma\,\mathbf{V}'\,{\mathbf{T}'}^\top\big), \quad \widetilde{\mathbf{T}} = \mathbf{A} \mathbf{T}',$$

and a multilayer perceptron to predict logit outputs per token,

$$\mathbf{u} = \mathrm{MLP}\big([\mathbf{V}; \widetilde{\mathbf{T}}]\big) \in \mathbb{R}^{B \times N_u \times 1}.$$

Reshaping yields $$U \in \mathbb{R}^{B \times 1 \times H_u \times W_u}$$. Here, $\gamma$ is a learnable scaling factor and MLP combines the feature representations.

Training of RUS is performed via an online error-consistency supervision strategy. At each step, the current mask prediction $$\hat{y}(x) = \mathbb{I}(\sigma(P_{\mathrm{fg}}(x)) > 0.5)$$ is compared with ground-truth $y(x)$, yielding an error map $e(x) = \mathbb{I}(\hat{y}(x) \neq y(x))$. This error map is optionally blurred for target smoothing, producing $z(x)$. The uncertainty map $U$ is supervised via a weighted per-pixel BCEWithLogits loss: $$\mathcal{L}_{\mathrm{unc}} = \frac{1}{BHW} \sum_{b=1}^B \sum_{x} w_b^{\mathrm{sample}} w_{b,x}^{\mathrm{pixel}} \operatorname{BCEWithLogits}\big(\tilde{U}_b(x), z_b(x)\big).$$ The weightings balance between sample-level difficulty (IoU) and pixel-level error class imbalance.

This approach forces the uncertainty map to highlight precisely the spatially localized regions where segmentation and grounding currently fail, enabling targeted adaptation by downstream modules.

4. Plug-and-Play Modules: UGF and UDLR

CroBIM-U utilizes the uncertainty map through two downstream mechanisms—Uncertainty-Gated Fusion (UGF) and Uncertainty-Driven Local Refinement (UDLR)—without modifications to the backbone architecture.

Uncertainty-Gated Fusion (UGF): At each decoder stage, UGF modulates the language-to-vision update via an uncertainty-guided gating mechanism. For an input-level visual token set $\mathbf{V}$, the module computes the standard cross-attention update $\Delta \mathbf{V}$, and gates it with

$\mathbf{g} = \sigma(\alpha \mathbf{u} + \beta),$

where $\mathbf{u}$ is the resized uncertainty logit, and $\alpha$, $\beta$ are learned scalars. The final update is

$$\mathbf{V}^+ = \operatorname{LayerNorm}(\mathbf{V} + \mathbf{g} \odot \Delta \mathbf{V}).$$

This enables full language grounding in ambiguous (high-uncertainty) regions, while suppressing unnecessary language influence in confident areas.

Uncertainty-Driven Local Refinement (UDLR): For local segmentation refinement, UDLR exploits the soft uncertainty mask

$$\mathbf{M} = \operatorname{StopGrad}\left[\sigma\left((\operatorname{Blur}(\tilde{U}^p) - \tau)/t\right)\right],$$

where $\tau$ is a threshold and $t$ is a temperature. A lightweight convolutional head $\mathcal{R}(\cdot)$ predicts a residual $\Delta$, and the mask logit map is updated: $$P_{\mathrm{fg}}^{\mathrm{ref}} = P_{\mathrm{fg}} + \mathbf{M} \odot \Delta.$$ This focuses refinement capacity strictly on error-prone regions, especially boundaries.

5. Training Regimen and Benchmark Evaluation

CroBIM-U employs ConvNeXt-Base (SMLFR pretrained) as the visual backbone and BERT-Base (12-layer) as the text encoder. Images are resized to $480\times480$ pixels; text is truncated to 20 tokens. Optimization is conducted with AdamW (initial learning rate $5\times10^{-5}$, weight decay $0.01$, polynomial decay) over 40 epochs on 8 $\times$ A800 GPUs, batch size 32. The total loss integrates segmentation, uncertainty, and optional refinement terms: $$\mathcal{L} = \mathcal{L}_{\mathrm{seg}} + \lambda_{\mathrm{unc}} \mathcal{L}_{\mathrm{unc}} + \lambda_{\mathrm{ref}} \mathcal{L}_{\mathrm{ref}}$$ with $\lambda_{\mathrm{unc}}=1$, $\lambda_{\mathrm{ref}}=0.5$ (or 0 otherwise).

Three public datasets—RefSegRS, RRSIS-D, and RISBench—are used for evaluation, each presenting challenges in scale, density, and boundary complexity.

Quantitative results demonstrate that CroBIM-U consistently surpasses prior state-of-the-art in mean IoU (mIoU) and precision-at-X (Pr@X) metrics:

Dataset	mIoU	[email protected]	[email protected]
RRSIS-D	65.07	75.60	24.16
RefSegRS	60.08	—	6.57
RISBench	69.62	77.55	35.52

Ablation studies indicate both UGF and UDLR contribute complementary gains: UGF improves moderate-threshold precision; UDLR notably strengthens high-threshold precision and overall mIoU (Sun et al., 7 Jan 2026).

6. Limitations and Future Research Directions

CroBIM-U’s online RUS training produces noisy targets at early stages, requiring a warm-up period. While UGF and UDLR are designed to be lightweight, their cumulative compute and memory overhead is nonzero. At present, CroBIM-U models only a single error-likelihood map; a more nuanced decomposition of uncertainty—e.g., distinguishing aleatoric from epistemic uncertainty, or separately modeling grounding- versus boundary-focused risk—could enhance adaptivity. Extensions to video or multi-temporal inputs, and to expressive geospatial queries involving relational chains, are identified as promising avenues for further investigation.

A plausible implication is that risk-prior-based control of cross-modal fusion and refinement can be generalized to broader settings in remote sensing and multimodal segmentation.

7. Summary and Significance

CroBIM-U offers a unified, interpretable architecture for spatially adaptive cross-modal inference in remote sensing image segmentation. By making pixel-level uncertainty a central organizing principle—both in fusion and refinement—CroBIM-U achieves enhanced robustness and geometric precision across challenging benchmarks. Its plug-and-play design, error-consistency supervision, and clear empirical gains establish a new reference approach for uncertainty-guided multimodal remote sensing systems (Sun et al., 7 Jan 2026).