MURTreeFormer: Robust Tree Segmentation

• The paper introduces MURTreeFormer, a transformer-based multi-modal segmentation network that selectively mitigates patch-level uncertainty for accurate tree cover mapping.
• It integrates a VAE-based reconstruction module and a cross-modal distillation approach within a Swin Transformer encoder to address temporal misalignment and modality degradation.
• Empirical results on Zurich and Shanghai datasets demonstrate superior mIoU and boundary preservation compared to state-of-the-art methods.

MURTreeFormer, short for Multi-modal Uncertainty Robust Tree Cover Segmentation for High-Resolution Remote Sensing Images, is a transformer-based multi-modal semantic segmentation network for robust tree cover mapping in high-resolution remote sensing imagery. It is designed for settings in which one modality is treated as primary and others as auxiliary, with the central problem being aleatoric uncertainty induced by temporal misalignment and modality-specific degradation. The framework explicitly models patch-level uncertainty in auxiliary modalities, reconstructs uncertain auxiliary patches from the primary modality through a VAE-based resampling mechanism, and combines this front-end uncertainty mitigation with a Swin Transformer encoder, a cross-modal distillation module, a gradient magnitude attention module, and a lightweight refinement head (Gui et al., 5 Sep 2025).

1. Problem setting and motivation

MURTreeFormer is situated in the literature on semantic segmentation of multi-modal remote sensing images for tree cover mapping. The motivating observation is that integrating modalities such as optical imagery, light detection and ranging (LiDAR), and synthetic aperture radar (SAR) has shown superior performance over single-modality methods, but that these data are often acquired days or even months apart. During those intervals, vegetation disturbances such as logging and wildfires, as well as variations in imaging quality, may occur, producing temporal misalignment and cross-modal uncertainty that can severely degrade segmentation accuracy in high-resolution imagery (Gui et al., 5 Sep 2025).

The framework therefore targets a specific failure mode of multi-modal fusion rather than multi-modal fusion in general. It assumes that one modality—usually ORS, optical remote sensing—has the highest fidelity or is the most temporally relevant source, while other modalities such as DSM or SAR may be unreliable in localized regions. This suggests a selective correction strategy: uncertainty is not treated as uniformly distributed across the scene, but as concentrated in patch-level discrepancies between primary and auxiliary inputs.

The paper formulates the inputs as a Primary Modality $X^P$ and an Auxiliary Modality $X^A$. Each input is divided into $N$ non-overlapping patches, for example $4 \times 4$ pixels with $N = 16384$ for $512 \times 512$ images. The target task is segmentation of tree cover under substantial temporal and cross-sensor mismatch (Gui et al., 5 Sep 2025).

2. Architectural organization

The main pipeline proceeds as follows: Primary and auxiliary modalities are first processed by the Selective Uncertainty-guided Reconstruction Module (SURM); the resulting features are then passed to a Swin Transformer encoder with the Cross-modal Distillation Module (CDM) inserted at stage 2; decoding is performed by a progressive decoder equipped with a Gradient Magnitude Attention (GMA) module and AlignUnit/DecoderUnit blocks; finally, a Refinement Head (RH) produces the segmentation output together with an auxiliary edge map (Gui et al., 5 Sep 2025).

At the encoder level, the choice of Swin Transformer is motivated by long-range dependency modeling, multi-scale feature extraction, and efficient handling of large inputs typical in remote sensing. The architecture is therefore neither a purely convolutional fusion model nor a generic dual-stream transformer. Its organization reflects a division of labor: SURM handles local cross-modal uncertainty before fusion, CDM enforces intermediate feature alignment, and the decoder focuses on structural refinement of tree crowns and boundaries.

A plausible implication is that the model’s robustness derives from this staged design rather than from any single component in isolation. The ablation summary supports that interpretation: removal of SURM weakens performance in uncertain regions, removal of CDM causes the most pronounced accuracy drop on Zurich, and removal of GMA or RH yields marginal but consistent degradation in detail or boundary quality (Gui et al., 5 Sep 2025).

3. Selective uncertainty-guided reconstruction

SURM is the framework’s core uncertainty-mitigation module. For each patch $i$ in modality $M \in \{P, A\}$, the patch feature is modeled as a multivariate Gaussian:

$$p(z^i|x^i_M) = \mathcal{N}(\mu^i_M, \text{diag}((\sigma^i_M)^2)).$$

The shallow extractors $Le_1$ and $X^A$0 output $X^A$1 and $X^A$2. This probabilistic latent representation makes the uncertainty estimate explicit rather than implicit in deterministic features (Gui et al., 5 Sep 2025).

Patch-wise uncertainty is then scored through an entropy-difference proxy:

$X^A$3

These values are processed by an MLP and softmaxed to form a ScoreMap $X^A$4. The $X^A$5 highest-scoring patches, with $X^A$6 and an example of $X^A$7, are designated uncertain and selected for reconstruction. This selective mechanism is central: the framework does not overwrite the auxiliary modality globally, but only at patches deemed unreliable (Gui et al., 5 Sep 2025).

For those selected patches, PatchRecon performs VAE-based resampling from the primary modality’s learned patch distribution:

$X^A$8

The sampled latent variables are decoded via a small fully connected block, and the auxiliary patch $X^A$9 is replaced with the reconstruction only on the uncertain patches. Training on these patches uses MSE between reconstructed and primary features and KL-divergence between auxiliary and primary patch Gaussian posteriors. The surrogate module losses are their sum (Gui et al., 5 Sep 2025).

A common simplification in multi-modal segmentation is to assume that all modalities should always contribute directly to fusion. MURTreeFormer instead treats auxiliary evidence as conditionally reliable. The reported SURM patch visualizations place uncertain patches in mismatched or corrupted regions, and after reconstruction spatial consistency is restored. This suggests that the model operationalizes uncertainty as a localized repair problem rather than as a global confidence weighting problem.

4. Cross-modal distillation and transformer encoding

After SURM, the architecture uses a Cross-modal Distillation Module (CDM) at encoder stage 2. CDM projects latent features from the primary and auxiliary modalities into a common space and minimizes cosine distance through the distillation loss

$N$0

This encourages the enhanced auxiliary features to more closely match the primary features and reduces holistic modality discrepancies (Gui et al., 5 Sep 2025).

The placement of CDM within the encoder is significant. SURM operates patch-wise and explicitly repairs high-uncertainty auxiliary regions; CDM then acts at the feature level to reduce remaining global mismatch between modalities. The reported ablations indicate that CDM removal causes the most pronounced accuracy drop on Zurich, highlighting the importance of global feature alignment under substantial temporal misalignment between ORS and DSM (Gui et al., 5 Sep 2025).

The encoder backbone is a Swin Transformer, selected for long-range dependencies, multi-scale feature extraction, and efficient processing of large remote-sensing inputs. In this design, transformer encoding is not the source of uncertainty modeling itself. Rather, it supplies the representation space in which the repaired and aligned modalities can be fused effectively.

5. Decoder design: boundary emphasis and fine-grained refinement

The decoder is described as a Tree Cover Super Refinement structure with attention and detail modules. It uses a progressive design in which AlignUnit reduces channels and normalizes concatenated features from encoder stages using a Squeeze-and-Excitation Block for channel recalibration, and DecoderUnit progressively upsamples and refines spatial details via double convolutions and SE blocks (Gui et al., 5 Sep 2025).

The Gradient Magnitude Attention (GMA) module is introduced because tree boundaries are often faint or irregular. GMA computes a luminance map from the primary input; for RGB input, this is the average of $N$1, then inverted and normalized for forest scenes. Sobel filters produce gradient magnitudes

$N$2

and an attention map is then formed through exponential decay and inversion:

$N$3

This attention is multiplied with decoder features, with the stated effect of highlighting spatially continuous boundaries, especially tree crowns, and emphasizing high-gradient regions such as tree edges (Gui et al., 5 Sep 2025).

The Refinement Head (RH) follows the main decoder. Instead of a single upsampling step, RH performs progressive upsampling in two steps, each followed by a $N$4 convolution, batch normalization, and ReLU. It produces dual outputs: a final segmentation map and an auxiliary edge map, with the latter supervised via an edge-focused loss. Its stated function is to recover fine details and spatial granularity so that small fragments and irregular structures typical of tree crowns are sharply delineated (Gui et al., 5 Sep 2025).

The ablations report that removing GMA or RH yields a marginal but consistent reduction in detail and boundary quality, and that RH especially improves fine details though not always mIoU. The corresponding visualizations show GMA attention maps consistently highlighting tree boundaries and RH improving delineation of small or fragmented trees in dense, urban, or heterogeneous scenes (Gui et al., 5 Sep 2025).

6. Data modalities, datasets, and empirical results

The empirical evaluation uses multi-modal datasets from Zurich and Shanghai, both chosen for substantial cross-modal mismatch. The Zurich dataset combines ORS (aerial, 2022, 0.1m downsampled to 1m) with DSM (ALS, 2018, 1m), creating substantial temporal misalignment. The Shanghai dataset combines ORS (satellite, ~0.68m, 2019–2021) with SAR (~3m, same period; upsampled), with higher temporal and modality mismatches (Gui et al., 5 Sep 2025).

Dataset	Modalities	Reported results
Zurich	ORS + DSM	MURTreeFormer: mIoU 93.33%, Tree Cover IoU 91.03%, F1 95.30%; Best baseline (UNetFormer): mIoU 92.32%, Tree IoU 89.71%, F1 94.57%
Shanghai	ORS + SAR	MURTreeFormer: mIoU 85.15%, Tree Cover IoU 83.98%, F1 89.49%; Best baseline: SegFormer (83.67% mIoU), UNetFormer (83.96% mIoU)

These results are presented as evidence that MURTreeFormer significantly improves segmentation performance and reduces the impact of temporally induced aleatoric uncertainty (Gui et al., 5 Sep 2025). The visual results are reported to show that the model captures fragmented, fine tree structures and boundaries more faithfully than competing methods.

The ablation studies provide a more differentiated account of module contributions. SURM removal reduces the ability to handle uncertain regions, particularly under temporal misalignment and modality gaps. CDM removal leads to the largest drop on Zurich, emphasizing the need for global alignment. GMA/RH removal reduces detail or boundary quality. The full model is reported to yield the best results in all tested combinations, which supports the interpretation that the modules are complementary rather than redundant (Gui et al., 5 Sep 2025).

7. Interpretation, scope, and recurrent points of confusion

MURTreeFormer is specifically designed for aleatoric uncertainty associated with real-world acquisition mismatch across modalities. It does not treat uncertainty as an abstract regularizer detached from the imaging pipeline; the motivating cases are concrete, including logging, wildfires, and variations in imaging quality between acquisition times (Gui et al., 5 Sep 2025). A recurrent point of confusion in multi-modal remote sensing is to assume that additional modalities necessarily increase robustness. The paper instead frames auxiliary modalities as potentially harmful unless uncertainty is explicitly modeled and selectively mitigated.

Another important distinction is between local and global discrepancy handling. SURM addresses local patch-level inconsistency through probabilistic latent modeling, top-$N$5 selection, and VAE-based reconstruction from the primary modality, while CDM addresses residual global discrepancy through distillation in a shared feature space. This layered treatment suggests that temporal misalignment manifests at multiple scales and that effective mitigation benefits from combining localized repair with encoder-level alignment.

Within the decoder, GMA and RH are not presented as generic post-processing components. Their stated purpose is tied to the morphology of tree cover, especially faint boundaries, fragmented crowns, and fine-grained spatial detail. The auxiliary edge map in RH further indicates that tree cover segmentation is approached not only as semantic region classification but also as a boundary-sensitive delineation problem (Gui et al., 5 Sep 2025).

Taken together, the framework is best understood as a multi-stage segmentation system for high-resolution remote sensing in which uncertainty estimation, selective feature reconstruction, latent alignment, and boundary-aware decoding are jointly organized around the specific difficulties of tree cover mapping under temporally misaligned multi-modal acquisition.