STSF-Net: Multimodal Change Detection

• The paper introduces STSF-Net, a framework that fuses modality-specific and spatio-temporal common features with semantic priors for multimodal optical-SAR change detection.
• It employs a pseudo-Siamese encoder with dedicated optical and SAR branches, alongside feature interaction and graph structure modeling to suppress pseudo-changes.
• Empirical results on Delta-SN6, BRIGHT, and Wuhan-Het demonstrate up to 3.21% mIoU improvements over state-of-the-art methods, validating its adaptive fusion strategy.

STSF-Net is a framework for multimodal change detection between optical and SAR images, proposed to address the modality gap that causes pseudo-changes when bi-temporal observations are compared directly (Liu et al., 7 Apr 2026). Its central design principle is to jointly model modality-specific features and spatio-temporal common features, then fuse them with semantic priors derived from a pre-trained foundation model. In the reported formulation, STSF-Net targets semantic change detection from co-registered optical-SAR pairs, operates in an encoder-fusion-decoder regime, and is evaluated on Delta-SN6, BRIGHT, and Wuhan-Het, where it is reported to outperform the state of the art by 3.21%, 1.08%, and 1.32% in mIoU, respectively (Liu et al., 7 Apr 2026).

1. Problem setting and conceptual scope

STSF-Net addresses semantic change detection from bi-temporal optical-SAR pairs under strong modality gaps (Liu et al., 7 Apr 2026). The motivating difficulty is that optical imagery and SAR encode different physical properties: optical images emphasize spectral reflectance and texture, whereas SAR emphasizes scattering, roughness, and structural properties. Under this heterogeneity, naive cross-modal subtraction or overly shared latent modeling can confuse modality discrepancy with land-cover change, yielding pseudo-change activations.

The method is designed to resolve a specific tension. If a model over-emphasizes modality-invariant representations, it may suppress the modality-specific cues that reveal genuine changes. If it over-emphasizes modality-specific discrepancies, it may amplify cross-sensor inconsistency rather than semantic change. STSF-Net therefore separates two information sources: modality-specific features, which preserve change-sensitive evidence unique to optical and SAR data, and spatio-temporal common features, which encode cross-modal consistent semantics and structure for pseudo-change suppression (Liu et al., 7 Apr 2026).

The paper writes the input as

$\mathbf{I}=\{\mathbf{I}_{opt},\mathbf{I}_{sar}\},$

and seeks a mapping

$\phi:\mathcal{I}\to\mathcal{P}$

into a fused representation space suitable for multimodal change detection. At each feature scale $i\in\{1,2,3,4\}$, the network extracts modality-specific features $\mathbf{S}_{opt}^i$ and $\mathbf{S}_{sar}^i$, common features $\mathbf{C}_{opt}^i$ and $\mathbf{C}_{sar}^i$, semantic priors $\mathbf{P}_{opt}^i$ and $\mathbf{P}_{sar}^i$, and a prior-guided fused representation $\mathbf{F}_{fuse}^i$ (Liu et al., 7 Apr 2026).

2. Network organization and encoder design

The framework contains four named components: the Modal-Specific Feature Encoder (MSFE), the Spatio-Temporal Common Feature Modeling module (STCFM / STCFE in the text), the SAM2-based Semantic Priors Generator (SPG), and the Prior-Guided Feature Fusion Module (PGFFM) (Liu et al., 7 Apr 2026). These multi-scale components feed a decoder and classifier that produce a dense semantic change map.

The MSFE is built as a pseudo-Siamese architecture with unshared weights. This is a defining property of STSF-Net. It is not a strict Siamese network with shared parameters; rather, the optical and SAR branches are asymmetric because the modalities require different inductive biases. The optical encoder is obtained by fine-tuning SAM2. Each of its four stages contains Adapter blocks and Hiera blocks, and only the Adapter parameters are updated. The role assigned to this branch is to preserve rich semantics and edges in optical imagery while exploiting pretrained visual priors and limiting overfitting. The SAR encoder is a small Swin Transformer trained from scratch, with stage depths

$\phi:\mathcal{I}\to\mathcal{P}$0

Its role is to learn SAR-specific scattering and structural representations without forcing SAR into the same extraction bias as the optical stream (Liu et al., 7 Apr 2026).

For each scale $\phi:\mathcal{I}\to\mathcal{P}$1,

$\phi:\mathcal{I}\to\mathcal{P}$2

These modality-specific features are sent both to differencing operations and to common-feature modeling. The paper states that the fused multi-scale features are then processed by a decoder and classifier, but it does not provide a layer-by-layer decoder specification, an explicit upsampling formulation, or a head equation. What is specified is the architectural type: an encoder-fusion-decoder system for pixel-level semantic change prediction (Liu et al., 7 Apr 2026).

3. Common-feature modeling and prior-guided fusion

The STCFM is the branch responsible for learning spatio-temporal common features that reduce the optical-SAR modality gap while preserving structurally meaningful semantics (Liu et al., 7 Apr 2026). It consists of a Feature Interaction Module (FIM) followed by Graph Structure Feature Modeling (GSFM).

In the FIM, the modality-specific inputs are projected into a common feature space, concatenated, and transformed into an attention map: $\phi:\mathcal{I}\to\mathcal{P}$3 This attention is then used to recalibrate the modality features: $\phi:\mathcal{I}\to\mathcal{P}$4 where $\phi:\mathcal{I}\to\mathcal{P}$5. The text describes this stage as capturing long-range dependencies and temporal correlations through cross-attention, although the published equation is written as an attention map generated from concatenated features (Liu et al., 7 Apr 2026).

The GSFM then reshapes the attended features into a graph $\phi:\mathcal{I}\to\mathcal{P}$6, where spatial locations are nodes and edges encode structural relations. The first graph convolution is

$\phi:\mathcal{I}\to\mathcal{P}$7

A residual difference is introduced as

$\phi:\mathcal{I}\to\mathcal{P}$8

and the second graph convolution refines the result: $\phi:\mathcal{I}\to\mathcal{P}$9 According to the text, $i\in\{1,2,3,4\}$0 is ReLU at this stage. The stated purpose of this branch is to reduce false positives due to sensor differences, improve feature alignment, and increase intra-class compactness. The paper reports T-SNE evidence that, before STCFM, optical and SAR features cluster separately, whereas after STCFM, features of the same semantic category from different modalities become interwoven and more compact (Liu et al., 7 Apr 2026).

Semantic guidance is supplied by the SPG, which is based on SAM2. The prompt encoder and mask decoder are removed, and only the hierarchical backbone is retained. The stage configuration is

$i\in\{1,2,3,4\}$1

Hiera blocks across the four stages, and all SPG parameters are frozen during training. This produces semantic priors

$i\in\{1,2,3,4\}$2

A change prior is then estimated by per-pixel Euclidean distance followed by a shallow CNN $i\in\{1,2,3,4\}$3 and sigmoid normalization: $i\in\{1,2,3,4\}$4 The intended interpretation is that high $i\in\{1,2,3,4\}$5 indicates likely changed regions and low $i\in\{1,2,3,4\}$6 indicates likely unchanged regions (Liu et al., 7 Apr 2026).

The PGFFM then forms two change paths. The specific-path features are

$i\in\{1,2,3,4\}$7

and the common-path features are

$i\in\{1,2,3,4\}$8

Adaptive fusion is defined by

$i\in\{1,2,3,4\}$9

This is the key fusion equation of STSF-Net. In regions with high prior-estimated change likelihood, modality-specific discrepancies are weighted more heavily; in regions with low change likelihood, common-feature differences dominate in order to suppress pseudo-change. The paper further states that weighted features from both paths are concatenated and passed through a convolutional module to produce the final fused feature, again denoted $\mathbf{S}_{opt}^i$0 (Liu et al., 7 Apr 2026).

4. Datasets, supervision regimes, and implementation

STSF-Net is evaluated on three multimodal change detection datasets with different label spaces (Liu et al., 7 Apr 2026). Delta-SN6 is introduced in the paper as the first openly accessible multiclass MMCD benchmark consisting of very-high-resolution fully polarimetric SAR and optical images. It is developed from SpaceNet6 by adding historical optical imagery from 2007, constructing bi-temporal optical and co-registered SAR observations, and providing semantic change annotations. Its characteristics are: resolution 0.5 m, tile size $\mathbf{S}_{opt}^i$1, temporal span 2007–2019, area coverage 120 km$\mathbf{S}_{opt}^i$2, geographic region Port of Rotterdam, Netherlands, and a total of 2,818 finely annotated change instances. The class set is BG, AB, AR, AW, DB, DR, and DW. The split is 50% training, 30% testing, and 20% validation.

BRIGHT is used for multimodal building damage assessment with optical and SAR images. The reported properties are resolution 0.3–1 m, image size $\mathbf{S}_{opt}^i$3, time span 2014–2020, and 4246 image pairs, with splits of 2918 train, 450 validation, and 878 test. The evaluation categories are background, intact, damaged, and destroyed. Wuhan-Het is an urban change detection dataset over Wuhan, China, using Sentinel-2 optical imagery from March 2017 and COSMO-SkyMed StripMap SAR from March 2020, with image size $\mathbf{S}_{opt}^i$4, 552 training samples, 129 test samples, and 112 validation samples. Its task is binary change detection, especially for building and road changes (Liu et al., 7 Apr 2026).

The implementation details that are explicitly given are limited. The hardware is an NVIDIA L20 48GB GPU running Ubuntu 22.04, and the codebase is written in Python. Optimization uses Adam with $\mathbf{S}_{opt}^i$5 iterations, batch size 8, and initial learning rate $\mathbf{S}_{opt}^i$6. The paper does not specify a learning-rate decay schedule, weight decay, warm-up, image normalization values, crop size, augmentation strategy, epoch count, or early stopping policy. It also does not provide explicit loss equations, and there is no evidence in the text for contrastive learning, a consistency regularizer, or a dedicated modality-disentanglement loss (Liu et al., 7 Apr 2026).

The associated code and Delta-SN6 dataset are stated to be released at:

$\mathbf{S}_{sar}^i$4

5. Empirical results, ablations, and efficiency

The reported headline result is that STSF-Net outperforms the state of the art by 3.21%, 1.08%, and 1.32% in mIoU on Delta-SN6, BRIGHT, and Wuhan-Het, respectively (Liu et al., 7 Apr 2026). The paper presents detailed metrics for all three benchmarks.

Dataset	Headline results
Wuhan-Het	Recall 55.25, Precision 60.57, OA 88.71, F1 57.79, mIoU 64.25
BRIGHT	$\mathbf{S}_{opt}^i$7 91.71, $\mathbf{S}_{opt}^i$8 75.83, OA 96.10, mIoU 67.91
Delta-SN6	$\mathbf{S}_{opt}^i$9 94.60, $\mathbf{S}_{sar}^i$0 95.27, OA 99.54, mIoU 91.33

On Wuhan-Het, the paper compares STSF-Net with GSTM-SCD and reports F1 improving from 56.30 to 57.79 and mIoU from 62.93 to 64.25. On BRIGHT, mIoU improves from 66.83 for GSTM-SCD to 67.91, and $\mathbf{S}_{sar}^i$1 from 73.96 to 75.83. On Delta-SN6, the reported per-class IoU values are BG 99.55, AB 85.68, AR 81.42, AW 92.95, DB 92.77, DR 94.62, and DW 92.33, with mIoU 91.33. The qualitative description is that STSF-Net better preserves sharp boundaries, reduces omissions, and separates mixed change types more accurately than prior methods (Liu et al., 7 Apr 2026).

The ablation studies are central to the paper’s empirical argument. Module ablations start from an asymmetric pseudo-Siamese encoder baseline. On BRIGHT, the baseline has mIoU 63.99, adding FIM yields 64.76, adding GSFM yields 65.64, and the full model with PGFFM reaches 67.91. On Delta-SN6, the same sequence is 71.42, 74.45, 84.93, and 91.33. The paper interprets these gains as evidence that FIM improves common-feature extraction and cross-modal interaction, GSFM contributes major structural gains, and PGFFM provides the final boost through semantic-prior-guided adaptive fusion (Liu et al., 7 Apr 2026).

Encoder ablation on Delta-SN6 compares Siamese SwinTransformer, Siamese SAM2, and the proposed asymmetric SAM2 + SwinTransformer design; the proposed encoder gives the best $\mathbf{S}_{sar}^i$2, $\mathbf{S}_{sar}^i$3, OA, and mIoU. Prior-generator ablation further compares w/o PGFFM, PGFFM + SAM, PGFFM + DINOv3, and PGFFM + SAM2. On Delta-SN6, the corresponding mIoU values are 84.93, 88.16, 86.52, and 91.33; on BRIGHT they are 65.64, 66.86, 66.72, and 67.91. The paper also inserts PGFFM into SiamCRNN, DamageFormer, and GSTM-SCD, reporting average mIoU gains of about 1.15% on BRIGHT and about 1.84% on Delta-SN6, which suggests that PGFFM is portable as a cross-modal fusion block (Liu et al., 7 Apr 2026).

The efficiency figures reported for STSF-Net are 63.37 M parameters, 191.42 G FLOPs, and 51.60 ms inference time. The paper places this in a moderate-to-high complexity regime: heavier than some lightweight alternatives, but faster than several competitive models such as SiamAttnUNet. The feature-response analysis attributes STSF-Net’s behavior to a complementary combination: specific features strongly activate true changed objects but are noisy, common features are more spatially coherent but less sensitive, and the fused representation strengthens changed-region activation while suppressing background responses (Liu et al., 7 Apr 2026).

6. Terminological distinctions, limitations, and research position

The name STSF-Net belongs to the optical-SAR multimodal change detection framework introduced in 2026 and should not be conflated with several unrelated uses of the acronym “STSF” in other literatures (Liu et al., 7 Apr 2026). In space-weather time-series classification, STSF denotes Supervised Time Series Forest and is used as a classical interval-based ensemble for short-term classification of strong solar energetic particle events (Rotti et al., 2024). In time-series classification methodology, r-STSF denotes Randomized-Supervised Time Series Forest, an extension of STSF built around supervised interval search, multiple representations, and randomized trees rather than a neural network (Cabello et al., 2021). In graph forecasting, STSF is used as shorthand for spatial-temporal sequence forecasting in an LLM-guided neural architecture search framework, where the contribution is a search controller rather than a fixed architecture named STSF-Net (Xue et al., 23 Mar 2025). In that sense, the term “STSF-Net” is specific to multimodal change detection, not to time-series forests or general spatio-temporal forecasting.

The paper also states several limitations of the proposed system (Liu et al., 7 Apr 2026). STSF-Net mainly focuses on optical-SAR MMCD, and the authors note that they have not yet developed a unified architecture that can seamlessly handle both homogeneous and heterogeneous change detection. Delta-SN6 remains bi-temporal rather than long-term, covers only building, road, and water categories, and contains relatively few road and water samples. The model is not lightweight, and the authors identify adaptive feature selection, structured pruning, knowledge distillation, and lighter unified MMCD frameworks as future directions. Methodologically, the omission of an explicit training-loss formulation and the absence of specified contrastive or consistency constraints limit how precisely one can reconstruct the optimization objective from the paper text.

Within the multimodal remote-sensing literature, the defining claim of STSF-Net is that semantic change should be represented neither by purely modality-invariant features nor by purely modality-specific discrepancies, but by an adaptive fusion of both, modulated by semantic priors from a frozen foundation model (Liu et al., 7 Apr 2026). This suggests a broader design pattern for heterogeneous change detection: preserve modality-specific evidence of true change, learn cross-modal structural consistency to suppress pseudo-change, and use high-level priors to decide where each source should dominate.