Siamese U-Net: Paired Image Segmentation

• Siamese U-Net is a deep learning framework that extends U-Net by processing paired images with shared encoders to extract comparable relational features.
• It employs cross-image correlation and multi-scale dense decoding to enhance boundary sensitivity and improve segmentation in tasks like change detection and co-segmentation.
• Empirical studies show that Siamese U-Net variants achieve state-of-the-art precision, Jaccard, and F1-scores on datasets such as iCoseg, Internet, and Sen1Flood11.

Searching arXiv for relevant papers on Siamese U-Net and closely related variants to ground the article. Siamese U-Net denotes a family of U-Net-based encoder–decoder architectures extended to process paired inputs through parallel, typically weight-sharing branches, so that the model learns a comparable feature space before performing segmentation, co-segmentation, or change detection. In the clearest formulation, two related images are encoded by the same feature extractor, their representations are compared or fused, and a decoder reconstructs a dense prediction for the target structure shared across the pair or changed between acquisition times (Liu et al., 2021). In remote sensing, the same principle is framed as a U-Net-based architecture for multi-temporal imagery, in which “before” and “after” images are processed in parallel and their features are compared or fused to localize change (Amin et al., 9 Jun 2025). Across the literature represented here, Siamese U-Net is therefore less a single canonical model than a design pattern: shared encoders, paired inputs, cross-stream interaction, and a decoder that converts joint evidence into a pixel-wise output.

1. Architectural definition and distinguishing characteristics

The defining architectural property of Siamese U-Net is the use of two parallel streams with shared weights. In the image co-segmentation formulation of “A Dense Siamese U-Net trained with Edge Enhanced 3D IOU Loss for Image Co-segmentation” (Liu et al., 2021), the network has three parts: “the first part is a Siamese encoder which consists of two identical ResNet50 networks with shared parameters for feature extraction. The second part is the correlation block, through which the correlation maps are calculated from the two feature maps. The third part is a Siamese decoder network that consists of a hierarchy of decoders, each corresponding to an encoder layer.” This explicit decomposition captures the core Siamese U-Net principle: symmetric feature extraction, cross-image comparison, and decoder-based dense reconstruction.

Relative to a standard U-Net, the principal difference is not merely the duplication of the encoder but the introduction of an explicit pairwise comparison mechanism. Standard U-Net is organized around a single-image encoder/contracting path, a decoder/expanding path, and skip connections. By contrast, Siamese U-Net is designed for pairwise input and learns change-sensitive or commonality-sensitive representations rather than segmenting one image independently (Amin et al., 9 Jun 2025). This distinction is especially important in tasks where the target depends on relational evidence between two observations, such as discovering the common object in two images or isolating changed regions in multi-temporal satellite scenes.

The literature represented here also shows that “Siamese U-Net” is not restricted to one fusion rule or one encoder class. The shared-encoder, paired-input template appears with convolutional backbones such as ResNet50 (Liu et al., 2021), with dual-stream SAR-specific designs augmented by attention (Yadav et al., 2022), and with transformer-based encoders in remote sensing change detection, exemplified by Siamese Swin-U-Net (Amin et al., 9 Jun 2025). This suggests that the Siamese aspect is architectural and relational, whereas the specific feature extractor and fusion mechanism are task-dependent.

2. Encoder coupling, cross-stream interaction, and decoding

In the co-segmentation setting, Siamese U-Net is explicitly formulated as a paired-input, shared-weights segmentation problem. Two images are encoded by the same network weights so that both are mapped into a comparable feature space; a correlation module then computes correspondence between the two feature streams; the decoder reconstructs the dense prediction (Liu et al., 2021). The symmetry imposed by parameter sharing is central, because it constrains both inputs to be represented under the same feature semantics. This is what makes the model “Siamese” in the strict sense used in the paper: the network is intended to discover commonality between two related images rather than perform two unrelated segmentations.

The same paired-stream logic appears in multi-temporal flood detection. “Attentive Dual Stream Siamese U-net for Flood Detection on Multi-temporal Sentinel-1 Data” (Yadav et al., 2022) uses two Siamese encoders, one for the pre-flood image and one for the post-flood image, with shared weights. At several spatial scales—$256 \times 256$, $128 \times 128$, $64 \times 64$, and $32 \times 32$—the corresponding feature maps are enhanced and weighted channel-wise by attention, then fused using concatenation before being passed to a U-Net-like decoder. The decoder uses convolution, upsampling, padding, and normalization, and the overall design implies multi-scale feature propagation from encoder to decoder.

A recurring architectural choice is that fusion happens at multiple levels rather than only at the bottleneck. In the flood-detection model, the decoder receives fused representations containing appearance and context from each date together with implicit change cues (Yadav et al., 2022). In the co-segmentation model, the decoder is explicitly aligned with the encoder hierarchy and reconstructs a dense representation after correlation analysis (Liu et al., 2021). A plausible implication is that Siamese U-Net derives much of its effectiveness from preserving temporal or relational evidence throughout the full encoder–decoder hierarchy rather than compressing that evidence into a single late-stage comparison.

3. Dense decoder variants and feature reuse

One important architectural development is the introduction of dense connections in the decoder path. In the dense Siamese U-Net of (Liu et al., 2021), the main modification relative to a plain U-Net is that “the output of each module will be passed to all the following modules,” so each decoder block aggregates not only the corresponding encoder feature map but also outputs from earlier decoder stages. When a previous feature map has a different spatial resolution, it is transformed to the current resolution using “deconvolution, batch normalization and rectified linear unit (ReLU),” after which all available features are fused by concatenation.

This decoder design differs materially from ordinary U-Net skip connections. Instead of reusing only the feature map at the matching encoder level, the dense decoder more aggressively reuses information from multiple earlier decoding stages (Liu et al., 2021). The paper states that this design is intended to better exploit multi-scale information, preserve spatial detail, improve gradient flow, and reduce information loss in the decoder, particularly near object boundaries. These functions are consistent with the broader role of dense connectivity in encoder–decoder segmentation systems, but in this context the stated motivation is specifically to improve co-segmentation quality.

The dense-decoder idea also clarifies a common misconception. Siamese U-Net is not synonymous with “two encoders plus a standard decoder.” The dense Siamese U-Net paper makes clear that the decoder itself can be a major site of innovation, not merely a downstream reconstruction module (Liu et al., 2021). By contrast, in the flood-detection model the emphasis falls more heavily on attention-enhanced multi-scale fusion, while in the comparative study of U-Net variants for change detection the Siamese principle is presented primarily as a temporal-comparison mechanism that can be combined with other modifications, such as Swin Transformer blocks (Amin et al., 9 Jun 2025). Accordingly, Siamese U-Net should be understood as a compositional framework rather than a fixed layer-by-layer recipe.

4. Output representations and training objectives

A major divergence within Siamese U-Net research concerns what the network predicts. The dense co-segmentation model does not predict a binary mask directly. Instead, each ground-truth mask is converted into a signed normalized distance map (SNDM), and the network regresses that representation (Liu et al., 2021). For each pixel, the Euclidean distance to the nearest boundary pixel is computed, normalized, and assigned a sign: positive for foreground and negative for background. The paper defines

$$M(x) = \left\{ \begin{array}{cc} -\frac{\max(D(x)) + 1 - D(x)}{\max(D(x))}, & x \in background \ \frac{\max(D(x)) + 1 - D(x)}{\max(D(x))}, & x \in foreground \end{array} \right.$$

where

$D(x) = \min_{\forall b \in B} d(x,b),$

and $B$ is the set of boundary points (Liu et al., 2021). Because of the sign convention, segmentation is recovered by checking the sign of each predicted value: positive corresponds to foreground and negative to background.

The paper emphasizes that SNDM carries richer shape information than a binary mask, and that there is a “mutation from -1 to 1 around the object boundary,” which helps distinguish foreground from background around blurry edges (Liu et al., 2021). To reduce ambiguous near-zero predictions, the SNDM range is linearly transformed to $[0.1,1]$ for foreground and $[-1,0.1]$ for background. Supervision is then imposed through an edge-enhanced 3D IoU loss defined over the SNDM volume rather than over a binary plane. The loss treats the SNDM as a “3D shape,” introduces a penalty for sign mismatch through a factor $F$ with $128 \times 128$0, and further emphasizes boundary regions by weighting pixels proportionally to the labeled SNDM value; the authors report that the best value in their experiments is $128 \times 128$1 (Liu et al., 2021). Training is performed end-to-end with Adam.

By contrast, the flood-detection Siamese U-Net treats the problem as binary segmentation and uses the combined objective

$128 \times 128$2

to address class imbalance (Yadav et al., 2022). This objective is paired with a bi-temporal Sentinel-1 input representation in which each image is encoded with three channels—VV, VH, and a blank third channel of zeros—because the model uses a pre-trained 3-channel backbone. The contrast between these two papers shows that Siamese U-Net does not prescribe a single output space: it can support direct mask prediction or regression of a structured geometric surrogate such as SNDM, depending on the task and the desired inductive bias.

5. Application domains and representative instantiations

The applications represented here cluster around two main domains: image co-segmentation and change detection. In image co-segmentation, the task is to sample a pair of related images and segment the common object in both. The dense Siamese U-Net addresses this by pairing shared-weight encoding, cross-image correlation, dense decoding, and SNDM regression (Liu et al., 2021). The resulting architecture is presented as more than a standard two-branch U-Net: it is a correlation-based, dense-decoder system trained with a loss designed to reward correct signed distance structure and boundary sensitivity.

In flood mapping, the same Siamese principle is adapted to bi-temporal SAR change detection. The attentive dual-stream Siamese U-net processes pre-flood and post-flood Sentinel-1 acquisitions with shared encoders, enhances the resulting features with Concurrent Spatial and Channel Squeeze & Excitation (scSE) blocks, and fuses them by concatenation before decoding (Yadav et al., 2022). The model is evaluated on Sen1Flood11, which the authors extend into a bi-temporal dataset by constructing pre-flood images from Sentinel-1 acquisitions from the year before the flood date using Google Earth Engine and computing the pixel-wise median of those historical acquisitions. This is presented as a significant part of the paper’s novelty.

The broader remote-sensing literature positions Siamese U-Net as a change-detection-oriented extension of U-Net for paired multi-temporal imagery (Amin et al., 9 Jun 2025). The comparative study explicitly highlights Siamese Swin-U-Net, which replaces the standard convolutional encoder with Swin Transformer blocks inside a Siamese network. The reported rationale is that shifted window-based self-attention captures long-range dependencies and scene-wide spatial relationships, which are useful for remote sensing change detection (Amin et al., 9 Jun 2025). Within that study, the distinguishing property of Siamese U-Net remains its design around two input times and feature comparison, rather than a generic increase in segmentation capacity.

Two additional works delineate the boundaries of the term. ASFM-Net is described as a Siamese point-cloud completion network with a U-Net-like coarse-to-fine refinement stage, but not as a literal 2D Siamese U-Net (Xia et al., 2021). Barlow-Swin, similarly, uses a Siamese shared-encoder setup only during Barlow Twins pretraining and becomes a standard single-input segmentation network at inference time (Haftlang et al., 8 Sep 2025). These cases are useful because they show that “Siamese” and “U-Net-like” can coexist outside classical dual-image segmentation, but they are best treated as related architectures rather than direct instances of Siamese U-Net.

6. Empirical results, benefits, and limitations

The co-segmentation evidence in (Liu et al., 2021) attributes measurable gains to both dense decoder connectivity and SNDM-based supervision. On the Internet dataset, the traditional Siamese U-Net baseline from Gong et al., with no dense decoder and trained with traditional IoU loss, obtains average Precision $128 \times 128$3 and Jaccard $128 \times 128$4. Adding the dense decoder while still predicting binary masks and using traditional Dice loss improves the result to $128 \times 128$5. The full model—dense decoder plus edge-enhanced 3D IoU loss of SNDM—reaches average Precision $128 \times 128$6 and Jaccard $128 \times 128$7 (Liu et al., 2021). The same paper reports an average Jaccard of $128 \times 128$8 on the iCoseg subset, surpassing Gong et al. at $128 \times 128$9, and on SBCoseg reports average Precision $64 \times 64$0 and Jaccard $64 \times 64$1. The authors state that these results lead to state-of-the-art performance on the Internet and iCoseg datasets.

The flood-detection evidence points to the value of bi-temporal pairing and attention-enhanced fusion. On Sen1Flood11, the attentive dual-stream Siamese U-net reports IoU $64 \times 64$2 and F1-score $64 \times 64$3, compared with IoU $64 \times 64$4 for Uni-temporal DL Method 1 and F1-score $64 \times 64$5 for Uni-temporal DL Method 2 S1 (Yadav et al., 2022). The paper states that the proposed method achieves $64 \times 64$6 better IoU than the uni-temporal method of Bai et al. and $64 \times 64$7 better F1-score than the uni-temporal method of Konapala et al. This supports the claim that bi-temporal SAR inputs provide richer change information than uni-temporal inputs when coupled to an effective Siamese encoder–decoder design.

The comparative study of U-Net variants for satellite-image change detection adds a higher-level perspective on strengths and trade-offs (Amin et al., 9 Jun 2025). Its discussion emphasizes better handling of multi-temporal images, more effective feature matching through shared encoders, and, in the case of Siamese Swin-U-Net, improved modeling of long-range dependencies. At the same time, it notes broader limitations relevant to Siamese change-detection systems: computational complexity and the persistent challenge of separating true changes from nuisance variation such as illumination changes, seasonal differences, different viewing angles, registration errors, noise, and sensor differences. The paper does not present explicit equations for the Siamese fusion step, which underscores a general point: many practical Siamese U-Net systems are defined as much by engineering decisions about fusion, alignment, and training as by a single formal derivation.

Overall, the evidence here supports a concise synthesis. Siamese U-Net provides the paired-image framework; dense decoder designs, attention mechanisms, or transformer encoders modify how cross-image evidence is exploited; and task-specific supervision—binary overlap losses, focal reweighting, or SNDM-based geometric losses—strongly influences final performance (Liu et al., 2021). This suggests that the term identifies a research direction centered on shared-weight relational segmentation, rather than a single immutable architecture.