MergeSAM: Object-Based Change Detection

• MergeSAM is an unsupervised method that uses SAM-generated masks to form multitemporal object regions from high-resolution remote sensing images.
• Its MaskMatching and MaskSplitting modules accurately align and decompose objects to capture complex changes like merging, splitting, and partial transformations.
• Empirical results on GZ_CD_data demonstrate approximately 7% F1 improvement over baselines with enhanced precision and better false detection control.

MergeSAM is an unsupervised change detection method for high-resolution remote sensing images built on the Segment Anything Model (SAM). It converts a bitemporal image pair into a set of multitemporal object masks, then performs change analysis on those masks as object-structured units rather than relying only on pixelwise differencing. Its defining contribution is a pair of strategies, MaskMatching and MaskSplitting, designed to handle complex real-world changes such as object splitting, object merging, partial change, and irregular structural transformation across time. In the original formulation, MergeSAM is training-free, uses SAM in zero-shot automatic mask generation mode, and embeds land-cover spatial structure into the change detection process (Hu et al., 30 Jul 2025).

1. Problem formulation and conceptual basis

MergeSAM addresses unsupervised change detection on co-registered high-resolution remote sensing images acquired at two times, denoted $x_1$ and $x_2$. In this setting, no labeled training data are used. The target phenomena include new buildings, demolition or rebuilding, farmland conversion, urban expansion, land-cover fragmentation, and boundary shifts caused by seasonal appearance, registration noise, or illumination variation. The method is motivated by the observation that high-resolution scenes often contain complex object evolution that is poorly captured by direct pixel differencing or by one-to-one comparison of masks generated independently at each date (Hu et al., 30 Jul 2025).

The method uses SAM because SAM provides zero-shot segmentation, automatic mask generation, and transferable visual representations. MergeSAM does not fine-tune SAM. Instead, SAM is used as an off-the-shelf object proposal and feature extraction engine. This is a central point: MergeSAM is not a supervised adaptation method and does not introduce a training loss for SAM itself. A common misconception is to read it as a SAM fine-tuning framework for remote sensing; the paper instead presents a training-free pipeline built around SAM-generated masks and SAM image encoder features (Hu et al., 30 Jul 2025).

The conceptual move is to construct multitemporal homogeneous regions that better reflect the spatial structure of land cover across both dates. This replaces the individual pixel as the sole unit of comparison with a multitemporal object-region representation. The paper frames this as a response to the limitations of prior SAM-based unsupervised change detection, particularly AnyChange, which directly uses masks from each image but has difficulty with split, merge, and irregular structural changes (Hu et al., 30 Jul 2025).

2. Pipeline and use of SAM

The MergeSAM pipeline begins by proportionally scaling each image so that the longest side is resized to 1600. SAM is then run independently on each temporal image in automatic mask generation mode. In the absence of explicit prompts, SAM automatically generates a uniform grid of points over the image as implicit prompts to extract masks for all objects in the scene. The reported automatic mask generator settings are: points per side 64, NMS threshold 0.7, predicted IoU threshold 0.5, and stability score threshold 0.8 (Hu et al., 30 Jul 2025).

For the two dates, SAM produces one mask set per image and also yields image encoder embeddings, denoted $e_1$ and $e_2$. The notation for the two mask sets is inconsistent in the paper’s OCR-extracted text, but the intended meaning is unambiguous: one segmentation mask set is generated for $x_1$ and another for $x_2$. These masks and embeddings are then processed in two stages. First, MaskMatching identifies masks that likely correspond to the same object across time. Second, MaskSplitting reorganizes unmatched but spatially intersecting masks into multitemporal regions that capture more complex transformations (Hu et al., 30 Jul 2025).

The output of these stages is a comprehensive mask set $M_s$. Each mask in $M_s$ acts as an analysis unit. For each such unit, MergeSAM computes an average feature representation from the encoder embeddings at time 1 and time 2, measures their discrepancy using Mean Squared Error (MSE), and uses the Otsu algorithm to produce the final binary change map. The method therefore combines object segmentation, object correspondence, and feature-space comparison within a single unsupervised pipeline (Hu et al., 30 Jul 2025).

3. MaskMatching and MaskSplitting

MaskMatching is the first explicit mechanism for multitemporal alignment. Its purpose is to identify masks from the two dates that correspond to the same object despite slight displacement, boundary inconsistency, registration error, or solar-angle-induced variation. For mask pairs $(M_{11,i}, M_{12,j})$, MergeSAM computes intersection-over-union and regards the pair as matched when

$$\mathrm{IoU}(M_{11,i}, M_{12,j}) = \frac{|M_{11,i} \cap M_{12,j}|}{|M_{11,i} \cup M_{12,j}|} > T_{\mathrm{IOU}}, \qquad T_{\mathrm{IOU}} = 0.75.$$

This matching rule is unsupervised and threshold-based. The paper does not specify a more elaborate assignment procedure, such as bipartite optimization or tie-breaking logic. The significance of MaskMatching is that it suppresses false changes caused by mask boundary inconsistency rather than actual land-cover change (Hu et al., 30 Jul 2025).

After matched masks are removed, the remaining residual masks from the two dates are examined by MaskSplitting. This is the more distinctive component of MergeSAM. It targets cases where whole-mask comparison is too coarse, including one-to-many splitting, many-to-one merging, partial change within a larger object, and irregular transformation of a land parcel. The method considers residual masks that spatially intersect and splits those intersecting masks into multitemporal subregions. The exact geometric splitting operator is not formalized in the paper, but the stated intent is clear: the overlapping and non-overlapping portions of unmatched residual masks are decomposed into more meaningful analysis units (Hu et al., 30 Jul 2025).

The matched masks from MaskMatching and the split-derived multitemporal masks from MaskSplitting are then combined into the comprehensive mask set $x_2$0. This two-stage correspondence model is important. Stable objects are handled through high-IoU direct matching, while structurally transformed objects are handled through intersection-based decomposition. This suggests that MergeSAM’s name refers not to parameter merging or checkpoint interpolation, but to the merging of multitemporal object information into a structured change analysis representation (Hu et al., 30 Jul 2025).

4. Change representation and empirical results

Once $x_2$1 is constructed, each mask is treated as a homogeneous analysis unit. The paper states that the average feature within each mask serves as the feature embedding and MSE is used as the similarity measurement between dates. The exact printed MSE formula is not provided, but the operational meaning is that region-level SAM encoder features are pooled within each multitemporal mask and compared across time. The resulting score distribution is thresholded with the Otsu algorithm to produce the final changed-versus-unchanged decision (Hu et al., 30 Jul 2025).

Experiments are reported on GZ_CD_data, a high-resolution remote sensing change detection dataset collected from 2006 to 2019 in suburban Guangzhou, China. It contains 20 pairs of very high resolution images with spatial resolution $x_2$2, and image sizes ranging from $x_2$3 to $x_2$4. The compared unsupervised baselines are CVA, CVA+SAM, and AnyChange, evaluated with SAM backbones ViT-B, ViT-L, and ViT-H. The reported computational setup uses an NVIDIA 2080 GPU (Hu et al., 30 Jul 2025).

Across all three backbones, MergeSAM achieves the best overall results. With ViT-B, it reports F1 31.65, Precision 20.80, Recall 66.18, OA 74.45, and Kappa 20.89. With ViT-L, it reports F1 31.59, Precision 21.01, Recall 63.64, OA 75.36, and Kappa 20.96. With ViT-H, it reports F1 30.13, Precision 20.37, Recall 57.81, OA 76.03, and Kappa 19.48. By comparison, AnyChange reports F1 values of 24.84, 25.29, and 25.83 for ViT-B, ViT-L, and ViT-H respectively. The paper therefore emphasizes an F1 improvement of about 7 percentage points over AnyChange, together with higher precision and a better balance between false detections and missed detections (Hu et al., 30 Jul 2025).

The evidence for the two core strategies is indirect rather than ablation-based. The paper does not provide a separate numerical ablation table isolating MaskMatching and MaskSplitting individually. Their contribution is inferred from the overall improvement over CVA+SAM and AnyChange and from qualitative examples, particularly parcel-consistent results on farmland and urban scenes (Hu et al., 30 Jul 2025).

5. Position within the broader literature on SAM “merging”

Within the broader SAM literature, MergeSAM belongs to a family of methods that combine SAM with additional structure rather than using SAM as a standalone segmenter. That family is technically heterogeneous. In "SAMDA: Leveraging SAM on Few-Shot Domain Adaptation for Electronic Microscopy Segmentation," SAM is merged with nnUNet at the deepest embedding level, where projected SAM features are concatenated with the last UNet encoder feature and decoded jointly; the method is explicitly described as embedding-space fusion plus adaptor-based domain adaptation rather than prompt-based SAM use (Wang et al., 2024).

Other works use “merge” in different senses. "MemorySAM: Memorize Modalities and Semantics with Segment Anything Model 2 for Multi-modal Semantic Segmentation" treats different sensor modalities as a sequence of frames, reuses SAM2’s memory mechanism across modalities, and averages modality-specific predictions, making it a sequential memory-based multimodal merging strategy rather than an object-matching pipeline (Liao et al., 9 Mar 2025). "Remote SAMsing: From Segment Anything to Segment Everything" addresses large remote-sensing scenes by contextual padding and a parameter-free best-match merge that stitches SAM2 masks across tile boundaries, so its merging operation is spatial reconstruction across tiles rather than cross-time change analysis (Carvalho et al., 30 Apr 2026).

A separate line of work extends SAM by unifying or specializing tasks. "Segment and Matte Anything in a Unified Model" adds a Multi-View Localization Encoder, a Localization Adapter, and dual task-specific heads so that segmentation and matting masks can be generated simultaneously in one frozen-SAM-based architecture (Fan et al., 17 Jan 2026). "SAMRI: Segment Anything Model for MRI" keeps SAM’s image encoder and prompt encoder frozen and fine-tunes only the mask decoder on 1.1 million labeled MR slices, showing that a decoder-only specialization can reach a mean Dice of 0.87 while reducing training time by 94% and trainable parameters by 96% versus full-model retraining (Wang et al., 30 Oct 2025). "AMA-SAM: Adversarial Multi-Domain Alignment of Segment Anything Model for High-Fidelity Histology Nuclei Segmentation" uses a Conditional Gradient Reversal Layer and a High-Resolution Decoder to make multi-dataset SAM training beneficial rather than harmful under domain shift (Qian et al., 27 Mar 2025). "Group-Mix SAM: Lightweight Solution for Industrial Assembly Line Applications" instead replaces MobileSAM’s ViT-T encoder with a GroupMixFormer encoder, reporting 37.63% fewer parameters and 42.5% fewer FLOPs than MobileSAM with only a marginal mIoU drop on the industrial MALSD dataset (Liang et al., 2024).

This comparison clarifies the technical specificity of MergeSAM. It is not a parameter-space model-merging method, not a prompt-refinement method, and not a SAM fine-tuning strategy. Its contribution is a training-free multitemporal object-construction procedure for remote sensing change detection (Hu et al., 30 Jul 2025).

6. Limitations and interpretation

Several implementation details remain underspecified. The paper does not provide a formal pseudocode algorithm, the exact geometric rule used in MaskSplitting, explicit tie-breaking when multiple masks exceed $x_2$5, or a printed mathematical expression for the MSE score. It also does not state whether mask matching is strictly one-to-one or many-to-many. These omissions matter because the method’s performance depends on how multitemporal masks are constructed from intersecting residual regions (Hu et al., 30 Jul 2025).

The evaluation also leaves some open questions. The reported comparisons are restricted to GZ_CD_data, and the paper does not include a standalone ablation isolating MaskMatching from MaskSplitting. The method likely depends on SAM mask quality in remote sensing scenes, may be sensitive to the IoU threshold $x_2$6, and may lose information through the resize-to-1600 preprocessing step. These latter points are plausible implications rather than explicit experimental findings. The paper itself mainly emphasizes strengths and does not include a dedicated limitations section (Hu et al., 30 Jul 2025).

Even with those caveats, MergeSAM is best understood as a SAM-based unsupervised change detection framework that embeds multitemporal object spatial structure into change analysis. Stable regions are handled through IoU-based correspondence, transformed regions through intersection-based splitting, and final decisions through region-level SAM feature discrepancy and Otsu thresholding. In that sense, its significance lies in shifting unsupervised remote sensing change detection from pixelwise differencing toward object-structured multitemporal reasoning built directly on foundation-model masks and embeddings (Hu et al., 30 Jul 2025).