SAMerging: Unified Model Merging Techniques

• SAMerging is a suite of model merging techniques that combine weights, features, and representations from SAM-based models to balance specialization and generalization.
• It employs methods like layer-wise convex merging with zero-order search, sensitivity-guided balancing, and flatness-aware knowledge distillation to optimize performance.
• Applications span medical image segmentation, remote sensing, and speech/language tasks, achieving significant metric improvements with minimal calibration data.

SAMerging refers to a family of model merging techniques that leverage the Segment Anything Model (SAM) or analogous architectures, typically by combining weights, features, or intermediate representations across models to create composite models with improved or generalized capabilities. The "SAMerging" concept spans vision, language, and speech domains, encompassing diverse algorithms for convex model interpolation, sensitivity-guided balancing, and multi-teacher knowledge distillation, often pursued in a training-free or data-efficient regime.

1. Foundations and Definitions

SAMerging originated from the need to reconcile the generalization breadth of foundation models like SAM with the specialized performance of fine-tuned or domain-adapted variants. In the context of model merging, the objective is to synthesize a new model whose parameters $\theta_{\mathrm{merge}}$ combine information from multiple source models $\{\theta_t\}_{t=1}^T$ such that, for an input $x$ and a set of evaluation tasks, the resulting outputs achieve superior or at least balanced performance compared to any single constituent parent.

Formally, for models sharing an architecture, the merge space can often be parameterized layer-wise:

$$W_\ell^{\mathrm{merge}} = \alpha_\ell W_\ell^{(1)} + (1-\alpha_\ell) W_\ell^{(2)},\quad 0 \le \alpha_\ell \le 1,$$

where $W_\ell^{(1)}$ and $W_\ell^{(2)}$ are weights at layer $\ell$ of models 1 and 2 respectively (Yang et al., 14 Aug 2025). Extensions to multi-model settings and task-arithmetic domains employ tensor-linear or task-vector combinations (Liu et al., 18 Feb 2025, Dalili et al., 24 Dec 2025).

2. Algorithms and Optimization Regimes

Layer-wise Convex Merging with Zero-Order Search

"MedSAMix" applies a convex linear combination per Transformer layer to merge a generalist SAM with a specialist MedSAM, searching for optimal coefficients $\alpha$ via a zero-order Bayesian optimizer (SMAC with a Random-Forest surrogate). The optimization seeks to maximize segmentation metrics (e.g., Dice coefficient) on a small calibration set, efficiently discovering layer-specific $\alpha_\ell$ (Yang et al., 14 Aug 2025).

Pseudocode Outline for Zero-Order SAMerging:

$\{\theta_t\}_{t=1}^T$6

Sensitivity-Guided SAMerging

In LLMs, "Sens-Merging" generalizes SAMerging by weighting the contribution of each expert model per layer based on: (1) parameter-level within-task sensitivity, and (2) cross-task transferability. Merging coefficients $\sigma_i^\ell$ are computed via softmax-normalized products of these sensitivities (Liu et al., 18 Feb 2025):

$\{\theta_t\}_{t=1}^T$0

Parameters for layer $\{\theta_t\}_{t=1}^T$1 are merged as:

$\{\theta_t\}_{t=1}^T$2

Flatness-Aware Knowledge Distillation

"SAMerging" formulated in (Dalili et al., 24 Dec 2025) leverages Sharpness-Aware Minimization (SAM) in combination with multi-teacher knowledge distillation (KD). Here, each parent model acts as a teacher, and the merged student model is trained to minimize the weighted average KL divergence over unlabeled calibration samples:

$\{\theta_t\}_{t=1}^T$3

A sharpness-aware ascent step ensures the merged solution lies in a wide, robust minimum, promoting generalization and reducing cross-task interference.

3. Applications Across Modalities

Medical Image Segmentation

In medical imaging, MedSAMix merges SAM and MedSAM via convex linear combinations of corresponding layers, automatically tuned through zero-order search (Yang et al., 14 Aug 2025). Two optimization regimes are presented: single-task (for maximal expert accuracy) and multi-task (for balanced generalization across broad domains). MedSAMix consistently outperforms baseline SAM-family models, achieving a +6.67% average Dice gain on 25 specialized tasks, and +4.37% in multi-task scenarios.

Remote Sensing Change Detection

In unsupervised remote sensing change detection, MergeSAM utilizes SAM for multitemporal object segmentation and invents a deterministic "SAMerging" pipeline: MaskMatching (greedy IoU-based mask alignment) and MaskSplitting (partitioning masks to capture object splits/merges) (Hu et al., 30 Jul 2025). Feature differences between matched/split spatial units are thresholded to yield precise change maps, with substantial F1-score improvements compared to prior unsupervised approaches.

Speech and LLM Merging

Selective Attention Merge operates in low-resource ASR by merging only attention-layer task vectors from source (adult) and target (child) models, using exponentially decaying layer-wise mixing factors to balance domain knowledge (Shankar et al., 14 Jan 2025). In NLP, Sens-Merging generalizes this approach using parameter sensitivity and transferability, resulting in merged LLMs that can surpass individual fine-tuned experts on general, mathematical, and code-centric benchmarks (Liu et al., 18 Feb 2025).

4. Empirical Results and Analysis

Quantitative assessments across modalities consistently demonstrate SAMerging approaches outperforming naive baselines and even many gradient-based or data-intensive alternatives.

Summary of Empirical Gains:

Application	Merging Method	Specialized Metric	Improvement
Medical Segmentation	MedSAMix-S/M	Dice score	+6.67% / +4.37%
Remote Sensing	MergeSAM	F1-score	+7% (vs. baselines)
Speech (ASR)	SA Merge	WER (MyST-Whisper)	up to 14% rel. red.
LLMs	Sens-Merging	Task avg. acc.	+5–10 points avg.

A common theme is mitigation of catastrophic forgetting and task bias. For instance, MedSAMix's layer-wise $\{\theta_t\}_{t=1}^T$4 assignments frequently emphasize the generalist model in shallower layers and the specialist in deeper layers, balancing transfer and adaptation (Yang et al., 14 Aug 2025, Liu et al., 18 Feb 2025). Empirical ablations reveal scaling trade-offs (task-specific versus cross-task) and the benefits of flat minima, as quantified by PAC-Bayes bounds and empirical landscape visualizations (Dalili et al., 24 Dec 2025). Data efficiency is a hallmark: many schemes reach competitive performance with minimal or even no additional labelled data and very limited calibration samples.

5. Limitations, Extensions, and Future Directions

While SAMerging methods have demonstrated efficacy and data efficiency, several technical constraints remain:

• Architectural Homogeneity: Most algorithms assume identical or closely related model architectures (e.g., SAM-family, same backbone for LLMs or SFMs). Extending to heterogenous ensembles will require new alignment or adapter-based mechanisms (Yang et al., 14 Aug 2025).
• Limited Modalities: Medical and remote sensing experiments have focused mainly on 2D images; extending merging to 3D volumetric or multi-modal inputs is not yet fully explored (Yang et al., 14 Aug 2025).
• Hyperparameter Tuning: Methods such as SA Merge and Sens-Merging involve per-layer or per-domain hyperparameters whose optimal selection may require task-specific calibration (Shankar et al., 14 Jan 2025).
• Nonlinear and Dynamic Merging: Current strategies are predominantly linear or convex; prospective work identifies non-linear gating, input-dependent merging ($\{\theta_t\}_{t=1}^T$5), or graph-based region partitioning for vision as fruitful avenues (Yang et al., 14 Aug 2025, Hu et al., 30 Jul 2025).
• Calibration Data Constraints: Although data requirements are minimal, merging quality remains linked to the representativeness and quality of the small calibration sets used for coefficient determination (Dalili et al., 24 Dec 2025).

6. Theoretical Guarantees and Geometric Analysis

A distinguishing thread in recent SAMerging developments is the integration of theoretical generalization guarantees—specifically, PAC-Bayes flatness-aware bounds on excess risk in the multi-task setting (Dalili et al., 24 Dec 2025). The "cross-task heterogeneity" term in these bounds motivates both the allocation of merging weights to maximize transfer and the use of sharpness-aware optimization to encourage broad, low-sensitivity minima. These contributions underpin the high empirical data efficiency and robustness observed in practice and distinguish SAMerging from heuristic or parameter-only arithmetic baselines.

---

SAMerging unifies a range of principled model combination techniques centered on foundation architectures such as SAM. These strategies, spanning zero-order search, sensitivity analysis, and flatness-aware optimization, systematically enable composite models to recover both the generalization of base models and the task specialization of fine-tuned variants. Diverse empirical results demonstrate state-of-the-art data efficiency and robustness across modalities, with theoretical frameworks increasingly supporting empirical practice (Yang et al., 14 Aug 2025, Liu et al., 18 Feb 2025, Dalili et al., 24 Dec 2025, Hu et al., 30 Jul 2025, Shankar et al., 14 Jan 2025).