---
title: 'SAMerging: Unified Model Merging Techniques'
url: https://www.emergentmind.com/topics/samerging
type: topic
---

# SAMerging: Unified Model Merging Techniques

SAMerging refers to a family of model merging techniques that leverage the Segment Anything Model (SAM) or analogous architectures, typically by combining weights, features, or intermediate representations across models to create composite models with improved or generalized capabilities. The "SAMerging" concept spans vision, language, and speech domains, encompassing diverse algorithms for convex model interpolation, sensitivity-guided balancing, and multi-teacher knowledge distillation, often pursued in a training-free or data-efficient regime.

## 1. Foundations and Definitions

SAMerging originated from the need to reconcile the generalization breadth of foundation models like SAM with the specialized performance of fine-tuned or domain-adapted variants. In the context of model merging, the objective is to synthesize a new model whose parameters $\theta_{\mathrm{merge}}$ combine information from multiple source models $\{\theta_t\}_{t=1}^T$ such that, for an input $x$ and a set of evaluation tasks, the resulting outputs achieve superior or at least balanced performance compared to any single constituent parent.

Formally, for models sharing an architecture, the merge space can often be parameterized layer-wise:
$$
W_\ell^{\mathrm{merge}} = \alpha_\ell W_\ell^{(1)} + (1-\alpha_\ell) W_\ell^{(2)},\quad 0 \le \alpha_\ell \le 1,
$$
where $W_\ell^{(1)}$ and $W_\ell^{(2)}$ are weights at layer $\ell$ of models 1 and 2 respectively [2508.11032]. Extensions to multi-model settings and task-arithmetic domains employ tensor-linear or task-vector combinations [2502.12420, 2512.21288].

## 2. Algorithms and Optimization Regimes

### Layer-wise Convex Merging with Zero-Order Search

"MedSAMix" applies a convex linear combination per Transformer layer to merge a generalist SAM with a specialist MedSAM, searching for optimal coefficients $\alpha$ via a zero-order Bayesian optimizer (SMAC with a Random-Forest surrogate). The optimization seeks to maximize segmentation metrics (e.g., Dice coefficient) on a small calibration set, efficiently discovering layer-specific $\alpha_\ell$ [2508.11032].

**Pseudocode Outline for Zero-Order SAMerging**:
```python
# Inputs: parent models, calibration data, max iterations
for t in 1...T_max:
    # Fit surrogate model to current evaluations
    # Select α maximizing expected improvement
    # Synthesize merged model with α
    # Evaluate merged model, update history
return α* with best calibration score
```

### Sensitivity-Guided SAMerging

In large language models, "Sens-Merging" generalizes SAMerging by weighting the contribution of each expert model per layer based on: (1) parameter-level within-task sensitivity, and (2) cross-task transferability. Merging coefficients $\sigma_i^\ell$ are computed via softmax-normalized products of these sensitivities [2502.12420]:
$$
\sigma_i^\ell = \frac{\exp((\tau_i \cdot \alpha_i^\ell)/T)}{\sum_{j=1}^K \exp((\tau_j \cdot \alpha_j^\ell)/T)}
$$
Parameters for layer $\ell$ are merged as:
$$
\theta_{\mathrm{merge}}^\ell = \theta_{\mathrm{pre}}^\ell + \sum_i K \cdot \sigma_i^\ell (\theta_{\mathrm{SFT}}^{(i),\ell} - \theta_{\mathrm{pre}}^\ell)
$$

### Flatness-Aware Knowledge Distillation

"SAMerging" formulated in [2512.21288] leverages Sharpness-Aware Minimization (SAM) in combination with multi-teacher knowledge distillation (KD). Here, each parent model acts as a teacher, and the merged student model is trained to minimize the weighted average KL divergence over unlabeled calibration samples:
$$
L_{\mathrm{KD}}(\theta_{\mathrm{merge}},\beta) = \sum_{t=1}^T \alpha_t\, \mathbb{E}_{x \sim B_t}[\mathrm{KL}(p_t(\cdot\mid x) \| q(\cdot\mid x; \theta_{\mathrm{merge}}))]
$$
A sharpness-aware ascent step ensures the merged solution lies in a wide, robust minimum, promoting generalization and reducing cross-task interference.

## 3. Applications Across Modalities

### Medical Image Segmentation

In medical imaging, MedSAMix merges SAM and MedSAM via convex linear combinations of corresponding layers, automatically tuned through zero-order search [2508.11032]. Two optimization regimes are presented: single-task (for maximal expert accuracy) and multi-task (for balanced generalization across broad domains). MedSAMix consistently outperforms baseline SAM-family models, achieving a +6.67% average Dice gain on 25 specialized tasks, and +4.37% in multi-task scenarios.

### Remote Sensing Change Detection

In unsupervised remote sensing change detection, MergeSAM utilizes SAM for multitemporal object segmentation and invents a deterministic "SAMerging" pipeline: MaskMatching (greedy IoU-based mask alignment) and MaskSplitting (partitioning masks to capture object splits/merges) [2507.22675]. Feature differences between matched/split spatial units are thresholded to yield precise change maps, with substantial F1-score improvements compared to prior unsupervised approaches.

### Speech and Language Model Merging

Selective Attention Merge operates in low-resource ASR by merging only attention-layer task vectors from source (adult) and target (child) models, using exponentially decaying layer-wise mixing factors to balance domain knowledge [2501.08468]. In NLP, Sens-Merging generalizes this approach using parameter sensitivity and transferability, resulting in merged LLMs that can surpass individual fine-tuned experts on general, mathematical, and code-centric benchmarks [2502.12420].

## 4. Empirical Results and Analysis

Quantitative assessments across modalities consistently demonstrate SAMerging approaches outperforming naive baselines and even many gradient-based or data-intensive alternatives.

**Summary of Empirical Gains**:

| Application         | Merging Method | Specialized Metric  | Improvement      |
|---------------------|---------------|--------------------|------------------|
| Medical Segmentation| MedSAMix-S/M  | Dice score         | +6.67% / +4.37%  |
| Remote Sensing      | MergeSAM      | F1-score           | +7% (vs. baselines)|
| Speech (ASR)        | SA Merge      | WER (MyST-Whisper) | up to 14% rel. red.|
| Language Models     | Sens-Merging  | Task avg. acc.     | +5–10 points avg.|

A common theme is mitigation of catastrophic forgetting and task bias. For instance, MedSAMix's layer-wise $\alpha$ assignments frequently emphasize the generalist model in shallower layers and the specialist in deeper layers, balancing transfer and adaptation [2508.11032, 2502.12420]. Empirical ablations reveal scaling trade-offs (task-specific versus cross-task) and the benefits of flat minima, as quantified by PAC-Bayes bounds and empirical landscape visualizations [2512.21288]. Data efficiency is a hallmark: many schemes reach competitive performance with minimal or even no additional labelled data and very limited calibration samples.

## 5. Limitations, Extensions, and Future Directions

While SAMerging methods have demonstrated efficacy and data efficiency, several technical constraints remain:

- Architectural Homogeneity: Most algorithms assume identical or closely related model architectures (e.g., SAM-family, same backbone for LLMs or SFMs). Extending to heterogenous ensembles will require new alignment or adapter-based mechanisms [2508.11032].
- Limited Modalities: Medical and remote sensing experiments have focused mainly on 2D images; extending merging to 3D volumetric or multi-modal inputs is not yet fully explored [2508.11032].
- Hyperparameter Tuning: Methods such as SA Merge and Sens-Merging involve per-layer or per-domain hyperparameters whose optimal selection may require task-specific calibration [2501.08468].
- Nonlinear and Dynamic Merging: Current strategies are predominantly linear or convex; prospective work identifies non-linear gating, input-dependent merging ($\alpha(x)$), or graph-based region partitioning for vision as fruitful avenues [2508.11032, 2507.22675].
- Calibration Data Constraints: Although data requirements are minimal, merging quality remains linked to the representativeness and quality of the small calibration sets used for coefficient determination [2512.21288].

## 6. Theoretical Guarantees and Geometric Analysis

A distinguishing thread in recent SAMerging developments is the integration of theoretical generalization guarantees—specifically, PAC-Bayes flatness-aware bounds on excess risk in the multi-task setting [2512.21288]. The "cross-task heterogeneity" term in these bounds motivates both the allocation of merging weights to maximize transfer and the use of sharpness-aware optimization to encourage broad, low-sensitivity minima. These contributions underpin the high empirical data efficiency and robustness observed in practice and distinguish SAMerging from heuristic or parameter-only arithmetic baselines.

---

SAMerging unifies a range of principled model combination techniques centered on foundation architectures such as SAM. These strategies, spanning zero-order search, sensitivity analysis, and flatness-aware optimization, systematically enable composite models to recover both the generalization of base models and the task specialization of fine-tuned variants. Diverse empirical results demonstrate state-of-the-art data efficiency and robustness across modalities, with theoretical frameworks increasingly supporting empirical practice [2508.11032, 2502.12420, 2512.21288, 2507.22675, 2501.08468].

Source: https://www.emergentmind.com/topics/samerging