MedSAMix: SAM-based Medical Segmentation

• MedSAMix is a training-free model merging framework that combines SAM, MedSAM, and MedicoSAM to improve medical image segmentation.
• It employs a novel layer-wise strategy using heterogeneous merging operators like TIES, SLERP, and Linear to balance generalist and specialist behaviors.
• The framework achieves significant Dice score improvements across 25 segmentation tasks without modifying the underlying network architecture.

MedSAMix is a SAM-family medical image segmentation framework whose primary use in the literature represented here is as a training-free model merging method that combines a generalist foundation model such as SAM with specialist medical variants such as MedSAM and MedicoSAM into a single segmentation network (Yang et al., 14 Aug 2025). Its stated purpose is to improve both domain-specific accuracy and cross-task generalization without gradient-based retraining. In related usage, the name is also attached in some descriptions to SAM-Mix, a distinct multitask framework that uses classifier-derived Grad-CAM prompts for annotation-efficient medical segmentation (Ward et al., 2024). The term therefore denotes a family resemblance rather than a single technical recipe: the common theme is the combination of SAM-derived priors with medical-task adaptation, but the mechanisms differ substantially.

1. Nomenclature and scope

The name “MedSAMix” is associated with two distinct strands in the supplied literature. The later and more explicit usage is the model-merging framework introduced in 2025 (Yang et al., 14 Aug 2025). An earlier usage appears in the description of SAM-Mix, where “MedSAMix (SAM-Mix)” is used for a multitask, SAM-based framework for low-annotation liver segmentation (Ward et al., 2024).

Usage	Paper	Core mechanism
MedSAMix	(Yang et al., 14 Aug 2025)	Training-free layer-wise model merging of SAM-family models
SAM-Mix / “MedSAMix (SAM-Mix)”	(Ward et al., 2024)	Multitask classification-segmentation with Grad-CAM-derived prompts

This naming overlap can produce a common misconception that MedSAMix is a single architecture. The supplied papers instead indicate two different design paradigms. The model-merging MedSAMix is post-hoc and training-free (Yang et al., 14 Aug 2025), whereas SAM-Mix is a jointly trained multitask segmentation system with an auxiliary classifier, automatic prompt generation, and LoRA-adapted SAM (Ward et al., 2024).

2. Problem formulation and rationale

The central problem addressed by MedSAMix is the tension between specialization and generalization in universal medical image segmentation. Generalist models such as SAM exhibit strong zero-shot generalization but are not specialized for medical image characteristics. Fine-tuned medical variants such as MedSAM and MedicoSAM improve performance on some medical tasks but can underperform the original SAM on others, reflecting model bias, catastrophic forgetting, heterogeneous training data, scarce annotations, and distributional shifts (Yang et al., 14 Aug 2025).

The paper frames the main question as how to enhance domain-specific performance on a given medical task while preserving, or improving, generalization across many different tasks without retraining (Yang et al., 14 Aug 2025). Its answer is to treat multiple SAM-derived models as residing in compatible loss basins and to search for a merged parameterization that inherits useful properties from each source model.

This places MedSAMix in contrast with other SAM-based medical segmentation lines. SAM-Mix uses image-level labels, Grad-CAMs, automated box prompts, and LoRA adaptation to obtain annotation-efficient liver segmentation from abdominal CT (Ward et al., 2024). SemiSAM uses a frozen SAM-Med3D as an auxiliary pseudo-label teacher within a semi-supervised 3D V-Net framework (Zhang et al., 2023). PG-SAM uses Medical LLM-generated fine-grained text, CLIP alignment, a Fine-Grained Modality Prior Aligner, and a decoder with multi-level feature fusion and iterative mask refinement for prompt-free multi-organ CT segmentation (Zhong et al., 23 Mar 2025). MedSAMix differs from all three because it does not alter the training objective of a downstream segmentor; it algebraically recombines existing weights after training (Yang et al., 14 Aug 2025).

3. Architecture and model-merging formulation

MedSAMix assumes a pool of SAM-style models with a shared architecture. In the reported setup, the candidate pool consists of SAM, MedSAM, and MedicoSAM (Yang et al., 14 Aug 2025). All candidate models share the same major components:

• an image encoder with a ViT backbone and $l=12$ transformer layers,
• a prompt encoder with $k=4$ convolutional downsampling layers,
• a mask decoder with $z=2$ transformer layers and $p=3$ transposed convolutional layers (Yang et al., 14 Aug 2025).

The framework merges parameters across all of these modules. Rather than using a single global merge rule, it partitions the encoder, prompt encoder, and decoder into groups controlled by granularity parameters $g_{\text{enc}}$, $g_{\text{prompt}}$, and $g_{\text{dec}}$. The number of groups is defined as

$$G = \left\lceil \frac{l}{g_{\text{enc}}} \right\rceil +\left\lceil \frac{k}{g_{\text{prompt}}} \right\rceil +\left\lceil \frac{z+p}{g_{\text{dec}}} \right\rceil.$$

Each group receives its own merging method $d_i$ and method-specific hyperparameters $\mathbf{h}_i$, yielding a configuration

$k=4$0

This layer-group formulation is the principal structural feature of MedSAMix. It allows different depths and modules to inherit different proportions of “generalist” and “specialist” behavior, rather than forcing a single compromise over the entire network (Yang et al., 14 Aug 2025).

The search space includes four merging operators (Yang et al., 14 Aug 2025). Task Arithmetic defines task vectors $k=4$1 and constructs

$k=4$2

TIES-Merging keeps the top-$k=4$3 largest-magnitude task-vector entries, forms a consensus sign vector, averages only updates consistent with that sign, and then adds the result back to the base with scale $k=4$4. SLERP performs spherical linear interpolation between model weights,

$k=4$5

with $k=4$6. Linear merging forms a convex combination,

$k=4$7

A key technical implication is that MedSAMix does not merely “average models.” It searches over heterogeneous merge rules across depth. The appendix example reported in the supplied material includes TIES for patch embedding, SLERP for position embedding, and a mixture of TIES, Task Arithmetic, Linear, and SLERP across encoder, prompt encoder, and decoder layers (Yang et al., 14 Aug 2025). This suggests that the framework treats architectural depth as a locus of model bias rather than a uniform parameter field.

4. Zero-order optimization and deployment regimes

MedSAMix is explicitly training-free in the sense that it uses no gradient descent, no backpropagation, and no parameter updates from data; it manipulates existing weights and evaluates candidate merged models only through forward passes (Yang et al., 14 Aug 2025). The optimization problem is written as

$k=4$8

where $k=4$9 is the merged model induced by configuration $z=2$0, $z=2$1 is the full search space, and $z=2$2 is an objective computed on calibration data.

Two regimes are defined (Yang et al., 14 Aug 2025). In the single-task regime, intended for domain-specific deployment, the objective is

$z=2$3

where $z=2$4 is the calibration set for a target task. The resulting model is termed MedSAMix-S.

In the multi-objective regime, intended for universal deployment, the framework optimizes across multiple tasks $z=2$5 using ParEGO. With per-task objectives $z=2$6, the scalarized objective is

$z=2$7

with $z=2$8, $z=2$9, and small $p=3$0 such as $p=3$1 (Yang et al., 14 Aug 2025). The max term emphasizes the worst weighted task, while the sum term smooths the search landscape. The resulting model is termed MedSAMix-M.

Search is conducted with SMAC3, a Bayesian optimization framework using a Random Forest surrogate and Expected Improvement as acquisition function (Yang et al., 14 Aug 2025). The reported budgets are 120 trials for single-task optimization and 200 trials for multi-task optimization. Granularity parameters $p=3$2 are also searched in the range 1–4 (Yang et al., 14 Aug 2025).

This search formulation is important because it distinguishes MedSAMix from manually configured model-merging baselines. The framework’s central claim is not merely that merging helps, but that automatic discovery of layer-wise merge recipes is necessary because simple global recipes are often suboptimal (Yang et al., 14 Aug 2025).

5. Evaluation, empirical behavior, and computational profile

MedSAMix is evaluated on 25 medical segmentation tasks spanning multiple anatomies and modalities, including brain, abdominal CT, nasal and head-and-neck CT, pelvis, cardiac CT, rodent CT, and fundus tasks (Yang et al., 14 Aug 2025). The datasets are split into 80% test/evaluation and 20% calibration, and all experiments are conducted in 2D slice space for fairness across models. SAM-family methods are used in a zero-shot setting with inferred bounding boxes as prompts, and the primary metric is the Dice coefficient (Yang et al., 14 Aug 2025).

The main quantitative findings reported for MedSAMix are as follows (Yang et al., 14 Aug 2025):

Setting	Average Dice	Reported gain
SAM	70.81%	—
MedSAM	63.32%	—
MedicoSAM	72.97%	—
MedSAMix-S	79.64%	+6.67 over MedicoSAM
MedSAMix-M	77.34%	+4.37 over MedicoSAM

For single-task optimization, MedSAMix-S achieves an average Dice score of 79.64% across the 25 tasks, compared with 70.81% for SAM, 63.32% for MedSAM, 72.97% for MedicoSAM, and 67.38% for SAM-Med2D (Yang et al., 14 Aug 2025). The reported average improvement over the best SAM-family baseline, MedicoSAM, is 6.67 percentage points. Illustrative task-level examples include brain tumor from 73.17% with MedicoSAM to 78.36% with MedSAMix-S, nasal pharynx from 84.68% to 89.01%, and optic disk from 93.70% with MedicoSAM and 94.01% with MedSAM to 95.17% with MedSAMix-S (Yang et al., 14 Aug 2025).

For multi-task optimization, MedSAMix-M achieves 77.34% average Dice and outperforms existing universal models on 18 out of 25 tasks (Yang et al., 14 Aug 2025). In comparisons against fixed global merge rules, TIES attains 66.31%, Task Arithmetic 72.19%, Linear merging 74.00%, and SLERP 75.28%. The gap between these baselines and MedSAMix indicates that heterogeneous layer-wise merging, rather than any single merge operator, drives the main performance gain (Yang et al., 14 Aug 2025).

Ablative results refine that picture. In the single-task regime, performance is reported as relatively insensitive to layer granularity, suggesting redundancy in parameter groupings. In the multi-task regime, performance is more sensitive, with moderate granularity such as $p=3$3 giving the best results (Yang et al., 14 Aug 2025). Multi-task optimization also improves as the number of search tasks increases up to about 8 tasks, after which performance declines, which the paper attributes to the difficulty of maintaining Pareto-optimal trade-offs across too many objectives (Yang et al., 14 Aug 2025). Among model subsets, SAM + MedicoSAM gives the best pairwise performance, but adding MedSAM still improves final performance, suggesting that even a weaker individual model can supply complementary parameters at particular depths (Yang et al., 14 Aug 2025).

The computational profile is a practical part of the method’s identity. A reported single-task search uses 120 trials, 2 GPUs, about 8 GB each, and takes about 70 minutes, or about 1.5 minutes per trial per GPU. A multi-task search uses 200 trials, 4 GPUs, and takes about 20 hours (Yang et al., 14 Aug 2025). Once search is complete, deployment uses a single merged model with essentially the same inference cost as a standard SAM-based model because the architecture and parameter count are unchanged (Yang et al., 14 Aug 2025).

6. Relation to other SAM-based medical segmentation methods

MedSAMix belongs to a broader ecosystem of SAM-based medical segmentation, but it occupies a distinct methodological position. The difference becomes clear when compared with three representative approaches supplied alongside it.

SAM-Mix is an annotation-efficient multitask learning framework for liver segmentation from abdominal CT that couples a ResNet-38 auxiliary classifier with a SAM-based segmentation branch. The classifier produces Grad-CAM activation maps, which are thresholded and converted into bounding box prompts for SAM. The segmentation branch uses the image encoder of SAM (ViT-base), a prompt encoder, a mask decoder, and LoRA adaptation with rank $p=3$4. On LiTS, SAM-Mix-50 uses only 50 labeled 2D slices, described as 0.04% of the available labeled training data, trains for 10 epochs, and achieves DS = 0.948 \pm 0.002 and HD = 9.842 \pm 0.046, outperforming fully supervised U-Net, nnU-Net, and TransUNet baselines trained for 100 epochs (Ward et al., 2024). The same configuration achieves DS = 0.923 \pm 0.004 and HD = 11.164 \pm 0.021 on the cross-domain TotalSegmentator liver task, a reported +25.4% Dice improvement over U-Net and 62.26% HD reduction (Ward et al., 2024). MedSAMix, by contrast, does not learn prompts, Grad-CAMs, or LoRA adapters; it merges existing model weights (Yang et al., 14 Aug 2025).

SemiSAM addresses the extreme low-label 3D regime by adding a SAM-assisted consistency term to Mean Teacher or UA-MT frameworks with a 3D V-Net backbone. A frozen SAM-Med3D receives automatically generated prompts derived from student predictions, produces pseudo-labels, and regularizes the student through a consistency loss. On the Left Atrium dataset, SemiSAM-MT improves Dice from 30.60 to 41.38 with 1 labeled case, from 58.17 to 69.46 with 2 labeled cases, and from 72.40 to 80.42 with 4 labeled cases (Zhang et al., 2023). This is another “mixing” strategy, but the mixture is between a task-specific SSL network and a frozen foundation-model teacher, not between multiple SAM-family parameter sets.

PG-SAM advances a third paradigm: prompt-free, multimodal SAM adaptation. It uses a Medical LLM to generate fine-grained organ descriptions, CLIP encoders to process text and image features, a Fine-Grained Modality Prior Aligner (FGMPA) to compute a Semantic Guide Matrix $p=3$5, and a decoder composed of Multi-Level Feature Fusion (MLFF) and an Iterative Mask Optimizer (IMO) (Zhong et al., 23 Mar 2025). On Synapse, PG-SAM reports mDice = 84.79 and HD95 = 7.61 in the fully supervised setting, with an ablation from 72.80 without FGMPA, MLFF, or IMO to 84.79 with all three modules (Zhong et al., 23 Mar 2025). Here again the route to improvement is architectural redesign and multimodal guidance rather than post-hoc parameter synthesis.

A plausible implication is that the term “MedSAMix” now spans at least two conceptual families: one centered on post-hoc model composition (Yang et al., 14 Aug 2025), and another, seen in related naming and adjacent methods, centered on jointly learned guidance mechanisms such as CAM-based prompts, pseudo-label teachers, or text priors (Ward et al., 2024, Zhang et al., 2023, Zhong et al., 23 Mar 2025). The supplied literature therefore supports a broad encyclopedia view in which MedSAMix is best understood as a SAM-based medical segmentation direction organized around the integration of complementary priors, but with sharply different implementation strategies.

7. Limitations and prospective developments

The limitations explicitly reported for the model-merging MedSAMix are structural rather than task-specific (Yang et al., 14 Aug 2025). Although the evaluation spans 25 tasks, the authors note that these cannot represent all medical imaging applications. The framework also depends on models with identical architectures and shared initialization; architecturally modified variants such as adapter-based models are not directly mergeable under the same procedure. In the multi-task setting, performance depends on the choice of representative calibration tasks, so poor calibration-task selection can bias the merged model. The zero-order optimizer, based on a Random Forest surrogate and ParEGO, is effective but may miss the global optimum in a high-dimensional search space (Yang et al., 14 Aug 2025).

Several forward directions are suggested. The framework could be extended to a more diverse model pool so long as architectural alignment is maintained; it could be integrated with federated learning, where models trained at different hospitals are merged without data sharing; and it could be combined with gradient-based fine-tuning, either by merging first and then lightly tuning, or vice versa (Yang et al., 14 Aug 2025). The authors also identify more rigorous study of robustness and distribution shifts as an open problem.

Related methods highlight different unresolved issues. SAM-Mix remains limited to single-organ, 2D, binary CT liver segmentation, with no detailed ablation of threshold $p=3$6, LoRA rank, or classifier-versus-joint training contribution (Ward et al., 2024). SemiSAM is strongest in the 1–4 labeled volume regime but can provide only small gains or even slight degradation when label ratios are higher, because SAM guidance may then become less useful or detrimental (Zhang et al., 2023). PG-SAM is evaluated only on Synapse, CT, and 2D slices, and its performance depends on the quality of Medical LLM-generated descriptions and CLIP-SAM alignment (Zhong et al., 23 Mar 2025).

Taken together, these limitations suggest that “MedSAMix” is not a settled architecture but an evolving research category. In one branch it denotes a training-free, zero-order-optimized merge of generalist and specialist SAM-family models (Yang et al., 14 Aug 2025); in another it names or evokes training-based systems that combine SAM with auxiliary medical priors such as CAMs, pseudo-labels, or fine-grained text (Ward et al., 2024, Zhang et al., 2023, Zhong et al., 23 Mar 2025). The unifying research agenda is the same: reducing the gap between SAM’s broad generalization and the task-specific demands of medical segmentation.