SAMed-2: Memory-Enhanced Med Segmentation

• SAMed-2 is a selective memory-enhanced medical image segmentation model that extends SAM-2 with domain-specific innovations to handle noisy, diverse clinical data.
• It incorporates a temporal adapter with 3D convolution to capture volumetric context and a confidence-driven memory mechanism to mitigate catastrophic forgetting.
• Evaluated on MedBank-100k and external datasets, SAMed-2 achieves up to a 10.53% DSC improvement over prior models, ensuring robust multi-modal performance.

SAMed-2 is a selective memory–enhanced medical image segmentation foundation model that extends the Segment Anything Model 2 (SAM-2) with domain-specific architectural and training innovations designed to address the unique challenges of medical imaging. SAMed-2 integrates volumetric/temporal context through a temporal adapter and employs a confidence-driven memory mechanism to selectively store and retrieve high-certainty features, thereby handling both the heterogeneity and continual learning requirements across diverse anatomical structures and modalities in clinical data. Evaluated on the comprehensive MedBank-100k benchmark and numerous external datasets, SAMed-2 achieves state-of-the-art performance relative to prior "segment anything" and medical-domain adaptation models (Yan et al., 4 Jul 2025).

1. Motivation and Challenges in Medical Segmentation

Adapting foundation segmentation models such as SAM-2 to the medical imaging domain is impeded by several factors:

• Noisy and heterogeneous annotations: Medical datasets—spanning CT, MR, ultrasound, endoscopy, X-ray, fundus, and dermoscopy—are characterized by inter-observer variability, low contrast, imaging artifacts, and label noise.
• Support for diverse modalities and anatomies: Unlike natural image datasets, clinical segmentation encompasses dozens of anatomical targets across modalities, making single-task models impractical for universal application.
• Catastrophic forgetting: Sequential fine-tuning of foundation models on new organs or modalities typically degrades performance on earlier tasks unless explicit mechanisms preserve prior knowledge.

SAMed-2 is specifically designed to address these challenges via a unified model architecture and memory-based continual learning strategy (Yan et al., 4 Jul 2025).

2. Model Architecture and Core Innovations

SAMed-2 is architected on top of SAM-2 but introduces two principal enhancements:

Temporal Adapter

A temporal adapter is inserted into each Transformer layer of the image encoder. This adapter captures volumetric or temporal correlations, e.g., adjacent slices in CT or frames in endoscopy videos. The mechanism is as follows:

Given block input $x \in \mathbb{R}^{B \times H \times W \times C}$,

1. $x_{\text{attn}} = \text{MHSA}(\text{LN}(x))$
2. $$x_{\text{out}} = x + \text{DropPath}(\text{TemporalAdapter}(x_{\text{attn}}))$$
3. $$x_{\text{final}} = x_{\text{out}} + \text{MLP}(\text{LN}(x_{\text{out}}))$$

The TemporalAdapter applies a $3$D convolution after dimension-reducing projection and employs a non-linearity (e.g., GeLU) to model spatial–temporal dependencies efficiently (Yan et al., 4 Jul 2025).

Confidence-Driven Memory Mechanism

The dynamic memory bank in SAMed-2 stores tuples $(F_i, PE_i, \hat{y}_i, E_i)$, where $F_i$ is the mask feature embedding, $PE_i$ the positional encoding, $\hat{y}_i$ the predicted IoU, and $E_i$ the image encoder output. During inference:

• Cosine similarity $x_{\text{attn}} = \text{MHSA}(\text{LN}(x))$0 is computed between the current embedding $x_{\text{attn}} = \text{MHSA}(\text{LN}(x))$1 and stored $x_{\text{attn}} = \text{MHSA}(\text{LN}(x))$2.
• Each entry’s retrieval score is $x_{\text{attn}} = \text{MHSA}(\text{LN}(x))$3, combining feature similarity and model confidence.
• The top-K entries are aggregated and fused with the current image features via a dedicated memory attention module.

Memory updates are driven by confidence: new high-confidence features replace existing ones only if they are more certain. This strategy both denoises the support set and mitigates forgetting in continual learning scenarios (Yan et al., 4 Jul 2025).

3. Training Regime and Data Curation

SAMed-2 is trained on MedBank-100k, a diverse reference set comprising 122,594 frame–mask pairs across 21 clinical segmentation tasks and 7 imaging modalities, including CT, MR, X-ray, ultrasound, fundus, dermoscopy, colonoscopy, and echocardiography (Yan et al., 4 Jul 2025). The dataset construction pipeline standardizes aspect ratios, shuffles 2D samples, and preserves 3D/video order to maintain context for volumetric tasks.

The composite loss function is:

• Segmentation loss, $x_{\text{attn}} = \text{MHSA}(\text{LN}(x))$4
• IoU regression, $x_{\text{attn}} = \text{MHSA}(\text{LN}(x))$5
• Total loss, $x_{\text{attn}} = \text{MHSA}(\text{LN}(x))$6

Confidence-driven memory replacement acts as an implicit regularizer; no explicit memory regularization is required (Yan et al., 4 Jul 2025).

4. Quantitative Performance and Experimental Validation

Comprehensive benchmarking is conducted on both internal (MedBank-100k) and 10 external, previously unseen datasets, encompassing ultrasound, fundus, dermoscopy, and more. Representative results (Dice Similarity Coefficient, DSC):

Task	SAMed-2	MedSAM-2	SAM-2	MedSAM	SAM	U-Net
Thyroid (US)	0.8663	0.6756	0.4967	0.8863	0.8655	0.8052
Optic Cup	0.8936	0.3820	0.5589	0.8823	0.6064	0.8833
Melanoma (Derm)	0.9167	0.6601	0.6158	0.9364	0.8322	0.8884
... (7 more tasks)
Average DSC	0.6938	0.5796	0.4375	0.6277	0.5958	0.6879

SAMed-2 outperforms all baselines—including MedSAM-2 (Zhu et al., 2024), MedSAM, and U-Net—in multi-task settings, with up to 10.53% average DSC improvement over MedSAM and top-2 rank placement in every external test (Yan et al., 4 Jul 2025).

In ablation, adding the confidence-driven memory, temporal adapter, and selective retrieval yields consistent performance gains across major organ/tumor benchmarks (Spleen, IVC, Liver, ColonTumor), with largest gains observed for tasks requiring inter-slice context or resilience to annotation noise (Yan et al., 4 Jul 2025).

5. Component Analyses and Ablative Findings

Ablation experiments highlight the critical impact of each architectural component:

• Memory retrieval: Improves mean DSC, especially when combined with confidence-based filtering, by ensuring that only reliable context features support segmentation attention.
• Temporal adapter: Yields substantial gains in volumetric and video tasks by explicitly modeling inter-slice variation, a limitation of single-frame architectures.
• Memory size and retrieval: Larger memory enhances performance but with diminishing returns. Selecting memory entries by confidence and similarity (conf-sim) outperforms random or naive strategies. Adaptive or hierarchical memory could further optimize these trade-offs (Yan et al., 4 Jul 2025).

6. Discussion, Limitations, and Future Directions

SAMed-2's selective memory augments segmentation accuracy by filtering noisy or low-certainty features prevalent in heterogeneous clinical datasets. Its architecture is robust to catastrophic forgetting, facilitating continual learning across new organs and modalities (Yan et al., 4 Jul 2025).

Limitations include computational overhead and latencies associated with large memory retrieval, which may be mitigated by adaptive memory sizing or hierarchical strategies. Current retrieval is not class-aware and may be suboptimal for highly imbalanced multi-class scenarios. Further, full extension to 3D volumetric inference—and seamless real-time video analysis—warrants focused investigation.

A plausible implication is that SAMed-2, by bridging promptable zero-shot capabilities and continual-learning support, sets a precedent for unified foundation models in clinical imaging, superseding both general-purpose "segment anything" frameworks and modality/task-specific architectures.

7. Relation to Prior and Contemporary Approaches

SAMed-2 stands in contrast to approaches such as:

• MedSAM-2 (Zhu et al., 2024): Utilizes a confidence-sorted memory and weighted pick-up strategy derived from SAM-2, but does not integrate a temporal adapter within the encoder or a confidence-driven continual learning mechanism. MedSAM-2 introduces "One-Prompt Segmentation" that allows single-prompt propagation across unordered 2D sets, but is outperformed by SAMed-2 on internal and external benchmarks.
• SAM-2 Applied Directly (Dong et al., 2024): Yields robust results in 2D image segmentation but exhibits variability and rapidly decaying performance on deep volumetric (3D) tasks, especially when model memory exceeds its trained context window (8 frames) and under suboptimal propagation settings.
• MedSAM and U-Net: While well-established in the literature, these architectures lack the promptable, memory-augmented structure, resulting in lower average performance on multi-modal, multi-task evaluation.

SAMed-2 advances the state of the art by systematically addressing the memory, context exploitation, and continual adaptation needs unique to medical image segmentation (Yan et al., 4 Jul 2025).