SAM-Med3D: 3D Medical Segmentation Model

• SAM-Med3D is a 3D medical image segmentation model that processes entire volumetric scans using a fully learnable 3D Vision Transformer framework.
• The model integrates advanced prompt and mask decoders to achieve superior Dice and IoU scores on 247 anatomical and pathological categories across CT, MRI, and ultrasound.
• SAM-Med3D streamlines clinical and research workflows by reducing interactive prompt requirements and enabling robust, zero-shot segmentation for diverse imaging tasks.

SAM-Med3D is a general-purpose segmentation foundation model built for volumetric (3D) medical image analysis, extending the Segment Anything Model (SAM) paradigm originally introduced for promptable 2D image segmentation. Unlike task-specific segmentation networks or slice-based adaptations, SAM-Med3D processes entire 3D volumes in a unified framework, achieving high performance with a minimal number of prompts across diverse anatomical targets and imaging modalities. The underlying architecture leverages a fully learnable 3D Vision Transformer backbone and prompt-driven mask decoder, trained on a large-scale heterogeneous medical dataset comprising tens of thousands of annotated volumetric scans. SAM-Med3D serves both as a stand-alone segmenter and as a foundational initialization for downstream adaptation, enabling clinical and research workflows in diagnosis, annotation, and planning (Wang et al., 2023).

1. Motivation and Distinguishing Features

Medical image segmentation is essential for quantitative diagnosis, treatment planning, and biomedical research. Existing approaches predominantly rely on organ- or lesion-specific 3D architectures trained for narrow tasks, yielding limited generalizability. Early attempts to apply SAM to medical imaging (SAM-Med2D, slice-by-slice variants) showed promise but suffered from loss of volumetric continuity, excessive prompt requirements, and poor zero-shot transfer (Bui et al., 2023). SAM-Med3D was developed to overcome these deficits by:

• Redesigning the image, prompt, and mask encoder/decoder modules for native 3D processing.
• Training from scratch on a curated multi-modality dataset, extending coverage to 247 anatomical and pathological categories.
• Providing prompt efficiency, semantic diversity, and downstream transfer capabilities as a 3D foundation model.

This approach enables far fewer interactive inputs (3D points) to segment entire volumes, is robust to modality shifts (CT, MRI, ultrasound), and supports generalization to unseen structures.

2. Architecture and Model Components

SAM-Med3D consists of fully volumetric model blocks:

• 3D Image Encoder: A ViT-based encoder with learnable 3D patch embedding (kernel 16³, stride 16³), 3D absolute positional encoding, and sequential 3D transformer layers with multi-head self-attention augmented by volumetric relative bias.
• Prompt Encoder: Sparse 3D prompts (points, boxes) encoded via learned linear transformations and 3D positional embeddings; dense mask prompts use a small 3D convolutional neck.
• 3D Mask Decoder: Fuses image and prompt features through 3D cross-attention layers and feedforward modules, ending in a 3D upsampling pathway (transposed convolutions) and an MLP head for binary or multi-label mask prediction. The entire decoding pipeline operates at native 128³ resolution.

Architecture ablations in the literature confirm superior performance from training the full 3D network from scratch rather than initializing via duplicated 2D SAM weights (Wang et al., 2023).

3. Training Corpus, Procedures, and Evaluation Protocols

The SAM-Med3D-140K dataset was constructed from heterogeneous public and private sources:

• Data Composition: 21,000 3D images spanning 27 imaging modalities (26 MRI sequences, CT, ultrasound), standardized to 128×128×128 voxel resolution.
• Annotation Set: 131,000 expertly labeled binary masks, covering 247 semantic classes; all masks undergo quality refinement, connected-component analysis, and organ symmetry splitting.
• Preprocessing: Size and content filtering, Z-normalization, outlier removal, cropping/padding, data augmentation with random flips to enforce robust spatial modeling.

Training proceeds in two stages:

1. Pretraining: Minimize the combined Dice and cross-entropy loss over the entire corpus for 800 epochs with cyclic learning rate decay.
2. Prompt adaptation (optional): With encoder weights frozen, fine-tune prompt encoder and mask decoder on new/unseen targets, applying regularization (dropout, similar loss).

Evaluation leverages simulated interactive segmentation using foreground point prompts and error-region augmentation. Metrics include Dice Similarity Coefficient (DSC), Intersection-over-Union (IoU), and inference speed.

4. Comparative Performance and Prompt Efficiency

SAM-Med3D exhibits competitive or superior accuracy with minimal prompts:

Prompt vs. Dice Performance Table

Model	Prompts	Overall DSC (%)	Inference Time (s)
SAM	N points/slice	17.0	13
SAM-Med2D	N points/slice	42.8	4
SAM-Med3D	1 3D point	49.9	2
SAM-Med3D	3 3D points	56.4	3
SAM-Med3D	5 3D points	58.6	4
SAM-Med3D	10 3D points	60.9	6

On benchmarks covering 15 datasets and 153 classes, SAM-Med3D achieves:

• Cardiac, muscle, gland, and brain targets: up to +17.7 pp DSC improvement over slice-based models with one prompt.
• Generalization: consistent performance across CT, MRI, and ultrasound, including unseen modalities.
• Lesion segmentation: for in-domain targets, DSC ≈ 42% (one prompt) to ≈ 50% (five prompts); for truly novel classes, DSC ≈ 40% to ≈ 48% (ten prompts).

Transfer experiments reveal significant DSC gains (+5.6 pp on AMOS) when initializing other 3D architectures with SAM-Med3D's encoder.

5. Limitations, Failure Modes, and Remediation

Failure cases are primarily observed when segmenting:

• Thin, branching structures (vessels, nerves): 3D context integration helps but undersegmentation remains challenging.
• Lesions with highly variable intensity or complex shape: performance improvements plateau with increased prompts.
• Sparse prompts: beyond five global prompts, marginal gains diminish, suggesting prompt saturation.

GPU memory and training time remain substantial when scaling to very large volumes or more granular categories. Further, adaptation to rare classes often requires targeted fine-tuning.

Suggested avenues for enhancing performance include:

• Incorporation of new prompt types (3D boxes, scribbles)
• Integration of text/class prompts for fully automatic/supervised labeling
• Federated or self-supervised pretraining to augment data diversity
• Optimization for efficient inference in resource-constrained clinical environments

6. Extensions, Downstream Applications, and Impact

SAM-Med3D has catalyzed numerous architectural adaptations and downstream solutions:

• SAM-Med3D-MoE: Mixture-of-Experts logical integration that avoids catastrophic forgetting when adapting to rare or challenging clinical categories. MoE incorporates multiple task-specific mask decoders, with a cross-attention gating network for expert selection, yielding average Dice gains of +3.2 pp on weak categories and +16.6 pp on neuroblastoma over the baseline (Wang et al., 2024).
• Federated Fine-tuning: Privacy-preserving adaptation for diagnosis (e.g. dementia classification), leveraging frozen foundation encoders, lightweight heads, and advanced aggregation in decentralized multi-institutional settings (Mouheb et al., 29 Aug 2025).
• Zero-shot, semi-automatic workflows: Supported by the efficiency and prompt generality of SAM-Med3D, enabling rapid annotation, surgical planning, and dataset bootstrapping with minimal user input.

A plausible implication is that 3D promptable foundation models will continue to underpin scalable, generalizable medical imaging pipelines, driving both human-in-the-loop annotation and autonomous segmentation in clinical practice.

7. Future Directions and Research Frontiers

Ongoing research seeks to advance SAM-Med3D and related frameworks through:

• Multi-expert fusion strategies beyond top-1 decoder logic for ambiguous prompts (Wang et al., 2024).
• Dynamic thresholding and adaptive prompt routing in MoE gating networks.
• Extension of prompt modalities (dynamic point, box, scribble, or text) to further minimize annotation burden.
• Transfer and benchmarking across rare-pathology cohorts, non-CT/MRI modalities, and emerging high-resolution volumetric imaging.

Full dataset and code resources for SAM-Med3D are made available by the authors to support broad reproducibility and extension (Wang et al., 2023). The proliferation of efficient, general-purpose 3D foundation models promises robust segmentation across large-scale, heterogeneous medical imaging repositories.