SAM-Med3D: Towards General-purpose Segmentation Models for Volumetric Medical Images (2310.15161v3)

Abstract: Existing volumetric medical image segmentation models are typically task-specific, excelling at specific target but struggling to generalize across anatomical structures or modalities. This limitation restricts their broader clinical use. In this paper, we introduce SAM-Med3D for general-purpose segmentation on volumetric medical images. Given only a few 3D prompt points, SAM-Med3D can accurately segment diverse anatomical structures and lesions across various modalities. To achieve this, we gather and process a large-scale 3D medical image dataset, SA-Med3D-140K, from a blend of public sources and licensed private datasets. This dataset includes 22K 3D images and 143K corresponding 3D masks. Then SAM-Med3D, a promptable segmentation model characterized by the fully learnable 3D structure, is trained on this dataset using a two-stage procedure and exhibits impressive performance on both seen and unseen segmentation targets. We comprehensively evaluate SAM-Med3D on 16 datasets covering diverse medical scenarios, including different anatomical structures, modalities, targets, and zero-shot transferability to new/unseen tasks. The evaluation shows the efficiency and efficacy of SAM-Med3D, as well as its promising application to diverse downstream tasks as a pre-trained model. Our approach demonstrates that substantial medical resources can be utilized to develop a general-purpose medical AI for various potential applications. Our dataset, code, and models are available at https://github.com/uni-medical/SAM-Med3D.

Overview of SAM-Med3D: Advancements in 3D Medical Image Segmentation

The paper "SAM-Med3D" investigates the limitations of the Segment Anything Model (SAM) in handling 3D volumetric medical images and introduces a comprehensive solution tailored for this purpose. SAM, originally designed for 2D natural image segmentation, struggles with 3D spatial information, resulting in suboptimal performance and requiring numerous prompt points for reliable outcomes. The proposed SAM-Med3D addresses these deficiencies by reformulating SAM into a fully 3D architecture, which is trained on an extensive volumetric medical dataset. This paper provides a detailed exploration of its architecture, training methodology, and evaluation outcomes.

Key Contributions

1. 3D Architectural Reformulation: Unlike previous adaptations that attempted to apply 2D SAM architectures to 3D data with modifications, SAM-Med3D implements a fully 3D learnable framework. It modifies the image encoder, prompt encoder, and mask decoder to seamlessly incorporate 3D spatial information.
2. Large-Scale Training Dataset: SAM-Med3D employs a robust training dataset comprising over 21,000 3D medical images and 131,000 masks, encompassing 247 categories. This dataset amalgamates various publicly and privately sourced 3D medical image datasets, positioning it as a comprehensive resource for training and improving 3D medical image segmentation models.
3. Evaluation Across Multiple Dimensions: The model is assessed on 15 datasets, focusing on diverse aspects such as anatomical structures, modalities, targets, and generalization abilities. The findings demonstrate SAM-Med3D’s efficiency and broad segmentation capability, with significantly fewer prompt points than the fine-tuned SAM models.

Numerical Results and Performance

SAM-Med3D exhibits substantial performance improvements over other SAM adaptations. A key finding is that SAM-Med3D achieves an overall Dice score of 60.94% with only 10 prompt points, markedly outperforming the 2D fine-tuned models. This highlights its capacity to effectively segment 3D volumes with fewer human interactions, promoting its usability in clinical settings. The model's ability to maintain inter-slice consistency substantially enhances its segmentation accuracy compared to 2D slice-by-slice methods, which often fail to leverage inter-slice information effectively.

Implications and Future Directions

The implications of SAM-Med3D are vast for volumetric medical imaging. By enabling efficient and accurate segmentation with minimal prompts, SAM-Med3D holds promise for applications in medical diagnosis and treatment planning. The architectural innovations in adapting a 2D model for 3D data capture valuable insights that can be extended to other domains requiring volumetric data processing.

Future research may focus on further adaptations of the SAM-Med3D architecture to different modalities and the development of novel prompting strategies that exploit the volumetric nature of medical data. Additionally, investigating the transferability of SAM-Med3D as a pre-trained model for various downstream tasks in medical imaging can open new avenues for its application in enhancing other medical image analysis pipelines.

In conclusion, the proposed SAM-Med3D model represents a significant step forward in addressing the inadequacies of existing models in 3D medical image segmentation. Its comprehensive evaluation and substantial performance improvements suggest that SAM-Med3D can play a pivotal role in advancing medical image analysis technologies.