Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Segment Anything Model for Medical Images? (2304.14660v7)

Published 28 Apr 2023 in eess.IV, cs.CV, and cs.LG

Abstract: The Segment Anything Model (SAM) is the first foundation model for general image segmentation. It has achieved impressive results on various natural image segmentation tasks. However, medical image segmentation (MIS) is more challenging because of the complex modalities, fine anatomical structures, uncertain and complex object boundaries, and wide-range object scales. To fully validate SAM's performance on medical data, we collected and sorted 53 open-source datasets and built a large medical segmentation dataset with 18 modalities, 84 objects, 125 object-modality paired targets, 1050K 2D images, and 6033K masks. We comprehensively analyzed different models and strategies on the so-called COSMOS 1050K dataset. Our findings mainly include the following: 1) SAM showed remarkable performance in some specific objects but was unstable, imperfect, or even totally failed in other situations. 2) SAM with the large ViT-H showed better overall performance than that with the small ViT-B. 3) SAM performed better with manual hints, especially box, than the Everything mode. 4) SAM could help human annotation with high labeling quality and less time. 5) SAM was sensitive to the randomness in the center point and tight box prompts, and may suffer from a serious performance drop. 6) SAM performed better than interactive methods with one or a few points, but will be outpaced as the number of points increases. 7) SAM's performance correlated to different factors, including boundary complexity, intensity differences, etc. 8) Finetuning the SAM on specific medical tasks could improve its average DICE performance by 4.39% and 6.68% for ViT-B and ViT-H, respectively. We hope that this comprehensive report can help researchers explore the potential of SAM applications in MIS, and guide how to appropriately use and develop SAM.

PDF HTML Abstract

Evaluation of the Segment Anything Model on a Diverse Medical Image Dataset

The paper "Segment Anything Model for Medical Images?" presents a rigorous evaluation of the Segment Anything Model (SAM) across a substantial medical image segmentation dataset, termed COSMOS 1050K. This dataset, developed by Yuhao Huang and colleagues, is one of the most extensive collections available, encompassing 53 public datasets with 18 distinct modalities and 84 categories of objects. The paper sheds light on the relative proficiency of SAM in medical image segmentation (MIS), an area marked by complex object structures and boundaries.

Key Findings and Methodology

The research delineates a comprehensive evaluation of SAM using two model versions, ViT-B and ViT-H, along with six distinct testing strategies ranging from an automatic mode to several manual prompt modes. The evaluations utilize common metrics, including DICE Coefficient, Jaccard Similarity Coefficient, and Hausdorff Distance. The paper's findings are as follows:

  1. Model Size and Performance: ViT-H, a larger model compared to ViT-B, generally outperformed its smaller counterpart in all evaluation modes. This indicates that the superior capacity of ViT-H, facilitated by its larger parameter space, is better suited for the nuanced task of MIS.
  2. Automatic vs. Manual Prompts: SAM without prompts (Everything mode) showed notable deficiencies across different datasets. However, performance significantly improved when manual hints, particularly box prompts, were incorporated. This suggests that while SAM holds potential in zero-shot scenarios, its robustness in medical settings is enhanced through guided input.
  3. Impact of Object Attributes: The analysis of SAM’s segmentation performance correlates with object characteristics such as boundary complexity and intensity contrast. The results revealed a moderate influence of these traits on the efficiency of SAM, highlighting areas where model refinements could increase accuracy.
  4. Task-specific Finetuning: Finetuning SAM within specific medical tasks led to notable performance gains, with improvements in DICE scores for both ViT-B and ViT-H models. This underscores the potential for SAM to become a strong performer in focused medical applications through tailored training.
  5. Annotation Aid and Time Efficiency: The paper demonstrated SAM's capability to support human annotators by reducing labeling time and improving quality, a critical factor for large-scale medical data annotation.

Implications and Future Directions

The implications of the findings are manifold. Practically, SAM offers a substantial starting point for developing more efficient medical segmentation tools, with its ability to be finetuned and assist human annotators. Theoretically, the paper sets a precedence for evaluating foundation models on medical datasets, encouraging further research into domain-specific adaptations of general models like SAM.

For future developments, several avenues are highlighted. Firstly, improving SAM's interactive capabilities to handle multi-round prompt scenarios could bolster its utility in medical contexts. Additionally, exploring end-to-end semantic segmentation using combinations of SAM with models like CLIP or OVOD could pave the way for robust medical object classification and identification.

As SAM's performance still shows variability across modalities and tasks, subsequent efforts may focus on leveraging synthetic data generation to supplement medical datasets, enabling better training for zero-shot capabilities. Furthermore, enhancing SAM's adaptability to both 2D and 3D data would address the breadth of medical imaging modalities.

In conclusion, while SAM demonstrates potential for medical image segmentation, this research identifies significant opportunities for further enhancement and adaptation to meet the diverse demands of the field. The paper provides a crucial framework for researchers looking to extend the application of foundational models within the domain of medical imaging.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (101)
  1. Neural segmentation of seeding rois (srois) for pre-surgical brain tractography. IEEE transactions on medical imaging 39, 1655–1667.
  2. Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features. Scientific data 4, 1–13.
  3. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge. arXiv preprint arXiv:1811.02629 .
  4. Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Computerized medical imaging and graphics 43, 99–111.
  5. Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is the problem solved? IEEE transactions on medical imaging 37, 2514–2525.
  6. The liver tumor segmentation benchmark (lits). Medical Image Analysis 84, 102680.
  7. Universeg: Universal medical image segmentation. arXiv preprint arXiv:2304.06131 .
  8. Multi-centre, multi-vendor and multi-disease cardiac segmentation: the m&ms challenge. IEEE Transactions on Medical Imaging 40, 3543–3554.
  9. An integrated micro-and macroarchitectural analysis of the drosophila brain by computer-assisted serial section electron microscopy. PLoS biology 8, e1000502.
  10. Monai: An open-source framework for deep learning in healthcare. arXiv preprint arXiv:2211.02701 .
  11. Ma-sam: Modality-agnostic sam adaptation for 3d medical image segmentation. arXiv:2309.08842.
  12. Focalclick: Towards practical interactive image segmentation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1300--1309.
  13. Sam-med2d. arXiv preprint arXiv:2308.16184 .
  14. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv:1902.03368 .
  15. Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic), in: 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), IEEE. pp. 168--172.
  16. Generalized overlap measures for evaluation and validation in medical image analysis. IEEE transactions on medical imaging 25, 1451--1461.
  17. Chest CT Segmentation Dataset. [EB/OL]. https://www.kaggle.com/datasets/polomarco/chest-ct-segmentation.
  18. 4c2021 c04 tls01 dataset. [EB/OL]. https://aistudio.baidu.com/aistudio/projectdetail/1952488?channelType=1&channel=1.
  19. IXI Dataset. [EB/OL]. https://brain-development.org/ixi-dataset/.
  20. Segment anything model (sam) for digital pathology: Assess zero-shot segmentation on whole slide imaging. arXiv preprint arXiv:2304.04155 .
  21. An image is worth 16x16 words: Transformers for image recognition at scale, in: International Conference on Learning Representations.
  22. Mild-net: Minimal information loss dilated network for gland instance segmentation in colon histology images. Medical image analysis 52, 199--211.
  23. Masked autoencoders are scalable vision learners, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000--16009.
  24. Accuracy of segment-anything model (sam) in medical image segmentation tasks. arXiv preprint arXiv:2304.09324 .
  25. Dense biased networks with deep priori anatomy and hard region adaptation: Semi-supervised learning for fine renal artery segmentation. Medical image analysis 63, 101722.
  26. Meta grayscale adaptive network for 3d integrated renal structures segmentation. Medical image analysis 71, 102055.
  27. Comparison and evaluation of methods for liver segmentation from ct datasets. IEEE transactions on medical imaging 28, 1251--1265.
  28. The kits19 challenge data: 300 kidney tumor cases with clinical context, ct semantic segmentations, and surgical outcomes. arXiv:1904.00445.
  29. The endotect 2020 challenge: evaluation and comparison of classification, segmentation and inference time for endoscopy, in: Pattern Recognition. ICPR International Workshops and Challenges: Virtual Event, January 10-15, 2021, Proceedings, Part VIII, Springer. pp. 263--274.
  30. Locating blood vessels in retinal images by piecewise threshold probing of a matched filter response. IEEE Transactions on Medical imaging 19, 203--210.
  31. Flip learning: Erase to segment, in: Medical Image Computing and Computer Assisted Intervention--MICCAI 2021: 24th International Conference, Strasbourg, France, September 27--October 1, 2021, Proceedings, Part I 24, Springer. pp. 493--502.
  32. Palm: Pathologic myopia challenge. Comput. Vis. Med. Imaging .
  33. nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods 18, 203--211.
  34. Two public chest x-ray datasets for computer-aided screening of pulmonary diseases. Quantitative imaging in medicine and surgery 4, 475.
  35. Kvasir-instrument: Diagnostic and therapeutic tool segmentation dataset in gastrointestinal endoscopy, in: MultiMedia Modeling, Springer International Publishing, Cham. pp. 218--229.
  36. Kvasir-seg: A segmented polyp dataset, in: International Conference on Multimedia Modeling, Springer. pp. 451--462.
  37. Sam struggles in concealed scenes--empirical study on" segment anything". arXiv preprint arXiv:2304.06022 .
  38. Segment anything is not always perfect: An investigation of sam on different real-world applications. arXiv preprint arXiv:2304.05750 .
  39. Learning calibrated medical image segmentation via multi-rater agreement modeling, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12341--12351.
  40. Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation. arXiv preprint arXiv:2206.08023 .
  41. Scaling up visual and vision-language representation learning with noisy text supervision, in: International Conference on Machine Learning, PMLR. pp. 4904--4916.
  42. Thyroid nodule segmentation and classification in ultrasound images through intra-and inter-task consistent learning. Medical image analysis 79, 102443.
  43. Evaluation of state-of-the-art segmentation algorithms for left ventricle infarct from late gadolinium enhancement mr images. Medical image analysis 30, 95--107.
  44. Chaos challenge-combined (ct-mr) healthy abdominal organ segmentation. Medical Image Analysis 69, 101950.
  45. Segment anything. arXiv preprint arXiv:2304.02643 .
  46. Elliptic fourier features of a closed contour. Computer graphics and image processing 18, 236--258.
  47. Deep learning for segmentation using an open large-scale dataset in 2d echocardiography. IEEE transactions on medical imaging 38, 2198--2210.
  48. Mediar: Harmony of data-centric and model-centric for multi-modality microscopy. arXiv:2212.03465.
  49. Learning local shape and appearance for segmentation of knee cartilage in 3d mri, in: Medical Image Analysis for the Clinic: a Grand Challenge. In Proceedings of the 13th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2010), Beijing, China, pp. 231--240.
  50. Computer-aided detection and diagnosis for prostate cancer based on mono and multi-parametric mri: a review. Computers in biology and medicine 60, 8--31.
  51. Automated segmentation of prostate mr images using prior knowledge enhanced random walker, in: 2013 International Conference on Digital Image Computing: Techniques and Applications (DICTA), IEEE. pp. 1--7.
  52. Semantic-sam: Segment and recognize anything at any granularity. arXiv preprint arXiv:2307.04767 .
  53. Self-supervised feature learning via exploiting multi-modal data for retinal disease diagnosis. IEEE Transactions on Medical Imaging 39, 4023--4033.
  54. Sketch guided and progressive growing gan for realistic and editable ultrasound image synthesis. Medical Image Analysis 79, 102461.
  55. A computed tomography vertebral segmentation dataset with anatomical variations and multi-vendor scanner data. Scientific Data 8, 284.
  56. Evaluation of prostate segmentation algorithms for mri: the promise12 challenge. Medical image analysis 18, 359--373.
  57. Simpleclick: Interactive image segmentation with simple vision transformers. arXiv preprint arXiv:2210.11006 .
  58. Full-resolution network and dual-threshold iteration for retinal vessel and coronary angiograph segmentation. IEEE Journal of Biomedical and Health Informatics 26, 4623--4634.
  59. Samm (segment any medical model): A 3d slicer integration to sam. arXiv preprint arXiv:2304.05622 .
  60. A vertebral segmentation dataset with fracture grading. Radiology: Artificial Intelligence 2, e190138.
  61. Learning for structured prediction using approximate subgradient descent with working sets, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1987--1994.
  62. Word: A large scale dataset, benchmark and clinical applicable study for abdominal organ segmentation from ct image. Medical Image Analysis 82, 102642.
  63. Segment anything in medical images. arXiv:2304.12306.
  64. The multi-modality cell segmentation challenge: Towards universal solutions. arXiv:2308.05864 .
  65. Abdomenct-1k: Is abdominal organ segmentation a solved problem? IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 6695--6714. doi:10.1109/TPAMI.2021.3100536.
  66. Zero-shot performance of the segment anything model (sam) in 2d medical imaging: A comprehensive evaluation and practical guidelines. arXiv:2305.00109.
  67. Segment anything model for medical image analysis: an experimental study. Medical Image Analysis 89, 102918.
  68. The multimodal brain tumor image segmentation benchmark (brats). IEEE transactions on medical imaging 34, 1993--2024.
  69. Brain extraction comparing segment anything model (sam) and fsl brain extraction tool. arXiv preprint arXiv:2304.04738 .
  70. Spineparsenet: spine parsing for volumetric mr image by a two-stage segmentation framework with semantic image representation. IEEE Transactions on Medical Imaging 40, 262--273.
  71. An automatic multi-tissue human fetal brain segmentation benchmark using the fetal tissue annotation dataset. Scientific Data 8, 167.
  72. 3d slicer, in: 2004 2nd IEEE international symposium on biomedical imaging: nano to macro (IEEE Cat No. 04EX821), IEEE. pp. 632--635.
  73. Han-seg: The head and neck organ-at-risk ct & mr segmentation dataset. Medical Physics .
  74. Indian diabetic retinopathy image dataset (idrid): a database for diabetic retinopathy screening research. Data 3, 25.
  75. Highly accurate dichotomous image segmentation, in: European Conference on Computer Vision, Springer. pp. 38--56.
  76. Annotating 8,000 abdominal ct volumes for multi-organ segmentation in three weeks. arXiv preprint arXiv:2305.09666 .
  77. Learning transferable visual models from natural language supervision, in: International conference on machine learning, PMLR. pp. 8748--8763.
  78. Zero-shot text-to-image generation, in: International Conference on Machine Learning, PMLR. pp. 8821--8831.
  79. Verse: A vertebrae labelling and segmentation benchmark for multi-detector ct images. Medical image analysis 73, 102166.
  80. Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the luna16 challenge. Medical image analysis 42, 1--13.
  81. Laparoscopic partial nephrectomy with segmental renal artery clamping: technique and clinical outcomes. European urology 59, 849--855.
  82. Precise segmental renal artery clamping under the guidance of dual-source computed tomography angiography during laparoscopic partial nephrectomy. European urology 62, 1001--1008.
  83. An artificial intelligence framework for automatic segmentation and volumetry of vestibular schwannomas from contrast-enhanced t1-weighted and high-resolution t2-weighted mri. Journal of neurosurgery 134, 171--179.
  84. A large annotated medical image dataset for the development and evaluation of segmentation algorithms. arXiv preprint arXiv:1902.09063 .
  85. Gland segmentation in colon histology images: The glas challenge contest. Medical image analysis 35, 489--502.
  86. Multi-site infant brain segmentation algorithms: the iseg-2019 challenge. IEEE Transactions on Medical Imaging 40, 1363--1376.
  87. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems 33, 7537--7547.
  88. Can sam segment anything? when sam meets camouflaged object detection. arXiv preprint arXiv:2304.04709 .
  89. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data 5, 1--9.
  90. Weakly-supervised segmentation for disease localization in chest x-ray images, in: Artificial Intelligence in Medicine: 18th International Conference on Artificial Intelligence in Medicine, AIME 2020, Minneapolis, MN, USA, August 25--28, 2020, Proceedings 18, Springer. pp. 249--259.
  91. A deep learning based medical image segmentation technique in internet-of-medical-things domain. Future Generation Computer Systems 108, 135--144.
  92. Autolaparo: A new dataset of integrated multi-tasks for image-guided surgical automation in laparoscopic hysterectomy. arXiv:2208.02049.
  93. Totalsegmentator: Robust segmentation of 104 anatomic structures in ct images. Radiology: Artificial Intelligence 5, e230024. doi:10.1148/ryai.230024.
  94. The confidence interval that wasn’t: Bootstrapped “confidence intervals” in l1-regularized partial correlation networks .
  95. Medical sam adapter: Adapting segment anything model for medical image segmentation. arXiv:2304.12620.
  96. Colonoscopic image synthesis with generative adversarial network for enhanced detection of sessile serrated lesions using convolutional neural network. Scientific reports 12, 261.
  97. Robust retinal vessel segmentation via locally adaptive derivative frames in orientation scores. IEEE transactions on medical imaging 35, 2631--2644.
  98. A coarse-to-fine framework for the 2021 kidney and kidney tumor segmentation challenge, in: Kidney and Kidney Tumor Segmentation: MICCAI 2021 Challenge, KiTS 2021, Held in Conjunction with MICCAI 2021, Strasbourg, France, September 27, 2021, Proceedings. Springer, pp. 53--58.
  99. Iou loss for 2d/3d object detection, in: 2019 international conference on 3D vision (3DV), IEEE. pp. 85--94.
  100. Can sam segment polyps? arXiv preprint arXiv:2304.07583 .
  101. Prior-aware neural network for partially-supervised multi-organ segmentation, in: Proceedings of the IEEE/CVF international conference on computer vision, pp. 10672--10681.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (19)
  1. Yuhao Huang (46 papers)
  2. Xin Yang (312 papers)
  3. Lian Liu (25 papers)
  4. Han Zhou (72 papers)
  5. Ao Chang (7 papers)
  6. Xinrui Zhou (14 papers)
  7. Rusi Chen (6 papers)
  8. Junxuan Yu (3 papers)
  9. Jiongquan Chen (5 papers)
  10. Chaoyu Chen (70 papers)
  11. Sijing Liu (16 papers)
  12. Haozhe Chi (5 papers)
  13. Xindi Hu (14 papers)
  14. Kejuan Yue (2 papers)
  15. Lei Li (1293 papers)
  16. Vicente Grau (28 papers)
  17. Deng-Ping Fan (88 papers)
  18. Fajin Dong (2 papers)
  19. Dong Ni (89 papers)
Citations (27)
View on Semantic Scholar