Papers
Topics
Authors
Recent
2000 character limit reached

Segment Anything Model (SAM) for Radiation Oncology

Published 20 Jun 2023 in eess.IV, cs.CV, and cs.LG | (2306.11730v2)

Abstract: In this study, we evaluate the performance of the Segment Anything Model (SAM) in clinical radiotherapy. Our results indicate that SAM's 'segment anything' mode can achieve clinically acceptable segmentation results in most organs-at-risk (OARs) with Dice scores higher than 0.7. SAM's 'box prompt' mode further improves the Dice scores by 0.1 to 0.5. Considering the size of the organ and the clarity of its boundary, SAM displays better performance for large organs with clear boundaries but performs worse for smaller organs with unclear boundaries. Given that SAM, a model pre-trained purely on natural images, can handle the delineation of OARs from medical images with clinically acceptable accuracy, these results highlight SAM's robust generalization capabilities with consistent accuracy in automatic segmentation for radiotherapy. In other words, SAM can achieve delineation of different OARs at different sites using a generic automatic segmentation model. SAM's generalization capabilities across different disease sites suggest that it is technically feasible to develop a generic model for automatic segmentation in radiotherapy.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (111)
  1. When brain-inspired ai meets agi. arXiv preprint arXiv:2303.15935, 2023.
  2. Summary of chatgpt/gpt-4 research and perspective towards the future of large language models. arXiv preprint arXiv:2304.01852, 2023.
  3. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  4. R OpenAI. Gpt-4 technical report. arXiv, pages 2303–08774, 2023.
  5. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023.
  6. Evaluating large language models on a highly-specialized topic, radiation oncology physics. arXiv preprint arXiv:2304.01938, 2023.
  7. Radiology-gpt: A large language model for radiology. arXiv preprint arXiv:2306.08666, 2023.
  8. Impressiongpt: An iterative optimizing framework for radiology report summarization with chatgpt. arXiv preprint arXiv:2304.08448, 2023.
  9. Exploring the trade-offs: Unified large language models vs local fine-tuned models for highly-specific radiology nli task. arXiv preprint arXiv:2304.09138, 2023.
  10. Chatabl: Abductive learning via natural language interaction with chatgpt. arXiv preprint arXiv:2304.11107, 2023.
  11. Deid-gpt: Zero-shot medical text de-identification by gpt-4. arXiv preprint arXiv:2303.11032, 2023.
  12. Chataug: Leveraging chatgpt for text data augmentation. arXiv preprint arXiv:2302.13007, 2023.
  13. Differentiate chatgpt-generated and human-written medical texts. arXiv preprint arXiv:2304.11567, 2023.
  14. Exploring new frontiers in agricultural nlp: Investigating the potential of large language models for food applications. arXiv preprint arXiv:2306.11892, 2023.
  15. Ad-autogpt: An autonomous gpt for alzheimer’s disease infodemiology. arXiv preprint arXiv:2306.10095, 2023.
  16. Artificial general intelligence for medical imaging. arXiv preprint arXiv:2306.05480, 2023.
  17. Segment anything model for medical images? arXiv preprint arXiv:2304.14660, 2023.
  18. Segment anything. arXiv preprint arXiv:2304.02643, 2023.
  19. Inpaint anything: Segment anything meets image inpainting. arXiv preprint arXiv:2304.06790, 2023.
  20. Resolution-robust large mask inpainting with fourier convolutions. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2149–2159, 2022.
  21. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.
  22. Edit everything: A text-guided generative system for images editing. arXiv preprint arXiv:2304.14006, 2023.
  23. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  24. Any-to-any style transfer: Making picasso and da vinci collaborate. arXiv e-prints, pages arXiv–2304, 2023.
  25. Deep learning universal crater detection using segment anything model (sam). arXiv preprint arXiv:2304.07764, 2023.
  26. Segment anything is not always perfect: An investigation of sam on different real-world applications. arXiv preprint arXiv:2304.05750, 2023.
  27. The role of radiotherapy in cancer treatment: estimating optimal utilization from a review of evidence-based clinical guidelines. Cancer: Interdisciplinary International Journal of the American Cancer Society, 104(6):1129–1137, 2005.
  28. Proton beam therapy for locally advanced lung cancer: A review. World journal of clinical oncology, 5(4):568, 2014.
  29. Robust optimization in impt using quadratic objective functions to account for the minimum mu constraint. Medical physics, 45(1):460–469, 2018.
  30. An automatic approach for satisfying dose-volume constraints in linear fluence map optimization for impt. Journal of cancer therapy, 5(2):198, 2014.
  31. Comparison of linear and nonlinear programming approaches for “worst case dose” and “minmax” robust optimization of intensity-modulated proton therapy dose distributions. Journal of applied clinical medical physics, 18(2):15–25, 2017.
  32. Wei Liu. System and method for robust intensity-modulated proton therapy planning, August 2019. US Patent 10,369,381.
  33. Dosimetric comparison of distal esophageal carcinoma plans for patients treated with small-spot intensity-modulated proton versus volumetric-modulated arc therapies. Journal of applied clinical medical physics, 20(7):15–27, 2019.
  34. Small-spot intensity-modulated proton therapy and volumetric-modulated arc therapies for patients with locally advanced non-small-cell lung cancer: a dosimetric comparative study. Journal of applied clinical medical physics, 19(6):140–148, 2018.
  35. W Liu. Robustness quantification and robust optimization in intensity-modulated proton therapy. Particle Radiotherapy: Emerging Technology for Treatment of Cancer. Springer, pages 139–56, 2015.
  36. Robustness quantification methods comparison in volumetric modulated arc therapy to treat head and neck cancer. Practical radiation oncology, 6(6):e269–e275, 2016.
  37. Robust optimization of intensity modulated proton therapy. Medical physics, 39(2):1079–1091, 2012.
  38. Emphasizing conformal avoidance versus target definition for imrt planning in head-and-neck cancer. International Journal of Radiation Oncology* Biology* Physics, 77(3):950–958, 2010.
  39. Deep-learning based fast and accurate 3d ct deformable image registration in lung cancer. Medical Physics, 2023.
  40. The relationship between waiting time for radiotherapy and clinical outcomes: a systematic review of the literature. Radiotherapy and Oncology, 87(1):3–16, 2008.
  41. Impact of respiratory motion on worst-case scenario optimized intensity modulated proton therapy for lung cancers. Practical radiation oncology, 5(2):e77–e86, 2015.
  42. Dosimetric benefits of robust treatment planning for intensity modulated proton therapy for base-of-skull cancers. Practical radiation oncology, 4(6):384–391, 2014.
  43. Exploratory study of 4d versus 3d robust optimization in intensity modulated proton therapy for lung cancer. International Journal of Radiation Oncology* Biology* Physics, 95(1):523–533, 2016.
  44. Beam angle comparison for distal esophageal carcinoma patients treated with intensity-modulated proton therapy. Journal of applied clinical medical physics, 21(11):141–152, 2020.
  45. Robust treatment planning with conditional value at risk chance constraints in intensity-modulated proton therapy. Medical physics, 44(1):28–36, 2017.
  46. Effectiveness of robust optimization in intensity-modulated proton therapy planning for head and neck cancers. Medical physics, 40(5):051711, 2013.
  47. Robust optimization in intensity-modulated proton therapy to account for anatomy changes in lung cancer patients. Radiotherapy and Oncology, 114(3):367–372, 2015.
  48. Treatment planning system (tps) approximations matter—comparing intensity-modulated proton therapy (impt) plan quality and robustness between a commercial and an in-house developed tps for nonsmall cell lung cancer (nsclc). Medical physics, 46(11):4755–4762, 2019.
  49. Ptv-based impt optimization incorporating planning risk volumes vs robust optimization. Medical physics, 40(2):021709, 2013.
  50. Influence of robust optimization in intensity-modulated proton therapy with different dose delivery techniques. Medical physics, 39(6Part1):3089–3101, 2012.
  51. A deep learning-based auto-segmentation system for organs-at-risk on whole-body computed tomography images for radiation therapy. Radiotherapy and Oncology, 160:175–184, 2021.
  52. Rapid advances in auto-segmentation of organs at risk and target volumes in head and neck cancer. Radiotherapy and Oncology, 135:130–140, 2019.
  53. Improving automatic delineation for head and neck organs at risk by deep learning contouring. Radiotherapy and Oncology, 142:115–123, 2020.
  54. Clinical evaluation of commercial atlas-based auto-segmentation in the head and neck region. Frontiers in Oncology, 9:239, 2019.
  55. Deep learning for automatic target volume segmentation in radiation therapy: a review. Quantitative Imaging in Medicine and Surgery, 11(12):4847, 2021.
  56. Comparative clinical evaluation of atlas and deep-learning-based auto-segmentation of organ structures in liver cancer. Radiation Oncology, 14:1–13, 2019.
  57. Deep learning-based auto-segmentation of targets and organs-at-risk for magnetic resonance imaging only planning of prostate radiotherapy. Physics and imaging in radiation oncology, 12:80–86, 2019.
  58. Anatomynet: deep learning for fast and fully automated whole-volume segmentation of head and neck anatomy. Medical physics, 46(2):576–589, 2019.
  59. Applications and limitations of machine learning in radiation oncology. The British journal of radiology, 92(1100):20190001, 2019.
  60. Deep generative medical image harmonization for improving cross-site generalization in deep learning predictors. Journal of Magnetic Resonance Imaging, 55(3):908–916, 2022.
  61. Multi-institutional investigation of model generalizability for virtual contrast-enhanced mri synthesis. In Linwei Wang, Qi Dou, P. Thomas Fletcher, Stefanie Speidel, and Shuo Li, editors, Medical Image Computing and Computer Assisted Intervention – MICCAI 2022, pages 765–773, Cham, 2022. Springer Nature Switzerland.
  62. The segment anything foundation model achieves favorable brain tumor autosegmentation accuracy on mri to support radiotherapy treatment planning. arXiv preprint arXiv:2304.07875, 2023.
  63. Segment anything model for medical image analysis: an experimental study. arXiv preprint arXiv:2304.10517, 2023.
  64. Sam struggles in concealed scenes–empirical study on" segment anything". arXiv preprint arXiv:2304.06022, 2023.
  65. Brain extraction comparing segment anything model (sam) and fsl brain extraction tool. arXiv preprint arXiv:2304.04738, 2023.
  66. Segment anything model (sam) for digital pathology: Assess zero-shot segmentation on whole slide imaging. arXiv preprint arXiv:2304.04155, 2023.
  67. Can sam segment polyps? arXiv preprint arXiv:2304.07583, 2023.
  68. Accuracy of segment-anything model (sam) in medical image segmentation tasks. arXiv preprint arXiv:2304.09324, 2023.
  69. A multiphase level set framework for image segmentation using the mumford and shah model. International journal of computer vision, 50:271–293, 2002.
  70. Active contours without edges. IEEE Transactions on image processing, 50(2):266–277, 2001.
  71. Segmentation of brain mr images through a hidden markov random field model and the expectation-maximization algorithm. IEEE transactions on medical imaging, 20(1):45–57, 2001.
  72. A survey on deep learning in medical image analysis. Medical image analysis, 42:60–88, 2017.
  73. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, pages 234–241, 2015.
  74. Accurate and efficient deep neural network based deformable image registration method in lung cancer. In MEDICAL PHYSICS, volume 49, pages E148–E148. WILEY 111 RIVER ST, HOBOKEN 07030-5774, NJ USA, 2022.
  75. Community graph convolution neural network for alzheimer’s disease classification and pathogenetic factors identification. IEEE Transactions on Neural Networks and Learning Systems, 2023.
  76. Discovering dynamic functional brain networks via spatial and channel-wise attention. arXiv preprint arXiv:2205.09576, 2022.
  77. Graph representation neural architecture search for optimal spatial/temporal functional brain network decomposition. In International Workshop on Machine Learning in Medical Imaging, pages 279–287. Springer, 2022.
  78. A deep learning method for autism spectrum disorder identification based on interactions of hierarchical brain networks. Available at SSRN 4309357.
  79. U-net and its variants for medical image segmentation: A review of theory and applications. Ieee Access, 9:82031–82057, 2021.
  80. Beam mask and sliding window-facilitated deep learning-based accurate and efficient dose prediction for pencil beam scanning proton therapy. arXiv preprint arXiv:2305.18572, 2023.
  81. Survey on natural language processing in medical image analysis. Zhong nan da xue xue bao. Yi xue ban= Journal of Central South University. Medical Sciences, 47(8):981–993, 2022.
  82. End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography. IEEE transactions on medical imaging, 25(6):954–961, 2019.
  83. When sam meets medical images: An investigation of segment anything model (sam) on multi-phase liver tumor segmentation. arXiv preprint arXiv:2304.08506, 2023.
  84. Breastsam: A study of segment anything model for breast tumor detection in ultrasound images. arXiv preprint arXiv:2305.12447, 2023.
  85. Polyp-sam: Transfer sam for polyp segmentation. arXiv preprint arXiv:2305.00293, 2023.
  86. Skinsam: Empowering skin cancer segmentation with segment anything model. arXiv preprint arXiv:2304.13973, 2023.
  87. Multitask brain tumor inpainting with diffusion models: A methodological report. arXiv preprint arXiv:2210.12113, 2022.
  88. Deep generative models in the real-world: An open challenge from medical imaging. arXiv preprint arXiv:1806.05452, 2018.
  89. Self-supervised learning methods and applications in medical imaging analysis: A survey. PeerJ Computer Science, 8:e1045, 2022.
  90. A review of methods of analysis in contouring studies for radiation oncology. Journal of medical imaging and radiation oncology, 54(5):401–410, 2010.
  91. Uncertainties in volume delineation in radiation oncology: a systematic review and recommendations for future studies. Radiotherapy and Oncology, 121(2):169–179, 2016.
  92. A review of interventions to reduce inter-observer variability in volume delineation in radiation oncology. Journal of medical imaging and radiation oncology, 60(3):393–406, 2016.
  93. Inter-observer variability of organ contouring for preclinical studies with cone beam computed tomography imaging. Physics and Imaging in Radiation Oncology, 21:11–17, 2022.
  94. M Posiewnik and T Piotrowski. A review of cone-beam ct applications for adaptive radiotherapy of prostate cancer. Physica Medica, 59:13–21, 2019.
  95. Marcus D Seemann. Pet/ct: fundamental principles. European journal of medical research, 9(5):241–246, 2004.
  96. Acute toxicities and short-term patient outcomes after intensity-modulated proton beam radiation therapy or intensity-modulated photon radiation therapy for esophageal carcinoma: a mayo clinic experience. Advances in Radiation Oncology, 5(5):871–879, 2020.
  97. Robust radiotherapy planning. Physics in Medicine & Biology, 63(22):22TR02, 2018.
  98. Medical sam adapter: Adapting segment anything model for medical image segmentation. arXiv preprint arXiv:2304.12620, 2023.
  99. Deep learning for segmentation in radiation therapy planning: a review. Journal of Medical Imaging and Radiation Oncology, 65(5):578–595, 2021.
  100. Dabs: A domain-agnostic benchmark for self-supervised learning. arXiv preprint arXiv:2111.12062, 2021.
  101. Benchmd: A benchmark for modality-agnostic learning on medical images and sensors. arXiv preprint arXiv:2304.08486, 2023.
  102. Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13138, 2022.
  103. Coarse-to-fine knowledge graph domain adaptation based on distantly-supervised iterative training. arXiv preprint arXiv:2211.02849, 2022.
  104. Context matters: A strategy to pre-train language model for science education. arXiv preprint arXiv:2301.12031, 2023.
  105. Mask-guided bert for few shot text classification. arXiv preprint arXiv:2302.10447, 2023.
  106. Clinicalradiobert: Knowledge-infused few shot learning for clinical notes named entity recognition. In Machine Learning in Medical Imaging: 13th International Workshop, MLMI 2022, Held in Conjunction with MICCAI 2022, Singapore, September 18, 2022, Proceedings, pages 269–278. Springer, 2022.
  107. Agribert: knowledge-infused agricultural language models for matching food and nutrition. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, volume 7, pages 5150–5156, 2022.
  108. Reinforcement learning: An introduction. MIT press, 2018.
  109. Deep reinforcement learning in medical imaging: A literature review. Medical image analysis, 73:102193, 2021.
  110. Bayesian convolutional neural networks with bernoulli approximate variational inference. arXiv preprint arXiv:1506.02158, 2015.
  111. Alessio Micheli. Neural network for graphs: A contextual constructive approach. IEEE Transactions on Neural Networks, 20(3):498–511, 2009.
Citations (28)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up