Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

LLM-driven Multimodal Target Volume Contouring in Radiation Oncology (2311.01908v4)

Published 3 Nov 2023 in eess.IV and cs.CV

Abstract: Target volume contouring for radiation therapy is considered significantly more challenging than the normal organ segmentation tasks as it necessitates the utilization of both image and text-based clinical information. Inspired by the recent advancement of LLMs that can facilitate the integration of the textural information and images, here we present a novel LLM-driven multimodal AI, namely LLMseg, that utilizes the clinical text information and is applicable to the challenging task of target volume contouring for radiation therapy, and validate it within the context of breast cancer radiation therapy target volume contouring. Using external validation and data-insufficient environments, which attributes highly conducive to real-world applications, we demonstrate that the proposed model exhibits markedly improved performance compared to conventional unimodal AI models, particularly exhibiting robust generalization performance and data efficiency. To our best knowledge, this is the first LLM-driven multimodal AI model that integrates the clinical text information into target volume delineation for radiation oncology.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (36)
  1. Huynh, E. et al. Artificial intelligence in radiation oncology. Nature Reviews Clinical Oncology 17, 771–781 (2020).
  2. Shi, F. et al. Deep learning empowered volume delineation of whole-body organs-at-risk for accelerated radiotherapy. Nature Communications 13, 6566 (2022).
  3. Zhang, L. et al. Segment anything model (sam) for radiation oncology. arXiv preprint arXiv:2306.11730 (2023).
  4. Chung, S. Y. et al. Clinical feasibility of deep learning-based auto-segmentation of target volumes and organs-at-risk in breast cancer patients after breast-conserving surgery. Radiation Oncology 16, 1–10 (2021).
  5. Offersen, B. V. et al. Estro consensus guideline on target volume delineation for elective radiation therapy of early stage breast cancer. Radiotherapy and oncology 114, 3–10 (2015).
  6. Choi, M. S. et al. Clinical evaluation of atlas-and deep learning-based automatic segmentation of multiple organs and clinical target volumes for breast cancer. Radiotherapy and Oncology 153, 139–145 (2020).
  7. Gross tumor volume segmentation for head and neck cancer radiotherapy using deep dense multi-modality network. Physics in Medicine & Biology 64, 205015 (2019).
  8. Liu, C. et al. Artificial general intelligence for radiation oncology (2023). 2309.02590.
  9. Bubeck, S. et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 (2023).
  10. Touvron, H. et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023).
  11. Liu, Z. et al. Radiology-gpt: A large language model for radiology. arXiv preprint arXiv:2306.08666 (2023).
  12. Moor, M. et al. Foundation models for generalist medical artificial intelligence. Nature 616, 259–265 (2023).
  13. Singhal, K. et al. Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13138 (2022).
  14. Tu, T. et al. Towards generalist biomedical ai. arXiv preprint arXiv:2307.14334 (2023).
  15. Kirillov, A. et al. Segment anything. arXiv preprint arXiv:2304.02643 (2023).
  16. Zegot: Zero-shot segmentation through optimal transport of text prompts. arXiv preprint arXiv:2301.12171 (2023).
  17. Jia, M. et al. Visual prompt tuning. arXiv preprint arXiv:2203.12119 (2022).
  18. Conditional prompt learning for vision-language models (2022). 2203.05557.
  19. Hu, E. J. et al. Lora: Low-rank adaptation of large language models (2021). 2106.09685.
  20. Deep learning in medical image analysis. Annual review of biomedical engineering 19, 221–248 (2017).
  21. De Fauw, J. et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nature medicine 24, 1342–1350 (2018).
  22. Rajpurkar, P. et al. Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225 (2017).
  23. Choi, B. G. et al. Machine learning for the prediction of new-onset diabetes mellitus during 5-year follow-up in non-diabetic patients with cardiovascular risks. Yonsei medical journal 60, 191–199 (2019).
  24. Yoo, T. K. et al. Osteoporosis risk prediction for bone mineral density assessment of postmenopausal women using machine learning. Yonsei medical journal 54, 1321–1330 (2013).
  25. The current and future state of ai interpretation of medical images. New England Journal of Medicine 388, 1981–1990 (2023).
  26. Artificial intelligence in radiology. Nature Reviews Cancer 18, 500–510 (2018).
  27. Tiu, E. et al. Expert-level detection of pathologies from unannotated chest x-ray images via self-supervised learning. Nature Biomedical Engineering 6, 1399–1406 (2022).
  28. Multi-modal understanding and generation for medical images and text via vision-language pre-training. IEEE Journal of Biomedical and Health Informatics 26, 6070–6080 (2022).
  29. Kiut: Knowledge-injected u-transformer for radiology report generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 19809–19818 (2023).
  30. Contextual net: A multimodal vision-language model for segmentation of pneumothorax. arXiv preprint arXiv:2303.01615 (2023).
  31. Llm itself can read and generate cxr images. arXiv preprint arXiv:2305.11490 (2023).
  32. 3d u-net: learning dense volumetric segmentation from sparse annotation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II 19, 424–432 (Springer, 2016).
  33. Radford, A. et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 8748–8763 (PMLR, 2021).
  34. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR) (San Diega, CA, USA, 2015).
  35. Paszke, A. et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019).
  36. Generalized overlap measures for evaluation and validation in medical image analysis. IEEE transactions on medical imaging 25, 1451–1461 (2006).
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Yujin Oh (23 papers)
  2. Sangjoon Park (22 papers)
  3. Hwa Kyung Byun (5 papers)
  4. Jin Sung Kim (18 papers)
  5. Jong Chul Ye (210 papers)
  6. Yeona Cho (1 paper)
  7. Ik Jae Lee (5 papers)
Citations (7)
View on Semantic Scholar