Papers
Topics
Authors
Recent
2000 character limit reached

Benchmarking a foundation LLM on its ability to re-label structure names in accordance with the AAPM TG-263 report (2310.03874v1)

Published 5 Oct 2023 in physics.med-ph and cs.CL

Abstract: Purpose: To introduce the concept of using LLMs to re-label structure names in accordance with the American Association of Physicists in Medicine (AAPM) Task Group (TG)-263 standard, and to establish a benchmark for future studies to reference. Methods and Materials: The Generative Pre-trained Transformer (GPT)-4 application programming interface (API) was implemented as a Digital Imaging and Communications in Medicine (DICOM) storage server, which upon receiving a structure set DICOM file, prompts GPT-4 to re-label the structure names of both target volumes and normal tissues according to the AAPM TG-263. Three disease sites, prostate, head and neck, and thorax were selected for evaluation. For each disease site category, 150 patients were randomly selected for manually tuning the instructions prompt (in batches of 50) and 50 patients were randomly selected for evaluation. Structure names that were considered were those that were most likely to be relevant for studies utilizing structure contours for many patients. Results: The overall re-labeling accuracy of both target volumes and normal tissues for prostate, head and neck, and thorax cases was 96.0%, 98.5%, and 96.9% respectively. Re-labeling of target volumes was less accurate on average except for prostate - 100%, 93.1%, and 91.1% respectively. Conclusions: Given the accuracy of GPT-4 in re-labeling structure names of both target volumes and normal tissues as presented in this work, LLMs are poised to be the preferred method for standardizing structure names in radiation oncology, especially considering the rapid advancements in LLM capabilities that are likely to continue.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. Standardising breast radiotherapy structure naming conventions: A machine learning approach. Cancers, 15(3), 2023.
  2. Big data readiness in radiation oncology: An efficient approach for relabeling radiation therapy structures with their tg-263 standard name in real-world data sets. Advances in Radiation Oncology, 4(1):191–200, 2019.
  3. A machine learning method for relabeling arbitrary dicom structure sets to tg-263 defined labels. Journal of Biomedical Informatics, 109:103527, 2020.
  4. Machine learning for contour classification in tg-263 noncompliant databases. Journal of Applied Clinical Medical Physics, 23(9):e13662, 2022.
  5. Integrated natural language processing and machine learning models for standardizing radiotherapy structure names. Healthcare, 8(2), 2020.
  6. Multimodal deep learning methods on image and textual data to predict radiotherapy structure names. BioMedInformatics, 3(3):493–513, 2023.
  7. Retrospective analysis of breast radiotherapy treatment plans: Curating the ‘non-curated’. Journal of Medical Imaging and Radiation Oncology, 63(4):517–529, 2019.
  8. Alexis Andrew Miller. A rational informatics-enabled approach to standardised nomenclature of contours and volumes in radiation oncology planning. Journal of Radiation Oncology Informatics, 6(1):53–97, Oct. 2017.
  9. Head and neck radiation therapy patterns of practice variability identified as a challenge to real-world big data: Results from the learning from analysis of multicentre big data aggregation (lambda) consortium. Advances in Radiation Oncology, 8(1):100925, 2023.
  10. A national approach for automated collection of standardized and population-based radiation therapy data in sweden. Radiotherapy and Oncology, 119(2):344–350, 2016.
  11. Bulk evaluation and comparison of radiotherapy treatment plans for breast cancer. Australasian physical & engineering sciences in medicine, 39:633–644, 2016.
  12. Radiotherapy standardisation and artificial intelligence within the national cancer institute’s clinical trials network. Clinical Oncology, 34(2):128–134, 2022.
  13. Technical note: An open source solution for improving tg-263 compliance. Journal of Applied Clinical Medical Physics, 20(9):163–165, 2019.
  14. Nomenclature standardization of radiotherapy in cervical cancer. Chinese Journal of Radiation Oncology, pages 180–185, 2021.
  15. Machine learning in radiation oncology: Opportunities, requirements, and needs. Front Oncol, 8:110, 2018. 2234-943x Feng, Mary Valdes, Gilmer Dixit, Nayha Solberg, Timothy D Journal Article Switzerland 2018/05/03 Front Oncol. 2018 Apr 17;8:110. doi: 10.3389/fonc.2018.00110. eCollection 2018.
  16. Empirical relative biological effectiveness (rbe) for mandible osteoradionecrosis (orn) in head and neck cancer patients treated with pencil-beam-scanning proton therapy (pbspt): a retrospective, case-matched cohort study. Frontiers in Oncology, 12:843175, 2022.
  17. Seed spots analysis to characterize linear energy transfer (let) effect in the adverse event regions of head and neck cancer patients treated by intensity modulated proton therapy (impt). International journal of radiation oncology, biology, physics, 111(3):e388, 2021.
  18. Exploratory investigation of dose-linear energy transfer (let) volume histogram (dlvh) for adverse events study in intensity modulated proton therapy (impt). International Journal of Radiation Oncology* Biology* Physics, 110(4):1189–1199, 2021.
  19. Cardiopulmonary toxicity following intensity-modulated proton therapy (impt) vs. intensity-modulated radiation therapy (imrt) for stage iii non-small cell lung cancer. Clinical Lung Cancer, 2022.
  20. Early outcomes of patients with locally advanced non-small cell lung cancer treated with intensity-modulated proton therapy versus intensity-modulated radiation therapy: the mayo clinic experience. Advances in Radiation Oncology, 5(3):450–458, 2020.
  21. Acute toxicities and short-term patient outcomes after intensity-modulated proton beam radiation therapy or intensity-modulated photon radiation therapy for esophageal carcinoma: a mayo clinic experience. Advances in Radiation Oncology, 5(5):871–879, 2020.
  22. Intensity modulated proton therapy for hepatocellular carcinoma: initial clinical experience. Advances in Radiation Oncology, 6(4):100675, 2021.
  23. Artificial intelligence in radiation oncology. Nature Reviews Clinical Oncology, 17(12):771–781, 2020.
  24. Beam mask and sliding window-facilitated deep learning-based accurate and efficient dose prediction for pencil beam scanning proton therapy. arXiv preprint arXiv:2305.18572, 2023.
  25. Accurate and efficient deep neural network based deformable image registration method in lung cancer. In MEDICAL PHYSICS, volume 49, pages E148–E148. WILEY 111 RIVER ST, HOBOKEN 07030-5774, NJ USA, 2022.
  26. Psa-net: Deep learning–based physician style–aware segmentation network for postoperative prostate cancer clinical target volumes. Artificial Intelligence in Medicine, 121:102195, 2021.
  27. Clinicalradiobert: Knowledge-infused few shot learning for clinical notes named entity recognition. In Machine Learning in Medical Imaging: 13th International Workshop, MLMI 2022, Held in Conjunction with MICCAI 2022, Singapore, September 18, 2022, Proceedings, pages 269–278. Springer, 2022.
  28. Mask-guided bert for few shot text classification. arXiv preprint arXiv:2302.10447, 2023.
  29. Evaluating large language models on a highly-specialized topic, radiation oncology physics. Frontiers in Oncology, 13, 2023.
  30. Radonc-gpt: A large language model for radiation oncology. arXiv preprint arXiv:2309.10160, 2023.
  31. Segment anything model (sam) for radiation oncology. arXiv preprint arXiv:2306.11730, 2023.
  32. Deid-gpt: Zero-shot medical text de-identification by gpt-4. arXiv preprint arXiv:2303.11032, 2023.
  33. Chataug: Leveraging chatgpt for text data augmentation. arXiv preprint arXiv:2302.13007, 2023.
  34. Exploring the trade-offs: Unified large language models vs local fine-tuned models for highly-specific radiology nli task. arXiv preprint arXiv:2304.09138, 2023.
  35. Operational ontology for oncology (o3): A professional society-based, multistakeholder, consensus-driven informatics standard supporting clinical and research use of real-world data from patients treated for cancer. International Journal of Radiation Oncology*Biology*Physics, 117(3):533–550, 2023.
  36. American association of physicists in medicine task group 263: Standardizing nomenclatures in radiation oncology. International Journal of Radiation Oncology*Biology*Physics, 100(4):1057–1066, 2018.
  37. OpenAI. Gpt-4 technical report, 2023.
  38. Benchmarking chatgpt-4 on a radiation oncology in-training exam and red journal gray zone cases: potentials and challenges for ai-assisted medical education and decision making in radiation oncology. Frontiers in Oncology, 13:1265024, 2023.
  39. Evaluating large language models on a highly-specialized topic, radiation oncology physics. arXiv preprint arXiv:2304.01938, 2023.
  40. Artificial general intelligence for radiation oncology, 2023.
  41. Unlocking the power of chatgpt, artificial intelligence, and large language models: practical suggestions for radiation oncologists. Practical Radiation Oncology, 2023.
  42. Large language models are zero-shot reasoners. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 22199–22213. Curran Associates, Inc., 2022.
  43. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc., 2022.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now

Whiteboard

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up