LLM-Based Section Identifiers Excel on Open Source but Stumble in Real World Applications (2404.16294v1)
Abstract: Electronic health records (EHR) even though a boon for healthcare practitioners, are growing convoluted and longer every day. Sifting around these lengthy EHRs is taxing and becomes a cumbersome part of physician-patient interaction. Several approaches have been proposed to help alleviate this prevalent issue either via summarization or sectioning, however, only a few approaches have truly been helpful in the past. With the rise of automated methods, ML has shown promise in solving the task of identifying relevant sections in EHR. However, most ML methods rely on labeled data which is difficult to get in healthcare. LLMs on the other hand, have performed impressive feats in NLP, that too in a zero-shot manner, i.e. without any labeled data. To that end, we propose using LLMs to identify relevant section headers. We find that GPT-4 can effectively solve the task on both zero and few-shot settings as well as segment dramatically better than state-of-the-art methods. Additionally, we also annotate a much harder real world dataset and find that GPT-4 struggles to perform well, alluding to further research and harder benchmarks.
- Dhananjay Ashok and Zachary C Lipton. 2023. Promptner: Prompting for named entity recognition. arXiv preprint arXiv:2305.15444.
- Inspire the large language model by external knowledge on biomedical named entity recognition.
- Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
- CCSI. 2022. Six healthcare workflows primed for cloud faxing. https://healthitsecurity.com/news/six-healthcare-workflows-primed-for-cloud-faxing. Accessed: 2023-12-15.
- US Congress. 2009. Hr 1: American recovery and reinvestment act of 2009. Washington, DC (February 2009).
- Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
- Launching into clinical space with medspaCy: a new clinical text processing toolkit in Python. AMIA Annu Symp Proc, 2021:438–447.
- Hierarchical annotation for building a suite of clinical natural language processing tasks: Progress note understanding. In LREC… International Conference on Language Resources & Evaluation:[proceedings]. International Conference on Language Resources & Evaluation, volume 2022, page 5484. NIH Public Access.
- William R Hersh and Robert E Hoyt. 2018. Health Informatics: Practical Guide Seventh Edition. Lulu. com.
- Identifying sections in scientific abstracts using conditional random fields. In Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-I.
- Mistral 7b.
- Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1–9.
- A new public corpus for clinical section identification: Medsecid. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3709–3721.
- “note bloat” impacts deep learning-based nlp models for clinical prediction tasks. Journal of Biomedical Informatics, 133:104149.
- Deid-gpt: Zero-shot medical text de-identification by gpt-4.
- Larry McKnight and Padmini Srinivasan. 2003. Categorization of sentence types in medical abstracts. In AMIA 2003, American Medical Informatics Association Annual Symposium, Washington, DC, USA, November 8-12, 2003. AMIA.
- Clinical note section identification using transfer learning. In Proceedings of Sixth International Congress on Information and Communication Technology: ICICT 2021, London, Volume 1, pages 533–542. Springer.
- Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375.
- OpenAI. 2023. Gpt-4 technical report.
- SOAP Notes. StatPearls Publishing.
- Current approaches to identify sections within clinical narratives from electronic health records: a systematic review. BMC Medical Research Methodology, 19.
- Lance A Ramshaw and Mitchell P Marcus. 1999. Text chunking using transformation-based learning. In Natural language processing using very large corpora, pages 157–176. Springer.
- Length and Redundancy of Outpatient Progress Notes Across a Decade at an Academic Medical Center. JAMA Network Open, 4(7):e2115334–e2115334.
- Statistical section segmentation in free-text clinical records. In Lrec, pages 2001–2008.
- Llama 2: Open foundation and fine-tuned chat models.
- 2010 i2b2/va challenge on concepts, assertions, and relations in clinical text. Journal of the American Medical Informatics Association, 18(5):552–556.
- Gpt-ner: Named entity recognition via large language models. arXiv preprint arXiv:2304.10428.
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.
- Improving model transferability for clinical note section classification models using continued pretraining. medRxiv.
- Saranya Krishnamoorthy (1 paper)
- Ayush Singh (24 papers)
- Shabnam Tafreshi (6 papers)