Bespoke Large Language Models for Digital Triage Assistance in Mental Health Care (2403.19790v1)
Abstract: Contemporary LLMs may have utility for processing unstructured, narrative free-text clinical data contained in electronic health records (EHRs) -- a particularly important use-case for mental health where a majority of routinely-collected patient data lacks structured, machine-readable content. A significant problem for the the United Kingdom's National Health Service (NHS) are the long waiting lists for specialist mental healthcare. According to NHS data, in each month of 2023, there were between 370,000 and 470,000 individual new referrals into secondary mental healthcare services. Referrals must be triaged by clinicians, using clinical information contained in the patient's EHR to arrive at a decision about the most appropriate mental healthcare team to assess and potentially treat these patients. The ability to efficiently recommend a relevant team by ingesting potentially voluminous clinical notes could help services both reduce referral waiting times and with the right technology, improve the evidence available to justify triage decisions. We present and evaluate three different approaches for LLM-based, end-to-end ingestion of variable-length clinical EHR data to assist clinicians when triaging referrals. Our model is able to deliver triage recommendations consistent with existing clinical practices and it's architecture was implemented on a single GPU, making it practical for implementation in resource-limited NHS environments where private implementations of LLM technology will be necessary to ensure confidential clinical data is appropriately controlled and governed.
- NHS Digital. Mental health services monthly statistics dashboard. Technical report, 2024. URL https://digital.nhs.uk/data-and-information/data-tools-and-services/data-services/mental-health-data-hub/dashboards/mental-health-services-monthly-statistics.
- Large language models encode clinical knowledge. Nature, 620(7972):172–180, 2023.
- OpenAI. GPT-4 Technical Report, March 2023. URL http://arxiv.org/abs/2303.08774. arXiv:2303.08774 [cs].
- Claude 2. URL https://www.anthropic.com/news/claude-2.
- MLaaS: Machine Learning as a Service. In 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA), pages 896–902, Miami, FL, USA, December 2015. IEEE. ISBN 978-1-5090-0287-0. doi: 10.1109/ICMLA.2015.152.
- ChatGPT and mental healthcare: Balancing benefits with risks of harms. BMJ Ment Health, 26(1), November 2023. ISSN 2755-9734. doi: 10.1136/bmjment-2023-300884.
- Ethical Considerations of Using ChatGPT in Health Care. Journal of Medical Internet Research, 25(1):e48009, August 2023. doi: 10.2196/48009.
- NHS Digital. Mental health bulletin: 2019-20 annual report. Technical report, 2020. URL https://digital.nhs.uk/data-and-information/publications/statistical/mental-health-bulletin/2019-20-annual-report.
- A qualitative study of referral to community mental health teams in the uk: exploring the rhetoric and the reality. BMC Health Services Research, 7(1):1–9, 2007a.
- Hidden waits force more than three quarters of mental health patients to seek help from emergency services. https://www.rcpsych.ac.uk/news-and-features/latest-news/detail/2022/10/10/hidden-waits-force-more-than-three-quarters-of-mental-health-patients-to-seek-help-from-emergency-services.
- Med7: A transferable clinical natural language processing model for electronic health records. Artificial Intelligence in Medicine, 118:102086, August 2021. ISSN 1873-2860. doi: 10.1016/j.artmed.2021.102086.
- Multi-domain Clinical Natural Language Processing with MedCAT: the Medical Concept Annotation Toolkit, March 2021. URL http://arxiv.org/abs/2010.01165. arXiv:2010.01165 [cs].
- Longformer: The long-document transformer. 2020. URL https://arxiv.org/abs/2004.05150.
- RoBERTa: A Robustly Optimized BERT Pretraining Approach, July 2019. URL http://arxiv.org/abs/1907.11692. arXiv:1907.11692 [cs].
- BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423.
- R. E. Kendell. Diagnostic Criteria of American and British Psychiatrists. Archives of General Psychiatry, 25(2):123, August 1971. ISSN 0003-990X. doi: 10.1001/archpsyc.1971.01750140027006.
- Unpacking the Differences in US/UK Rates of Clinical Diagnoses of Early-Onset Bipolar Disorder. Journal of the American Academy of Child & Adolescent Psychiatry, 53(6):609–611, June 2014. ISSN 0890-8567. doi: 10.1016/j.jaac.2014.02.013.
- Reasoning like a doctor or like a nurse? A systematic integrative review. Frontiers in Medicine, 10, 2023. ISSN 2296-858X.
- National Prescription Patterns of Antidepressants in the Treatment of Adults With Major Depression in the US Between 1996 and 2015: A Population Representative Survey Based Analysis. Frontiers in Psychiatry, 11, 2020. ISSN 1664-0640.
- Clinicalbert: Modeling clinical notes and predicting hospital readmission, 2019. URL https://arxiv.org/abs/1904.05342.
- Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72–78, Minneapolis, Minnesota, USA, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-1909. URL https://aclanthology.org/W19-1909.
- Clinical Prompt Learning With Frozen Language Models. IEEE Transactions on Neural Networks and Learning Systems, pages 1–11, 2023. ISSN 2162-2388. doi: 10.1109/TNNLS.2023.3294633. URL https://ieeexplore.ieee.org/document/10215061.
- Challenges and opportunities beyond structured data in analysis of electronic health records. WIREs Computational Statistics, 13(6), November 2021. ISSN 1939-5108, 1939-0068. doi: 10.1002/wics.1549. URL https://onlinelibrary.wiley.com/doi/10.1002/wics.1549.
- Challenges in clinical natural language processing for automated disorder normalization. Journal of Biomedical Informatics, 57:28–37, 2015. ISSN 1532-0464. doi: https://doi.org/10.1016/j.jbi.2015.07.010. URL https://www.sciencedirect.com/science/article/pii/S1532046415001501.
- Estimating redundancy in clinical text. Journal of Biomedical Informatics, 124:103938, December 2021. ISSN 1532-0480. doi: 10.1016/j.jbi.2021.103938.
- Llama 2: Open Foundation and Fine-Tuned Chat Models, July 2023. URL http://arxiv.org/abs/2307.09288. arXiv:2307.09288 [cs].
- PaLM: Scaling Language Modeling with Pathways, October 2022. arXiv:2204.02311 [cs].
- A Comparison of Representation Learning Methods for Medical Concepts in MIMIC-IV, August 2022. URL https://www.medrxiv.org/content/10.1101/2022.08.21.22278835v1. Pages: 2022.08.21.22278835.
- Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. npj Digital Medicine, 4(1):1–13, May 2021. ISSN 2398-6352. doi: 10.1038/s41746-021-00455-y. URL https://www.nature.com/articles/s41746-021-00455-y. Publisher: Nature Publishing Group.
- Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequences, April 2022. URL http://arxiv.org/abs/2201.11838. arXiv:2201.11838 [cs].
- Clinical Outcome Prediction from Admission Notes using Self-Supervised Knowledge Integration. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 881–893, Online, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.eacl-main.75. URL https://aclanthology.org/2021.eacl-main.75.
- PLM-ICD: Automatic ICD Coding with Pretrained Language Models. In Proceedings of the 4th Clinical Natural Language Processing Workshop, pages 10–20, Seattle, WA, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.clinicalnlp-1.2. URL https://aclanthology.org/2022.clinicalnlp-1.2.
- Explaining decisions made with AI. https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/artificial-intelligence/explaining-decisions-made-with-artificial-intelligence/, February 2024.
- A qualitative study of referral to community mental health teams in the UK: exploring the rhetoric and the reality. BMC Health Services Research, 7(1):117, July 2007b. ISSN 1472-6963. doi: 10.1186/1472-6963-7-117. URL https://doi.org/10.1186/1472-6963-7-117.
- Developing nhs language model embedding spaces, April 2024.
- Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
- MIMIC-III, a freely accessible critical care database. Scientific Data, 3, May 2016. doi: 10.1038/sdata.2016.35. Publisher: Nature Publishing Groups.
- A Label Attention Model for ICD Coding from Clinical Text. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, pages 3335–3341, July 2020. doi: 10.24963/ijcai.2020/461. URL http://arxiv.org/abs/2007.06351. arXiv:2007.06351 [cs].
- Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’23, pages 2572–2582, New York, NY, USA, July 2023. Association for Computing Machinery. ISBN 978-1-4503-9408-6. doi: 10.1145/3539618.3591918. URL https://dl.acm.org/doi/10.1145/3539618.3591918.
- Explainable Prediction of Medical Codes from Clinical Text. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1101–1111, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-1100. URL https://aclanthology.org/N18-1100.
- LoRA: Low-Rank Adaptation of Large Language Models, October 2021. URL http://arxiv.org/abs/2106.09685. arXiv:2106.09685 [cs].
- Explainable artificial intelligence for mental health through transparency and interpretability for understandability. npj Digital Medicine, 6(1):6, 2023.
- Extracting Training Data from Large Language Models, June 2021. URL http://arxiv.org/abs/2012.07805. arXiv:2012.07805 [cs].
- Poisoning Web-Scale Training Datasets is Practical, February 2023. URL http://arxiv.org/abs/2302.10149. arXiv:2302.10149 [cs].
- Laurens van der Maaten and Geoffrey Hinton. Visualizing Data using t-SNE. Journal of Machine Learning Research, 9(86):2579–2605, 2008. ISSN 1533-7928. URL http://jmlr.org/papers/v9/vandermaaten08a.html.
- Niall Taylor (8 papers)
- Andrey Kormilitzin (22 papers)
- Isabelle Lorge (5 papers)
- Alejo Nevado-Holgado (14 papers)
- Dan W Joyce (3 papers)