Automated clinical coding using off-the-shelf large language models (2310.06552v3)

Abstract: The task of assigning diagnostic ICD codes to patient hospital admissions is typically performed by expert human coders. Efforts towards automated ICD coding are dominated by supervised deep learning models. However, difficulties in learning to predict the large number of rare codes remain a barrier to adoption in clinical practice. In this work, we leverage off-the-shelf pre-trained generative LLMs to develop a practical solution that is suitable for zero-shot and few-shot code assignment, with no need for further task-specific training. Unsupervised pre-training alone does not guarantee precise knowledge of the ICD ontology and specialist clinical coding task, therefore we frame the task as information extraction, providing a description of each coded concept and asking the model to retrieve related mentions. For efficiency, rather than iterating over all codes, we leverage the hierarchical nature of the ICD ontology to sparsely search for relevant codes.

Assessment of Automated Clinical Coding via LLMs

The paper "Automated Clinical Coding using Off-the-Shelf LLMs" presents a systematic approach to ICD coding by leveraging generative LLMs such as Llama-2, GPT-3.5, and GPT-4, without the necessity of task-specific training. This paper provides an innovative framework that addresses the perennial challenge of automating the assignment of International Classification of Disease (ICD) codes, which are integral to a myriad of healthcare functions including billing, resource management, and epidemiological studies.

Core Methodological Approach

The authors adopt a methodology that exploits generative LLMs' innate abilities for language comprehension and pattern recognition to perform zero-shot and few-shot code assignments. Instead of conventional supervised learning, which confronts challenges due to rare code distributions, this paper reframes ICD coding as an information retrieval task. It utilizes the hierarchical nature of the ICD ontology to conduct a sparse, efficient tree-search for relevant codes. The proposed method enables the model to dynamically assess the relevance of each branch of the ICD taxonomy based on textual descriptions and progressively traverse to assignable codes.

Empirical Evaluation

The method was empirically evaluated using the CodiEsp dataset, a Spanish corpus of clinical documents, supplemented with machine translations to English. This dataset provided a diverse testing ground because of its extensive span-level annotations, although the evaluations considered document-level labels as applicable in real clinical environments. The authors report their tree-search strategy achieving a macro-F1 score of 0.225, outperforming existing models on rare codes, albeit showing a slight compromise in micro-F1 metrics (0.157 compared to 0.219 for PLM-ICD). This constitutes a significant advancement, particularly for applications requiring awareness and adaptability to seldom-encountered instances.

Implications and Future Directions

The implications of this paper are multifaceted. Practically, the approach alleviates the burdens associated with manual coding, optimizing both time and accuracy, thereby promising effective integration in healthcare systems. Theoretically, it questions the current dependencies on vast labeled databases for model training, accentuating the potential of LLMs in semantic synthesis and inference even in under-represented data scenarios.

Future explorations could be directed towards enhancing the model's precision through refined prompt engineering and considering ICD-specific proprietary rules within its logic. Furthermore, transferring this framework to new taxonomy revisions, such as ICD-11, may provide expansive utility across diverse medical environments, fostering an adaptable, future-proof coding infrastructure.

In conclusion, this research substantially contributes to advancing automated ICD coding by innovatively employing LLMs, endorsing the pivot from supervised to zero-shot strategies in addressing the complexities associated with medical coding applications. This work sets a critical precedent for ensuing developments in the field, spotlighting the importance of utilizing advanced computational linguistics in solving domain-specific challenges.