Large Language Models are Few-Shot Clinical Information Extractors (2205.12689v2)

Abstract: A long-running goal of the clinical NLP community is the extraction of important variables trapped in clinical notes. However, roadblocks have included dataset shift from the general domain and a lack of public clinical corpora and annotations. In this work, we show that LLMs, such as InstructGPT, perform well at zero- and few-shot information extraction from clinical text despite not being trained specifically for the clinical domain. Whereas text classification and generation performance have already been studied extensively in such models, here we additionally demonstrate how to leverage them to tackle a diverse set of NLP tasks which require more structured outputs, including span identification, token-level sequence classification, and relation extraction. Further, due to the dearth of available data to evaluate these systems, we introduce new datasets for benchmarking few-shot clinical information extraction based on a manual re-annotation of the CASI dataset for new tasks. On the clinical extraction tasks we studied, the GPT-3 systems significantly outperform existing zero- and few-shot baselines.

Analysis of "LLMs are Few-Shot Clinical Information Extractors"

The paper under review explores leveraging LLMs such as InstructGPT for extracting clinical information from medical text using few-shot learning techniques. The authors address a critical objective in clinical NLP — extracting relevant information embedded in free-text clinical notes, which traditional NLP tools struggle due to irregular language and ambiguous terminologies.

Key Contributions and Methodology

The paper makes three pivotal contributions: the introduction of new datasets tailored for benchmarking few-shot clinical information extraction; demonstration of how LLMs can replace complex hand-tailored systems for clinical NLP tasks; and, the introduction of guided prompt design for structured LLM outputs.

1. Datasets and Evaluation: The authors manually re-annotate the CASI dataset for tasks including sense disambiguation, evidence extraction, sequence classification, and coreference resolution to establish a benchmark for few-shot learning models. This effort bridges a notable gap in the clinical NLP domain where publicly available datasets are limited due to data sensitivity.
2. Prompt-Based Learning: The methodology employs a promising approach called prompt-based learning, where large models are fine-tuned with task-specific prompts without retraining significant underlying parameters. This facilitates zero- and few-shot learning across diversified NLP tasks including relation extraction and entity recognition.
3. Guided Prompt Design: Introducing guided one-shot examples to format outputs aligns LLM responses with structured label spaces, significantly reducing the post-processing complexities associated with unstructured LLM outputs.

Results and Impact

Performance Across Tasks: Across the board, the application of GPT-3 with guided prompts and simple post-processing, termed Resolved GPT-3, either matches or exceeds the performance of existing baselines which include state-of-the-art fine-tuned models. For example, in sense disambiguation, Resolved GPT-3 outperforms models specifically trained on clinical text, suggesting that even when domain-specific data is scarce, LLMs can be effective with few-shot configurations.

Weak Supervision: A potent takeaway from the paper is the proposal to use outputs from GPT-3 as a weak supervision tool to inform the training of smaller models. This mechanism potentially enhances deployability while retaining LLM-backed performance gains.

Theoretical and Practical Implications

The findings elucidate that LLMs, despite being trained on general data, can successfully be employed for domain-specific tasks with minimal task-specific data. This asserts a significant theoretical shift showing that model architecture and in-context learning are potent vectors to achieve high performance without exhaustive supervised datasets.

Practically, the work showcases an immediate application in clinical settings. It offers a scalable alternative for clinical text mining, which otherwise is constrained by labor-intensive manual curation or brittle rule-based techniques. By efficiently utilizing small, annotated datasets and enhancing entity extraction through large models, the authors open avenues for broader accessibility to advanced NLP solutions in healthcare, without data-intensive retraining.

Future Perspectives

Given the promising results, subsequent explorations could include:

• Model Transparency: Increasing model interpretability, helping stakeholders comprehend and trust AI decisions.
• Integration with Clinical Workflows: Embedding these few-shot learning techniques into EHR systems for real-time data abstraction could transform clinician documentation practices.
• Multi-Language Adaptability: Adapting these models to different languages, crucial for non-English speaking regions' healthcare advancements.

This paper underscores a significant advance in clinical NLP, paving the way for further research into efficient learning methods using minimal data in sensitive domains. The practical adoption of LLMs for structured information extraction could transform patient record management, clinical summarization tasks, and beyond, promising substantial utility in both clinical settings and research domains.