Zero-Shot Clinical Trial Patient Matching with LLMs

Abstract: Matching patients to clinical trials is a key unsolved challenge in bringing new drugs to market. Today, identifying patients who meet a trial's eligibility criteria is highly manual, taking up to 1 hour per patient. Automated screening is challenging, however, as it requires understanding unstructured clinical text. LLMs offer a promising solution. In this work, we explore their application to trial matching. First, we design an LLM-based system which, given a patient's medical history as unstructured clinical text, evaluates whether that patient meets a set of inclusion criteria (also specified as free text). Our zero-shot system achieves state-of-the-art scores on the n2c2 2018 cohort selection benchmark. Second, we improve the data and cost efficiency of our method by identifying a prompting strategy which matches patients an order of magnitude faster and more cheaply than the status quo, and develop a two-stage retrieval pipeline that reduces the number of tokens processed by up to a third while retaining high performance. Third, we evaluate the interpretability of our system by having clinicians evaluate the natural language justifications generated by the LLM for each eligibility decision, and show that it can output coherent explanations for 97% of its correct decisions and 75% of its incorrect ones. Our results establish the feasibility of using LLMs to accelerate clinical trial operations.

• The paper demonstrates a zero-shot LLM system that evaluates unstructured clinical text against trial criteria without task-specific training.
• It employs a two-stage retrieval pipeline to reduce token usage and computational costs while improving Macro-F1 and Micro-F1 scores.
• The approach provides interpretable eligibility rationales, supporting efficient patient matching with potential for human-in-the-loop oversight.

Essay: Zero-Shot Clinical Trial Patient Matching with LLMs

The integration of Artificial Intelligence, particularly LLMs, in clinical trial operations offers promising prospects for enhancing efficiency in patient recruitment. The paper "Zero-Shot Clinical Trial Patient Matching with LLMs" explores the application of LLMs for automating the selection of eligible patients for clinical trials. This research is primarily concerned with designing a zero-shot LLM-based system that can evaluate patient eligibility based on unstructured clinical text against a trial's inclusion criteria, all without any additional training specific to this task.

Summary of Contributions

1. Zero-Shot Capability: The paper explores the zero-shot capabilities of LLMs, specifically focusing on GPT-4, to process and evaluate clinical data for trial matching. Unlike traditional systems, no pre-training or in-context examples are required, thus enabling a direct address of various clinical trials by merely amending the trial criteria provided to the model.
2. Data and Cost Efficiency: By adopting a novel prompting strategy, the research introduces a two-stage retrieval pipeline, which significantly reduces both data and computational costs. This pipeline curtails token usage by prefacing the retrieval step with a lightweight, embedding-based filtration of patient notes, thus alleviating the load on the more computationally intensive LLM.
3. Interpretability: The interpretability of LLM decisions is also a focal point, as the system is structured to output rationales for each eligibility determination. Clinicians assessed these rationales, finding coherent explanations in a significant portion of both accurate and inaccurate model outputs, suggesting promise for human-in-the-loop oversight implementations.

Empirical Results

The system was benchmarked using the 2018 n2c2 Clinical Trial Cohort Selection dataset, achieving state-of-the-art results on this dataset. The zero-shot system improved Macro-F1 and Micro-F1 scores by +6 and +2 points, respectively, compared to previous best models. While open-source models like Llama-2 and Mixtral lag behind GPT-4, the latter's robust performance underscores the leap in LLM efficacy for tasks involving complex clinical linguistics and structures.

Practical and Theoretical Implications

Practically, the implementation of this zero-shot LLM-based system in clinical settings could substantially mitigate the manual labor and resources traditionally required in patient trial matching. By leveraging the scalable nature of LLMs, health systems could potentially conduct large-scale screening of patient populations against trial eligibility criteria rapidly and with substantially lower costs. This could lead to more efficient patient recruitment processes, thereby reducing the financial and time burdens commonly experienced in clinical trials.

Theoretically, the findings demonstrate the maturation of LLM technology, showcasing its capability to excel in highly specialized tasks without the need for extensive domain-specific tuning. This reinforces the emerging paradigm in NLP where general-purpose models can adapt to nuanced domains through more sophisticated retrieval and prompting methodologies.

Future Directions

Looking forward, several challenges and opportunities remain. Key among them is the need to validate the generalizability of these findings across other datasets and real-world trial settings. Additionally, exploring how these models might integrate with evolving criteria structures or dynamically generated criteria could further enhance their utility. Furthermore, addressing concerns around data privacy and ensuring compliance with healthcare regulations will remain crucial as these models transition from research to practice.

In conclusion, the paper presents a compelling case for the application of LLMs in clinical trial matching, providing a scalable, efficient, and interpretable solution that aligns with both current technological capabilities and clinical demands. It sets the stage for future research that could refine these methodologies, extending their applicability across the broader landscape of healthcare operations.