The Need for Guardrails with Large Language Models in Medical Safety-Critical Settings: An Artificial Intelligence Application in the Pharmacovigilance Ecosystem (2407.18322v2)

Abstract: LLMs are useful tools with the capacity for performing specific types of knowledge work at an effective scale. However, LLM deployments in high-risk and safety-critical domains pose unique challenges, notably the issue of ``hallucination,'' where LLMs can generate fabricated information. This is particularly concerning in settings such as drug safety, where inaccuracies could lead to patient harm. To mitigate these risks, we have developed and demonstrated a proof of concept suite of guardrails specifically designed to mitigate certain types of hallucinations and errors for drug safety, and potentially applicable to other medical safety-critical contexts. These guardrails include mechanisms to detect anomalous documents to prevent the ingestion of inappropriate data, identify incorrect drug names or adverse event terms, and convey uncertainty in generated content. We integrated these guardrails with an LLM fine-tuned for a text-to-text task, which involves converting both structured and unstructured data within adverse event reports into natural language. This method was applied to translate individual case safety reports, demonstrating effective application in a pharmacovigilance processing task. Our guardrail framework offers a set of tools with broad applicability across various domains, ensuring LLMs can be safely used in high-risk situations by eliminating the occurrence of key errors, including the generation of incorrect pharmacovigilance-related terms, thus adhering to stringent regulatory and quality standards in medical safety-critical environments.

This paper explores the use of LLMs in pharmacovigilance (PV), a safety-critical domain focused on monitoring adverse events associated with medications and vaccines. The core challenge addressed is the potential for LLMs to "hallucinate" or generate incorrect information, which could have serious consequences for patient safety in PV.

To mitigate these risks, the authors develop a suite of "guardrails" designed to improve the accuracy and reliability of LLMs in this context. These guardrails include:

1. Document-level Uncertainty Quantification (DL-UQ): This "soft" guardrail identifies and filters out documents that are unlikely to be valid Individual Case Safety Reports (ICSRs). This prevents the LLM from processing irrelevant or inappropriate data. DL-UQ is based on calculating the distance between the embedding of a submitted document and a cache of ICSR embeddings.
2. MISMATCH (Drug and AE mismatching): This "hard" guardrail identifies instances where drug names or adverse event (AE) terms appear in either the source text (e.g., the original Japanese ICSR) or the target text (the LLM-generated English translation) but not in both. This helps to prevent mistranslations or hallucinations of critical drug and AE information. The guardrail relies on dictionaries of drug names and AE terms (MedDRA). Mismatched instances are flagged for review, potentially requiring human intervention.
3. Token-level Uncertainty Quantification (TL-UQ): This "soft" guardrail assesses the LLM's uncertainty at the word or sub-word level. It assigns a log probability to each token in the vocabulary and uses the entropy of this probability distribution as a measure of uncertainty. High entropy indicates greater uncertainty in the model's prediction for that particular token. TL-UQ helps to flag potentially problematic sections of the generated text for closer inspection.

The authors fine-tuned several LLMs (mt5-xl, mpt-7b-instruct, and stablelm-japanese) for a text-to-text task: translating Japanese ICSRs into English narrative text. They created a multilingual corpus of ICSRs paired with human-generated summaries to train the models. The guardrails were then integrated with the fine-tuned LLM to improve its performance on this translation task.

The effectiveness of the guardrails was evaluated through expert human review and quantitative metrics. The results showed that the MISMATCH guardrail was able to successfully identify all instances of hallucinated drug names in a sample of translated ICSRs. The DL-UQ guardrail showed good discrimination between valid ICSRs and extraneous documents. The TL-UQ guardrail provided a useful signal of model correctness.

The paper concludes that LLMs have the potential to be valuable tools in PV, but that guardrails are essential to ensure their safe and reliable use in this safety-critical domain. The proposed framework offers a comprehensive approach to mitigating the risks associated with LLM hallucinations and promoting responsible AI adoption in medical settings.