KGARevion: An AI Agent for Knowledge-Intensive Biomedical QA (2410.04660v2)

Abstract: Biomedical reasoning integrates structured, codified knowledge with tacit, experience-driven insights. Depending on the context, quantity, and nature of available evidence, researchers and clinicians use diverse strategies, including rule-based, prototype-based, and case-based reasoning. Effective medical AI models must handle this complexity while ensuring reliability and adaptability. We introduce KGARevion, a knowledge graph-based agent that answers knowledge-intensive questions. Upon receiving a query, KGARevion generates relevant triplets by leveraging the latent knowledge embedded in a LLM. It then verifies these triplets against a grounded knowledge graph, filtering out errors and retaining only accurate, contextually relevant information for the final answer. This multi-step process strengthens reasoning, adapts to different models of medical inference, and outperforms retrieval-augmented generation-based approaches that lack effective verification mechanisms. Evaluations on medical QA benchmarks show that KGARevion improves accuracy by over 5.2% over 15 models in handling complex medical queries. To further assess its effectiveness, we curated three new medical QA datasets with varying levels of semantic complexity, where KGARevion improved accuracy by 10.4%. The agent integrates with different LLMs and biomedical knowledge graphs for broad applicability across knowledge-intensive tasks. We evaluated KGARevion on AfriMed-QA, a newly introduced dataset focused on African healthcare, demonstrating its strong zero-shot generalization to underrepresented medical contexts.

• The paper proposes KGARevion, which integrates large language models with knowledge graphs to systematically address complex medical QA.
• It employs a multi-step Generate, Review, Revise, and Answer process to ensure accurate and comprehensive reasoning.
• KGARevion achieves up to 10.4% accuracy improvement over existing models on advanced medical QA benchmarks.

Knowledge Graph Based Agent for Complex, Knowledge-Intensive QA in Medicine

The paper introduces KGARevion, a knowledge graph-based agent designed to tackle complex medical question-answering (QA) by integrating LLMs with the domain-specific structured knowledge available in Knowledge Graphs (KGs). This agent addresses unique challenges in medical reasoning that are not adequately managed by existing QA models.

Methodological Overview

KGARevion operates through a sequence of actions: Generate, Review, Revise, and Answer. This procedural approach ensures robustness in reasoning and adaptability to various reasoning models inherent in medical contexts.

1. Generate Action:
   • The LLM generates triplets related to the input medical query, focusing on either choice-aware queries (considering answer candidates) or non-choice-aware queries.
2. Review Action:
   • Uses a fine-tuned LLM combined with pre-trained KG embeddings to verify these generated triplets, distinguishing between factually incorrect data and incomplete knowledge contexts.
3. Revise Action:
   • Iteratively corrects erroneous triplets, based on feedback from the Review action, ensuring only accurate information guides the final answer.
4. Answer Action:
   • Utilizes verified triplets to select the most relevant answer, showcasing an adaptive reasoning framework for both multi-choice and open-ended QA.

Numerical Results and Evaluation

KGARevion outperforms several models across multiple benchmarks, notably showing a 5.2% accuracy improvement over 15 comparison models on established medical QA datasets. Further evaluations on the newly curated complex medical QA set, MedDDx, demonstrated an even greater accuracy improvement of 10.4% reflecting a particular strength in handling semantically complex questions.

Implications and Future Directions

The proposed framework enhances the QA capabilities of LLMs, showing that combining LLMs with multi-source knowledge from KGs significantly boosts performance in complex, domain-specific tasks. The ability to accurately resolve complex queries broadens the applicability of machine intelligence in medicine, particularly in scenarios requiring differential diagnosis or intricate knowledge synthesis.

For future developments, KGARevion's robust architecture could explore incorporating more granular medical datasets and integrating more advanced KG completion techniques. This foresight promises further advancements in AI-driven medical applications, particularly those requiring nuanced, multi-faceted reasoning capabilities. Additionally, the adaptation to other domains with similar knowledge-intensive requirements can be considered to leverage this synergistic approach.

In summary, KGARevion exemplifies a significant progression toward handling the intricacies of medical reasoning with AI, setting a precedent for future integrative approaches in complex QA systems.