- The paper introduces EpiKG, a 24,324-entity, 32,009-triplet epilepsy knowledge graph built from 48,166 publications and EpiBench, a five-task benchmark covering clinical decisions, report generation, precision medicine, treatment recommendations, and research planning.
- Graph-RAG improves every evaluated task across six LLMs, including average gains of 11.3 percentage points on clinical decision multiple-choice questions, 30–41% on pharmacogenomic reasoning, and 15.6% in treatment recommendation accuracy.
- The results show that structured multi-hop retrieval is especially valuable for drug safety and pharmacogenomics, with PPR-PCST retrieval outperforming flat semantic search and four-hop subgraphs improving performance on gene–pathway–drug reasoning chains.
EpiGraph is a framework for evidence-intensive clinical reasoning in epilepsy, comprising two components: EpiKG, a large-scale domain-specific knowledge graph, and EpiBench, a five-task benchmark for evaluating LLMs under controlled knowledge-augmentation conditions. The work addresses two gaps the authors identify in prior resources: existing biomedical knowledge graphs are either disease-agnostic or designed for semantic annotation rather than multi-hop clinical inference, and no benchmark isolates whether LLM errors in a specialized clinical domain stem from missing knowledge, weak multi-step reasoning, or inadequate retrieval.
The EpiKG knowledge graph
EpiKG is constructed through an evidence-to-graph pipeline grounded in seven authoritative resources: ILAE 2022 (syndrome taxonomy), MeSH, OMIM and HGNC (gene–disease associations), ChEBI (antiseizure medication identifiers), HPO (phenotype vocabulary), AES 2024 guidelines, and UMLS as the cross-ontology linking hub. Entities are organized into five clinical layers — syndromes, diagnostics (EEG patterns, imaging, biomarkers), genes, treatments, and outcomes — with 1,370 predefined cross-layer relation types such as caused_by_gene, treated_with, and contraindicated_with.
Evidence is drawn from over 120,000 PubMed/PMC publications indexed between 1990 and 2024. A two-stage screening process — an LLM-assisted classifier followed by expert adjudication of borderline cases — retains 48,166 papers. Relations are extracted via two complementary pipelines: rule-based pattern matching on dependency-parsed sentences (contributing 9,670 triplets, 30.2%) and LLM-based extraction using MiniMax-Text-01 over full text (22,339 triplets, 69.8%). Extracted mentions are normalized through exact match, alias resolution, sentence-transformer semantic matching, and UMLS CUI verification.
The resulting graph contains 24,324 entities and 32,009 triplets, of which 14,576 (45.5%) are cross-layer connections spanning all pairwise layer combinations. The densest cross-layer links connect Syndrome to Treatment (3,217 triplets) and Gene to Syndrome (2,845 triplets) — precisely the gene → syndrome → treatment chains that carry direct clinical meaning. Each triplet carries a paper count as an evidential-strength proxy; the median is 3 (IQR: 1–8), and 4,612 triplets are supported by at least ten independent publications. Low-evidence triplets supported by fewer than two sources are retained but flagged.
The EpiBench benchmark
EpiBench defines each task as a prediction function over a primary input, retrieved evidence context from EpiKG via Graph-RAG, and optional task-specific context. Five tasks span distinct reasoning modalities:
- Clinical Decision Accuracy (CDA): 1,000 MCQs and 5,199 open-ended questions generated from papers published after the EpiKG construction cutoff, explicitly excluding them from the graph to prevent knowledge leakage.
- Clinical Report Generation (CRG): EEG-to-impression generation on the Harvard Electroencephalography Database v4.1, using neurologist-written impressions as gold standards.
- Biomarker-Driven Precision Medicine (BPM): 151 pharmacogenomic MCQs built from CPIC and ILAE 2022 guidelines, requiring multi-hop paths such as TSC2 → mTOR pathway → Everolimus while recognizing contraindicated alternatives.
- Treatment Recommendation (TR): 472 guideline-concordance cases filtered from MedQA-USMLE and MMLU Professional Medicine.
- Deep Research Planning (DRP): 163 PMC papers, with neurologist annotations for 30 (inter-annotator agreement κ=0.81 for research question quality) and LLM-as-Judge gold standards for the remaining 133.
Metrics extend beyond standard NLP measures to include Guideline Concordance against ILAE 2022/CPIC, Drug Safety Score (contraindication avoidance), KG Evidence Coverage, Clinical NER F1, and NLI-based Hallucination Rate.
Experimental results
Six LLMs — GPT-4o, Claude Sonnet 4, Gemini 2.0 Flash, Llama-3.3-70B, Qwen2.5-72B, and Mistral Small 3.1 — are evaluated with and without Graph-RAG across more than 126,000 assessments. Graph-RAG consistently improves performance on every task, with task-dependent magnitudes:
| Task |
Headline result |
| CDA (MCQ) |
Average +11.3 pp; Mistral +19.6%, Llama +15.8% |
| CDA (open-ended) |
GPT-4o reaches 4.33/5.0 (+19%) |
| BPM |
Largest gains (+30–41%); Claude S4 reaches 82%; Qwen +42% |
| TR |
Accuracy +15.6%; Drug Safety +12.4% (MedQA), +28.1% (MMLU) |
| DRP |
LLM-as-Judge +12.2% average; GPT-4o 4.25/5.0 |
Three findings deserve emphasis. First, on pharmacogenomic reasoning, open-source models improve more than closed-source ones, reversing the usual performance gap; without KG context, Mistral scores only 38% on BPM — near the 25% random baseline for four-option MCQs — indicating that pharmacogenomic knowledge is largely absent from general-purpose LLM parameters and cannot be elicited by prompting alone. Second, on treatment recommendation, safety and guideline-concordance metrics improve far more than raw accuracy on MMLU (+28.1% and +28.4% versus +17.2%), suggesting Graph-RAG's primary value there is contraindication avoidance rather than answer correctness; the correlation between KG Evidence Coverage and Drug Safety gains supports this interpretation. Third, on research planning, the closed/open-source gap narrows substantially (Llama reaches 3.87 versus GPT-4o's 4.25 under Graph-RAG), implying that plan generation depends more on structural reasoning ability than parametric domain knowledge.
Ablations show that PPR-PCST retrieval outperforms flat semantic retrieval by +2.6–7.7 pp on CDA and +7.7–8.4 pp on BPM, with hybrid retrieval adding a further +1.2–2.4 pp. Optimal subgraph size is 30 nodes at path depth 4; BPM exhibits the strongest depth sensitivity (58.7% at depth 2 versus 69.0% at depth 4 for GPT-4o), confirming that pharmacogenomic queries require 3–4 hop traversals that flat retrieval cannot exploit. Graph-RAG also dominates MedRAG (~62%), DR.KNOWS, and AMG-RAG baselines while maintaining competitive inference time.
Limitations and open questions
The authors concede several constraints. EpiKG is built exclusively from English-language ontologies and literature, so syndromes documented primarily in non-English sources are likely underrepresented, and rare syndromes with fewer than five supporting papers may lack reliable extracted relations because the LLM extraction pipeline requires sufficient co-occurrence evidence. On T5, 133 of 163 gold standards come from LLM-as-Judge rather than human experts, introducing potential bias toward fluent outputs regardless of scientific quality. CRG evaluation is restricted to four locally deployed small models due to dataset usage restrictions, and METEOR scores remain low overall, indicating that clinically faithful EEG impression generation remains unsolved even with KG augmentation. The paper also leaves unresolved whether Graph-RAG primarily supplies missing domain knowledge or provides reasoning structure that weaker models lack — a distinction its results suggest but do not definitively separate.
Conclusion
EpiGraph contributes the first large-scale, multi-hop relational knowledge graph dedicated to epileptology together with a modular, publicly released benchmark spanning five clinical reasoning tasks. Its central empirical claim — that structured, evidence-grounded domain knowledge yields consistent improvements across all evaluated models and tasks, with the largest effects where multi-hop pharmacogenomic reasoning is required — is supported by controlled ablations over retriever strategy, subgraph size, and path depth. The framework positions future work toward retrieval-augmented fine-tuning approaches that could internalize epilepsy-specific clinical rules, and toward evaluation protocols that align generated clinical language with neurologist-level judgment rather than automated proxy metrics alone.