MedHopQA: A Disease-Centered Multi-Hop Reasoning Benchmark and Evaluation Framework for LLM-Based Biomedical Question Answering

Abstract: Evaluating LLMs in the biomedical domain requires benchmarks that can distinguish reasoning from pattern matching and remain discriminative as model capabilities improve. Existing biomedical question answering (QA) benchmarks are limited in this respect. Multiple-choice formats can allow models to succeed through answer elimination rather than inference, while widely circulated exam-style datasets are increasingly vulnerable to performance saturation and training data contamination. Multi-hop reasoning, defined as the ability to integrate information across multiple sources to derive an answer, is central to clinically meaningful tasks such as diagnostic support, literature-based discovery, and hypothesis generation, yet remains underrepresented in current biomedical QA benchmarks. MedHopQA is a disease-centered multi-hop reasoning benchmark consisting of 1,000 expert-curated question-answer pairs introduced as a shared task at BioCreative IX. Each question requires synthesis of information across two distinct Wikipedia articles, and answers are provided in an open-ended free-text format. Gold annotations are augmented with ontology-grounded synonym sets from MONDO, NCBI Gene, and NCBI Taxonomy to support both lexical and concept-level evaluation. MedHopQA was constructed through a structured process combining human annotation, triage, iterative verification, and LLM-as-a-judge validation. To reduce leaderboard gaming and contamination risk, the 1,000 scored questions are embedded within a publicly downloadable set of 10,000 questions, with answers withheld, on a CodaBench leaderboard. MedHopQA provides both a benchmark and a reusable framework for constructing future biomedical QA datasets that prioritize compositional reasoning, saturation resistance, and contamination resistance as core design constraints.

• The paper presents a novel disease-centered, multi-hop reasoning benchmark that overcomes limitations of existing biomedical QA datasets.
• It employs expert curation with AI-assisted generation and rigorous human review to ensure factuality, stability, and genuine multi-hop integration.
• Evaluation using both lexical and concept-level metrics reveals persistent challenges in LLMs’ ability to integrate and normalize complex biomedical information.

MedHopQA: Disease-Centered Multi-Hop Reasoning Benchmark for Biomedical QA

Motivation and Benchmark Limitations

MedHopQA directly addresses critical limitations in current biomedical QA benchmarks by focusing on disease-centered, multi-hop reasoning using open-ended questions. Existing resources (e.g., MedQA, MedMCQA, PubMedQA, MMLU Medical) predominantly employ multiple-choice formats, facilitating answer selection via elimination and pattern matching rather than genuine inference. The prevalence of performance saturation—frontier LLMs scoring above 95% on established datasets—and widespread training data contamination due to overlap with pretraining corpora further diminish the discriminative power and validity of legacy benchmarks. Additionally, prior datasets overwhelmingly favor single-hop retrieval tasks, failing to capture the compositional reasoning required for clinically meaningful scenarios such as cross-source diagnosis, literature-based discovery, and hypothesis generation.

Benchmark Construction Methodology

MedHopQA comprises 1,000 expert-curated, disease-centered, multi-hop QA pairs constructed from Wikipedia article pairs. Each question mandates synthesis across two distinct documents, precluding resolution via single-source memorization. The dataset creation pipeline integrates structured human annotation, systematic triage, iterative revision, and LLM-as-a-judge validation. Human annotators with biomedical expertise formulate questions according to stringent criteria (factuality, stability, uniqueness, and genuine multi-hop requirement) from carefully filtered Wikipedia page pairs. AI augmentation (using prompt-driven QA pair generation) increases scale, but all output is blinded, triaged, and reviewed under the same acceptance thresholds as human-authored content.

Answer annotation extends beyond canonical responses to include curated synonym sets sourced from biomedical ontologies (MONDO, NCBI Gene, NCBI Taxonomy), mitigating systematic bias in evaluation. Rigorous human review checks for grammar, conceptual accuracy, answer correctness, and uniqueness, with a structured debate framework involving LLMs to further validate quality. The final validation phase ensures each answer set reflects all valid lexical and conceptual variants, minimizing false negatives in downstream evaluation.

Evaluation Framework

MedHopQA employs two principal scoring metrics: Lexical Match and Concept-Level Validation. Lexical Match normalizes predicted answers and checks against both canonical and synonym variants, accommodating biomedical surface form variability (abbreviation, capitalization, colloquial vs. technical terminology). Concept-Level Validation leverages MedCPT embeddings trained on large-scale PubMed logs, mapping answers into structured concept spaces and performing semantic similarity-based matching for entity-type responses, while numerical and non-semantic types rely on strict string matching. These parallel metrics provide robust discrimination between correct conceptual answers and merely superficially similar strings.

To further mitigate leaderboard gaming and contamination, the 1,000 scored questions are embedded within a tenfold larger package for CodaBench evaluation, with answers withheld and test items hidden. This protects against manual overfitting and exact-instance leakage common in legacy exam datasets.

Dataset Characteristics

The dataset is strongly disease-centered, spanning rare and complex conditions, and constructed from 1,300 unique Wikipedia pages, emphasizing biomedical entities and requiring genuine multi-hop integration. Answer types are dominated by yes/no (30%) and biomedical entities (59.3%), with diseases (21.1%), gene/protein (11.9%), chemical/drugs (8.4%), and chromosomes (8.7%) being most frequent. The deliberate structuring ensures coverage of challenging cases with multi-system and compositional reasoning requirements.

LLM Evaluation Results

Zero-shot evaluation on four leading LLMs (GPT-4o, GPT-5.1, Claude Sonnet 4.5, Gemini 2.5 Pro) demonstrates persistent variation and challenge in multi-hop biomedical reasoning. Overall accuracy ranges from 66.3% to 83.4%, with GPT-5.1 outperforming others. Category-level results reveal strongest performance for chemicals (84.5–89.3%) and anatomical entities (71.4–83.3%), while disease and gene/protein questions exhibit marked variability (63.0–81.5%, 60.5–80.7%) due to the need for fine-grained semantic discrimination. Numerical questions are particularly challenging, highlighting normalization and formatting sensitivities.

Manual analysis of MedCPT’s judgment reveals that most errors are false negatives—correct answers rejected due to incomplete synonym normalization—while false positives remain limited, especially for canonical biomedical entities. Agreement varies by answer category, with gene/protein and anatomical answers exhibiting 95%+ agreement.

Practical and Theoretical Implications

MedHopQA sets new standards for biomedical QA benchmarks by explicitly requiring compositional reasoning, resisting saturation, and minimizing contamination through carefully controlled construction and evaluation. Its open-ended format and rigorous synonym annotation force models to reason and normalize, rather than memorize or rely on answer cues. As a reusable framework, MedHopQA provides a practical template for developing future benchmarks targeting multi-hop, open-ended QA in biomedicine and other domains.

The results underscore that multi-hop biomedical reasoning is an unsolved challenge—even frontier models fail to consistently integrate information across sources and discriminate closely related entities. The combination of outcome- and concept-based evaluation surfaces persistent error modes and provides reliable diagnostic headroom for system improvement. The benchmark also exposes category-dependent weaknesses that researchers can target for algorithmic and architectural advancements.

Domain-specific limitations include reliance on English Wikipedia, bias from annotator backgrounds, and disease-centered scope. Future extensions will need bespoke curation for domains such as pharmacology, genomics, or procedural medicine, as well as expanded annotator diversity.

Future Directions

MedHopQA’s methodology—combining seed curation, AI-assisted generation, and iterative human review—offers scalability without sacrificing biomedical precision. Future research avenues include:

• Extending the framework to non-disease biomedical domains (e.g., pharmacology, genomics).
• Incorporating intermediate reasoning step annotation and expert evaluation.
• Periodic dataset refreshes to maintain relevance and guard against contamination.
• Leveraging more diverse annotator pools for improved coverage.

Conclusion

MedHopQA provides a benchmark and evaluation framework that overcomes the critical limitations of prior biomedical QA datasets by prioritizing disease-centered multi-hop reasoning, open-ended answer generation, and contamination resistance. Empirical evaluation demonstrates that even state-of-the-art LLMs struggle with genuine biomedical reasoning, particularly where compositional, semantic, or normalization challenges arise. The benchmark, available via CodaBench, enables standardized, blind evaluation and paves the way for robust, scalable QA dataset development in biomedicine.