MedHopQA: Biomedical Multi-Hop QA Benchmark
- MedHopQA is a disease-centered benchmark that requires integrating evidence from two linked Wikipedia pages to generate open-ended answers.
- It employs a rigorous construction pipeline with human and AI curation, ontology-based answer normalization, and a hidden evaluation set to ensure answer precision.
- Empirical results from the BioCreative IX shared task show that retrieval-augmented methods and careful answer canonicalization are key to overcoming multi-hop reasoning challenges.
MedHopQA is a disease-centered benchmark for biomedical multi-hop question answering that was introduced through BioCreative IX to evaluate whether LLMs can integrate distributed biomedical evidence rather than rely on pattern matching, answer elimination, or saturated exam-style test sets. Its core task is open-ended free-text answer generation for questions whose solutions require synthesis across two distinct Wikipedia articles, with official scoring performed on 1,000 expert-curated question-answer pairs embedded inside a larger 10,000-question public package. The benchmark therefore combines a dataset, a shared-task evaluation protocol, and an answer-normalization framework intended for open-ended biomedical QA (Islamaj et al., 12 May 2026, Islamaj et al., 12 May 2026).
1. Origins, rationale, and design principles
MedHopQA was created in response to four limitations identified in prior biomedical QA evaluation: multiple-choice shortcutting, performance saturation on popular medical benchmarks, training-data contamination, and weak reasoning requirements. The benchmark paper argues that many widely used resources in biomedicine and medicine are either effectively single-hop or structurally vulnerable to test-taking strategies that do not require compositional inference. MedHopQA instead treats multi-hop reasoning as central to clinically meaningful tasks such as diagnostic support, literature-based discovery, and hypothesis generation, and it makes open-ended answer production a first-class requirement rather than a secondary reporting mode (Islamaj et al., 12 May 2026).
The benchmark is explicitly disease-centered. Construction begins from Wikipedia’s curated list of roughly 5,000 disease pages, and each final question is anchored in a pair comprising a disease page and a second biomedically relevant page linked from it. This keeps the task focused on disease-related reasoning while allowing the second hop to involve genes or proteins, chemicals or drugs, anatomical entities, pathogens, symptoms, chromosomes, and related biomedical concepts. Wikipedia was chosen deliberately: the benchmark is not designed to test access to obscure sources, but whether models can infer relations across sources even when the underlying facts are likely to be within general pretraining exposure (Islamaj et al., 12 May 2026).
A question is valid only if it is self-contained, factual, stable, uniquely answerable, and genuinely at least two-hop. The benchmark paper’s representative example makes the intended structure concrete: from the pages for Tetanus and GABA, the question asks, “Tetanus toxin blocks release of two inhibitory neurotransmitters. Which one is synthesized from glutamate by glutamate decarboxylase?” The first page yields the candidate neurotransmitters, and the second resolves which one is synthesized from glutamate. This is the defining MedHopQA pattern: neither page alone suffices, but the pair supports a short, indisputable answer (Islamaj et al., 12 May 2026).
2. Construction pipeline and dataset composition
MedHopQA’s final benchmark consists of 1,000 expert-curated question-answer pairs, but those scored items are hidden inside a publicly downloadable 10,000-question package. This hidden-subset design is a central anti-gaming measure: participants can run systems on the full pool, yet only the concealed 1,000-question subset is scored, and the answers are withheld. The benchmark paper does not present a conventional train/dev/test split for the benchmark itself; instead, it emphasizes the public package, the hidden scored subset, and the BioCreative shared-task deployment (Islamaj et al., 12 May 2026).
The construction workflow is organized into five phases: seed, augment, triage, verify, and evaluate. Sixteen annotators with backgrounds in medicine, biomedicine, bioinformatics, and informatics authored initial questions from Wikipedia page pairs. Outgoing links from seed pages were capped at 20, and links in References or External Links were excluded. Candidate second pages were filtered to retain biomedical relevance. Human-generated items were then expanded through an o1-based augmentation module prompted on batches of 100 human-written QA pairs. AI-generated items entered the same review pipeline as human-authored ones, and reviewers were blinded to authorship source (Islamaj et al., 12 May 2026).
The triage protocol was stringent. Advancement required acceptance by two independent reviewers; human-authored rejected questions could be revised, but rejected AI-authored questions were discarded. Acceptance reportedly rose from about 10% in the initial weeks to about 50% later in the process. By the time an item became a verified QA pair, it had been revised and or validated by at least three researchers. Final selection involved four researchers and an LLM-as-a-judge debate framework organized around grammar, conceptual correctness, answer validity, and answer uniqueness, with humans retaining final authority (Islamaj et al., 12 May 2026).
The benchmark is built from 1,300 unique Wikipedia pages. The benchmark paper reports the answer-type distribution as 29.7% yes/no, 21.1% disease, 11.9% gene/protein, 8.4% chemical/drug, 8.7% chromosome, 4.2% anatomical part, 2.6% sign/symptom, 2.9% numerical, 2.2% species, 2.0% description, and 6.3% other, with biomedical entity-based answers collectively accounting for 59.3% of the dataset. Answers are intentionally short, averaging approximately two words. Disease coverage includes rare and congenital disorders, developmental anomalies, cancers, infections, and clinical findings, and nearly three-quarters of diseases were assigned to multiple categories, which the authors argue contributes directly to the need for multi-hop reasoning (Islamaj et al., 12 May 2026).
3. Evaluation framework and answer normalization
Open-ended biomedical QA makes evaluation unusually difficult because lexical mismatch and conceptual mismatch are not the same phenomenon. MedHopQA addresses this by augmenting each gold answer with a curated lexicon of valid alternative forms grounded in resources such as MONDO, NCBI Gene, and NCBI Taxonomy, with additional supplemental synonyms from Wikipedia or the web under substantial human oversight. Ontology synonym inventories were not accepted automatically; contextual review was required because biomedical synonymy is often question-dependent rather than globally interchangeable (Islamaj et al., 12 May 2026).
For leaderboard scoring, the benchmark uses a lightweight lexical evaluation. The benchmark paper describes this as Lexical Match, while the shared-task overview clarifies that Codabench used string matching against a manually curated synonym lexicon and that this was not a simple exact-match comparison. This distinction matters because participant system papers often describe the lexical metric as EM, whereas the official overview presents the online metric as synonym-aware lexical matching and ranks systems by F1 under both Lexical Match and MedCPT (Islamaj et al., 12 May 2026, Islamaj et al., 12 May 2026).
The second evaluation layer is concept-level validation with MedCPT. For entity-based answer types such as disease, gene/protein, chemical, anatomy, sign/symptom, and species, predictions are embedded with MedCPT, projected into the appropriate domain lexicon by nearest-neighbor search, normalized, and then checked against curated gold synonym sets. For open or weakly structured categories such as person, date, procedure, treatment, and other descriptive answers, predictions are compared directly to curated references in embedding space, with correctness assigned iff the maximum cosine similarity exceeds 0.7. For non-semantic answer types such as yes/no, numerical, and chromosome location, lexical procedures are retained because semantic similarity is not well defined for scoring purposes (Islamaj et al., 12 May 2026).
The benchmark paper also reports a manual audit of 200 MedCPT decisions, with 151 correct judgments, corresponding to 75.5% agreement. Among the 49 errors, 28 were false negatives, 19 false positives, and 2 mixed or partial mismatches. MedCPT was most reliable for canonical biomedical entities such as gene/protein, anatomical, sign/symptom, and chemical answers, and less reliable for weakly structured categories. A persistent theme across MedHopQA evaluation is therefore that semantic correctness and leaderboard compliance are related but non-identical objectives (Islamaj et al., 12 May 2026).
4. Shared task, participation, and empirical performance
The MedHopQA shared task was announced in March 2025 as part of BioCreative IX, with an official evaluation period from May 27 to June 1, 2025. Each team could submit up to five runs. The challenge drew 42 registered teams, 13 official submitting teams, 48 official submissions, and 19 unofficial submissions in a later extension window; nine workshop papers were accepted. The shared-task overview identifies retrieval-augmented generation and related retrieval-based strategies as the dominant methodological pattern, especially among top systems (Islamaj et al., 12 May 2026).
The official best-run results, ranked by F1 under the Lexical Match and MedCPT metrics, were as follows:
| System | Lexical Match | MedCPT |
|---|---|---|
| DMIS Lab | 87.3 | 89.3 |
| UETQuintet | 83.8 | 86.0 |
| Insilicom | 80.0 | 83.1 |
| PreceptorAI | 73.4 | 78.8 |
| Baseline | 59.1 | 68.3 |
| CaresAI | 18.6 | 31.1 |
These results show a large gap between zero-shot prompting and retrieval-heavy competition systems, but also a nontrivial baseline: 23 of 48 official runs, or 48%, exceeded the zero-shot GPT-4o baseline, which implies that nearly half did not (Islamaj et al., 12 May 2026).
The shared-task overview emphasizes that retrieval-based systems were decisive. The top official system, DMIS Lab, combined Query2Doc-style reformulation, BM25 plus MedCPT reranking, rationale-driven subquery generation, Google Search, and a final o3-based decision module. UETQuintet, the second-place system, treated the benchmark as a mixture of direct and sequential questions: it first classified whether decomposition was necessary, then performed hop-wise retrieval over Google Search and Wikipedia with anchor passing and Wikipedia-based normalization. UETQuintet’s own system paper reports a best run of 0.840 Exact Match and 0.863 Concept Level Score, ranking second on the leaderboard at the time of writing (Islamaj et al., 12 May 2026, Nguyen et al., 11 Jan 2026).
Category-wise analysis further clarifies what MedHopQA measures. In the shared-task overview, numerical questions were the hardest category across runs, with an average MedCPT F1 of 22.5 and 12 of 48 runs scoring 0%; chromosome questions were the second hardest, with an average of 47.2 and five runs at 0%. Yes/no questions had unusually high variance, with a best run of 98.7, an average of 56.3, and a median of 74.4. Only 30 of the 1,000 questions were answered correctly by at least 90% of official runs, whereas 16 were not solved by any of the 48 submitted runs. The overview therefore portrays MedHopQA not as a benchmark dominated by a few pathological items, but as a broad test of retrieval, decomposition, normalization, and controlled answer expression (Islamaj et al., 12 May 2026).
5. System design patterns and recurrent failure modes
Participant papers show that MedHopQA performance is often constrained as much by answer realization as by latent biomedical knowledge. The clearest example is CaresAI’s system, which fine-tuned LLaMA 3 8B with 4-bit quantization and LoRA adapters on 10,000 filtered biomedical QA pairs drawn from MedQuAD, QALD, MASH-QA, MediQA, Wikipedia Medical QA, BiQA, BioASQ, and TREC. Three fine-tuning regimes were compared: combined short and long answers, short answers only, and long answers only. On validation, the paper reports approximately 0.5 EM and around 0.8 concept-level accuracy across approaches, but official test performance on 1,000 hidden examples was much lower: 0.2 EM and 0.3120 concept-level for combined short+long, 0.0 and 0.1140 for short-only, and 0.0 and 0.2250 for long-only. The authors attribute much of this gap to verbosity, formatting inconsistency, entity normalization mismatch, and over-specific answer variants rather than pure reasoning failure (Abdel-Salam et al., 31 Aug 2025).
CaresAI’s most benchmark-specific intervention was a two-stage inference pipeline for short-answer extraction. In stage 1, the model answered normally; in stage 2, a follow-up prompt asked it to extract the exact answer phrase or entity from its own verbose response. If extraction failed across three sampled attempts, the system fell back to the original longer output. Their error analysis includes canonical MedHopQA failure patterns such as returning “2”, “Chr.2”, “2 chromosome”, or “Chromosome 2p13” when the expected answer was “Chromosome 2”. The same paper reports that unofficial prompt refinement and lightweight post-processing raised EM to 0.49, which suggests that leaderboard compliance in MedHopQA is heavily mediated by output control and canonicalization rather than by factual recall alone (Abdel-Salam et al., 31 Aug 2025).
A second line of work addresses the benchmark from the retrieval-and-trajectory side rather than the answer-format side. DeepRAG combines DeepSeek R1 for hierarchical decomposition with RAG-Gym for process-supervised retrieval optimization and UMLS-informed concept-level rewards. On the MedHopQA dev set, it reports 62.4 EM and 71.8 Concept Accuracy, compared with 54.3 and 66.5 for standalone DeepSeek and 57.7 and 68.3 for vanilla RAG-Gym. Its ablations show that removing hierarchical reasoning caused the largest EM drop, removing process supervision reduced both EM and Concept Accuracy, and removing concept-level rewards produced the largest degradation on semantic accuracy. Although the paper is preliminary and under-specifies several formal details, it reinforces the view that MedHopQA rewards explicit decomposition, stepwise retrieval control, and ontology-aware supervision (Ji et al., 31 May 2025).
Across these system papers, three recurring bottlenecks are especially visible. First, retrieval and decomposition matter because MedHopQA requires linking facts across pages rather than recovering a single span. Second, answer normalization matters because short free-text responses are easy to express correctly in meaning but incorrectly in surface form. Third, evaluation-aware post-processing can materially alter leaderboard outcomes. Taken together, these findings suggest that MedHopQA probes a compound capability: multi-hop biomedical reasoning under stringent answer canonicalization constraints (Abdel-Salam et al., 31 Aug 2025, Ji et al., 31 May 2025).
6. Position within biomedical QA and open research directions
MedHopQA occupies a specific position in the biomedical QA landscape. Relative to MCQ-heavy resources such as MedQA, MedMCQA, PubMedQA, and medical subsets of MMLU, it removes answer-choice cueing and requires short free-text generation. Relative to the earlier QAngaroo MedHop benchmark, it replaces multiple-choice biomedical reasoning with open-ended answer generation. Relative to BioHopR, it is document-centered rather than KG-centered. The benchmark paper’s own zero-shot study with GPT-4o, GPT-5.1, Claude Sonnet 4.5, and Gemini 2.5 Pro shows that even frontier models remain well below saturation, with Lexical Match accuracies from 59.1% to 76.0% and Concept-level accuracies from 66.3% to 83.4%; 77 of the 1,000 questions were missed by all four models. MedHopQA was designed precisely to retain this discriminative headroom (Islamaj et al., 12 May 2026).
Its limitations are equally explicit. The benchmark is derived exclusively from English Wikipedia, remains disease-centered, evaluates answer correctness rather than reasoning traces, and was constructed by annotators from a single institution. It does not release supporting-fact annotations in the style of some general-domain multi-hop datasets, and several shared-task analyses show that normalization remains difficult for chromosome locations, mutations, and numerical answers. Some participant papers also reveal internal reporting inconsistencies around development-set size and validation setup, which indicates that training conditions around the shared task were substantially more weakly supervised than the hidden test framework might suggest. This suggests that MedHopQA is best understood as a challenging evaluation benchmark rather than a fully specified supervised training corpus (Islamaj et al., 12 May 2026, Abdel-Salam et al., 31 Aug 2025).
Subsequent benchmark work has treated MedHopQA as a reference point rather than an endpoint. BioMedHop explicitly positions itself as combining the KG-grounded complexity of BioHopR and the cross-document, open-ended strengths of MedHopQA while extending biomedical multi-hop QA to source-conditioned reasoning over KGs, PubMed-centered documents, web evidence, and hybrid settings. Its framing implies that MedHopQA established document-centered biomedical multi-hop reasoning as a meaningful evaluation target, but left cross-source evidence topology largely unexplored (Tan et al., 15 Jun 2026).
Other related directions push beyond MedHopQA along orthogonal axes. DocHop-QA extends biomedical multi-hop QA toward multimodal, multi-document reasoning over full PubMed Central papers, tables, and layout cues rather than Wikipedia article pairs, thereby stressing long scientific documents and structural evidence synthesis (Park et al., 20 Aug 2025). ShatterMed-QA approaches the same problem space from a topology-regularized multiple-choice setting designed to suppress shortcut learning through generic hub nodes in medical knowledge graphs; although it does not explicitly compare itself to MedHopQA, it is aimed at the same broader question of whether medical QA systems perform authentic multi-hop reasoning or exploit structural shortcuts (Zi et al., 12 Mar 2026).
MedHopQA’s enduring significance lies in the particular package it formalized: open-ended biomedical QA, two-document evidence composition, ontology-grounded answer normalization, contamination-aware hidden evaluation, and shared-task scale. It established that in biomedical QA, reasoning quality, retrieval quality, and answer realization quality are separable variables; high conceptual understanding does not guarantee high leaderboard accuracy, and exact-looking scores do not necessarily reflect raw string equality. In that sense, MedHopQA functions both as a benchmark and as a methodological statement about how LLM-era biomedical reasoning should be stress-tested (Islamaj et al., 12 May 2026, Islamaj et al., 12 May 2026).