---
title: Multi-hop QA Benchmarks
url: https://www.emergentmind.com/topics/multi-hop-question-answering-qa-benchmarks
type: topic
---

# Multi-hop QA Benchmarks

Multi-hop question answering (QA) benchmarks are datasets constructed to rigorously evaluate the ability of systems—principally large language models (LLMs)—to perform reasoning over multiple intermediate inference steps, integrating information from disparate sources or modalities, to derive correct answers. Unlike single-hop QA, where the answer is often found via a direct retrieval from a single passage or fact, multi-hop QA explicitly requires sequential evidence retrieval, sub-question answering, and synthesis (in the sense: each answer depends on the output of one or more prior intermediate reasoning steps). The field encompasses a rapidly evolving array of benchmark designs that probe different aspects of reasoning complexity, knowledge novelty, multi-modality, and the robustness of both retrieval and generation components.

## 1. Benchmark Taxonomy and Construction Methodologies

Benchmark design in multi-hop QA has evolved from simple linked-passage settings (HotpotQA, 2WikiMultiHopQA) to sophisticated constructions that enforce connected reasoning, control for memorization, and target previously underexplored axes such as knowledge novelty, semantic diversity, and multi-modal integration.

A foundational distinction concerns the construction methodology:

- **Top-down, annotation-centric approaches**: Classic benchmarks such as HotpotQA rely on crowdworkers to author questions over pairs of Wikipedia paragraphs, with answer and supporting fact annotations but without strong controls against reasoning shortcuts or data contamination [2108.00573].
  
- **Bottom-up, decomposition-driven pipelines**: For example, MuSiQue composes multi-hop questions by systematically chaining single-hop questions while imposing strong connectedness filters: each subsequent hop (sub-question) is only answerable if the previous-hop answer is known, implemented via masking and answerability tests with strong reading comprehension models. This bottom-up methodology yields DAG-structured reasoning chains and enables the creation of contrastive unanswerable pairs [2108.00573].

- **Knowledge-edited and counterfactual datasets**: CofCA (IRE) knowledge-edits Wikipedia passages—substituting all named entities, dates, and facts with novel surrogates—to prevent memorization from pre-training and enables joint evaluation over factual and counterfactual contexts. Each question is strictly decomposed into sub-questions with annotated intermediate answers [2402.11924].

- **Hybrid and cross-modal pipelines**: HybridQA requires reasoning over both tables and text, generating questions that necessitate interleaved operations on tabular data and unstructured passages. MuMuQA builds two-hop chains that traverse visual entity grounding (image-caption alignment) and textual span extraction from real-world news [2004.07347, 2112.10728].

- **Large-scale, automatic, and difficulty-controlled frameworks**: Advanced pipelines leverage LLM-guided evidence tree generation (MHTS), multi-hop reasoning over long contexts (NovelHopQA), or explicit control over knowledge popularity/novelty (MINTQA) to construct highly stratified corpora with fine-grained control of semantic diversity and reasoning depth [2412.17032, 2504.08756, 2506.02000].

A summary table of characteristic construction axes:

| Benchmark Type                      | Construction Pipeline       | Reasoning Control    |
|------------------------------------- |--------------------------- |--------------------- |
| HotpotQA, 2WikiMultiHopQA           | Manual question authoring  | Weak (some overlap)  |
| MuSiQue                             | Single-hop recombination   | Strong via masking   |
| CofCA (IRE)                         | Counterfactual editing     | Joint chain + non-leak |
| HybridQA                            | Table+text, pipeline       | Heterogeneity        |
| MuMuQA, DocHop-QA                    | Cross-modal, LLM-guided   | Visual/textual       |
| MHTS, MINTQA, NovelHopQA             | LLM auto-gen, stratified  | Difficulty, hops, knowledge axis |

## 2. Reasoning Chain Decomposition and Sub-question Annotation

A core organizing principle for modern benchmarks is the explicit decomposition of the target question into a sequence of sub-questions and annotated intermediate answers. This enables:

- **Measurement of chain integrity**: Evaluation extends beyond the final (“hop N”) answer to performance at each intermediate step, penalizing models that “bypass” the true reasoning chain (e.g., produce correct answers via spurious context matching rather than correct stepwise synthesis) [2402.11924].
  
- **Isolation of error propagation**: Performance drop-off from sub-QA1 to sub-QA(N), and error compounding across hops, is directly observable.
  
- **Training signal for decomposition-aware models**: Multi-hop QA models (e.g., MUPPET, HGN, PRISM) can leverage sub-question annotation for more precise retrieval and answer generation at each step [1906.06606, 1911.03631, 2510.14278].

Both MuSiQue and CofCA (IRE) annotate each N-hop question as a chain of N sub-questions plus final QA, with all intermediate answers, formally represented as
$$
Q\,=\, [q_{\mathrm{sub}_1},\,q_{\mathrm{sub}_2},\,\ldots, q_{\mathrm{sub}_N},\,q_{\mathrm{final}}],\quad A\,=\, [a_{\mathrm{sub}_1},\,a_{\mathrm{sub}_2},\,\ldots, a_{\mathrm{sub}_N},\,a_{\mathrm{final}}]
$$

Intensive validation (e.g., MuSiQue’s condition: every sub-question is unanswerable without its direct predecessor’s answer) ensures there are no reasoning shortcuts.

## 3. Evaluation Metrics and Experimental Protocols

Multi-hop QA evaluation encompasses a spectrum of metrics aimed at capturing answer correctness, evidence recall, and the integrity of the reasoning chain:

- **Answer metrics**: F1 (token overlap) and EM (Exact Match) scores on the final answer [2108.00573, 2402.11924].

- **Support/evidence metrics**: F1/EM on supporting sentences or paragraphs, especially in distractor-rich settings (HotpotQA, MuSiQue, PRISM) [1911.03631, 2510.14278].

- **Chain/Joint metrics**: Chain-level evaluation aggregates per-step precision/recall via products; joint metrics integrate all sub-QA and final QA results into a single score. E.g., in CofCA:
$$
P^{(\mathrm{joint})}=P^{(\mathrm{MHQA})}\times\prod_{i=1}^N P^{(\mathrm{sub}_{qa}^i)},\;\;
R^{(\mathrm{joint})}=R^{(\mathrm{MHQA})}\times\prod_{i=1}^N R^{(\mathrm{sub}_{qa}^i)}
$$
with joint F1
$$
\mathrm{Joint\,F1\,RC}=-\log\left(\frac{2\cdot P^{(\mathrm{joint})}R^{(\mathrm{joint})}}{P^{(\mathrm{joint})}+R^{(\mathrm{joint})}}\right)
$$
[2402.11924].

- **Difficulty and stratification metrics**: MHTS introduces a fine-grained query difficulty score:
$$
D = h - \lambda\,s
$$
where $h$ is the number of hops and $s$ is average semantic similarity (cosine) between the question and supporting chunks. This score correlates strongly with system failure rates (Pearson $r=0.99$ with win-rate) [2504.08756].

- **Semantic/structural diagnostics**: Metrics track not just accuracy but semantic diversity, supporting evidence diversity, and retrieval-vs-generation isolation.

- **RAG (retrieval-augmented generation) protocols**: NovelHopQA and PRISM explicitly test models in both full-context and RAG settings, stratifying results by accuracy across context window sizes, number of hops, and evidence recall [2506.02000, 2510.14278].

## 4. Key Datasets and Their Design Axes

Selected exemplars highlight the evolution of the field:

| Benchmark    | Modality  | Hops  | Reasoning Structure         | Novelty Control         | Type of Evidence                   | Notable Features                             |
|--------------|-----------|-------|----------------------------|------------------------|-------------------------------------|----------------------------------------------|
| HotpotQA     | Text      | 2     | Paragraph chains           | None                   | Wikipedia, supporting sentences     | Large, crowd-authored, distractors           |
| 2WikiMultiHop| Text      | 2–4   | Entity bridge/compositional| None                   | Wikipedia                           | Linked entity pairs, multi-hop               |
| MuSiQue      | Text      | 2–4   | DAG sub-questions, filtered| Hard anti-shortcut      | Wikipedia, per-hop contexts         | Rigorous connectedness, contrast pairs       |
| CofCA (IRE)  | Text      | 2–4   | Stepwise, counterfactual   | Counterfactual chains  | Wikipedia (rewritten, unseen)       | Explicit bypass/chain metrics, sub-QAs       |
| HybridQA     | Table+Text| 2–3   | Table/text hybrid hops     | None                   | Wikipedia tables/passages           | First large hybrid, detailed OP pipeline     |
| MuMuQA       | Text+Image| 2     | Visual-text hop chains     | None                   | News images/captions + body text    | Grounding + extraction, synthetic QA pipeline|
| DocHop-QA    | Multi-mod | 2–4   | Fan/chain-hop; multimodal  | None                   | PubMed: text, tables, layout        | Scientific PDFs, cross-doc, multimodal tasks |
| MINTQA       | Text/KG   | 1–4   | Fact chains (novel/tail)   | Popularity/recency      | Wikidata, KG-linearized             | Fine-grained knowledge novelty axis          |
| NovelHopQA   | Narrative | 1–4   | Paragraph chain (keyword)  | Long context           | Full-length novels (64k–128k tokens)| Hop/length stratified, open-source pipeline  |
| MHTS         | Text      | 1–6   | Tree-structured claim QA   | Difficulty (hops, sem)  | Gutenberg, diverse clustering       | Explicit D score, fine-grained strat.        |

## 5. Empirical Findings, Modeling Insights, and Limitations

Key experimental trends and failure analyses from recent benchmarks:

- **Shortcuts and data leakage**: Factual Wikipedia-based benchmarks tend to over-estimate “reasoning” due to model memorization or shallow shortcut exploitation. Counterfactual or knowledge-edited settings (IRE/CofCA, MINTQA) restore a 15–25 point gap in F1/EM, indicating a substantial memorization bias in prior results [2402.11924, 2412.17032].

- **Sub-question prompt inclusion**: Providing explicit sub-questions in the prompt (decomposition-aware prompting) leads to consistent gains in both final QA and joint chain performance [2402.11924].

- **Difficulty stratification**: As hop count and semantic dispersion increase, accuracy consistently decreases (e.g., average accuracy falls by 12–14 points from 1-hop→4-hop at fixed context; in long narratives, RAG accuracy falls 25–35 points below full-golden-context) [2506.02000, 2504.08756].

- **Chain bypass and error analysis**: A significant portion of answers is achieved via bypass chains or incomplete reasoning, with joint chain correctness typically below 40% for 2-hop and dropping steeply at higher depths [2402.11924].

- **Limitations and errors**: Dominant error types include missing final-hop evidence integration, entity confusion, and cumulative evidence drift, as well as retriever failures in multi-hop, long-context, or novel-fact settings [2506.02000, 2412.17032].

- **Robustness to question type and decomposition**: Performance and optimal modeling approaches are highly sensitive to question type (inference, comparison, temporal, null/open) [2505.11811].

## 6. Specialized Benchmark Axes: Retrieval, Modality, and Agentic Protocols

Modern benchmarks systematically probe:

- **Retrieval–Generation Disentanglement**: PRISM, MHTS, and MINTQA stratify evaluation to diagnose failure sources—retriever (missing/support-insufficient evidence) vs. reader/generator (span extraction/generation errors) [2510.14278, 2504.08756, 2412.17032].

- **Difficulty and semantic diversity**: MHTS introduces a continuous difficulty score tied to both the number of hops and average semantic proximity, empirically correlating with win-rate and system breakdowns [2504.08756].

- **Agentic, planner-based reasoning**: BELLE dynamically allocates reasoning “operators” (sub-step, single/iterative-step retrieval, CoT) via bi-level agentic debate, exploiting question-type sensitivity for cost-effective gains (up to +7 F1 over best fixed baselines) [2505.11811].

- **Multi-modality**: HybridQA and DocHop-QA operationalize QA that can only be solved by integrating both structured/tabular and unstructured text (HybridQA) or by jointly grounding in textual, tabular, and visual/PDF layout information (DocHop-QA). MuMuQA probes image–text grounding in news, revealing a strong bottleneck in current model cross-modal co-reference [2004.07347, 2508.15851, 2112.10728].

## 7. Implications, Recommendations, and Directions

Multi-hop QA benchmarks have catalyzed increasingly nuanced evaluations of system reasoning, moving the field from coarse answer accuracy on single-passage queries to stepwise, evidence-grounded, and knowledge-novel reasoning diagnostics. Clear recommendations—grounded in recent benchmarks—are:

- **Publish per-hop sub-question chains and enforce stepwise answer annotation** [2402.11924, 2108.00573].
- **Adopt counterfactual editing or knowledge-graph composition to eliminate pretraining data shortcuts** [2402.11924, 2412.17032].
- **Use joint chain evaluation metrics to penalize incomplete or bypassed reasoning** [2402.11924].
- **Tune retrieval–reasoning pipeline and agentic orchestration to question type and difficulty** [2510.14278, 2505.11811].
- **Explicitly stratify by context length, knowledge novelty, modality, and semantic diversity for diagnostic evaluation** [2504.08756, 2506.02000].

Recent advances suggest that, despite dramatic LLM progress, robust and general multi-hop reasoning—especially in the presence of novel, long-tail, deeply multi-modal, or long-context evidence—remains an open challenge, and that continual benchmark innovation is critical for meaningful progress in this domain.

Source: https://www.emergentmind.com/topics/multi-hop-question-answering-qa-benchmarks