MultiHop-RAG Benchmark for Multi-Hop Queries

• MultiHop-RAG is a benchmark that evaluates retrieval-augmented generation on multi-hop queries, requiring synthesis of evidence from multiple document chunks.
• The benchmark employs a five-step query synthesis pipeline using GPT-4 to rewrite claims and generate gold evidence for robust evaluation.
• Evaluation shows that even top reader models like GPT-4 suffer performance drops with imperfect multi-hop retrieval, highlighting critical retrieval gaps.

Searching arXiv for the main paper and closely related multihop RAG work. arXiv search query: "MultiHop-RAG Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries" and related multihop RAG frameworks. MultiHop-RAG is a benchmark for retrieval-augmented generation under multi-hop query conditions, introduced to evaluate systems that must retrieve and reason over multiple pieces of supporting evidence rather than answer from a single retrieved chunk. It consists of a knowledge base, a large collection of multi-hop queries, their ground-truth answers, and the associated supporting evidence, and it was built from English-language news articles harvested as the underlying RAG knowledge base. The benchmark was designed around the observation that existing RAG systems were inadequate for multi-hop queries and that, at the time of publication, no existing RAG benchmarking dataset focused on multi-hop queries (Tang et al., 2024).

1. Scope and formal task definition

In the benchmark formulation, the document corpus is written as $\mathcal{D}=\{d_1,\dots,d_N\}$, with chunked evidence space $D=\cup_i \text{chunks}(d_i)$. A multi-hop query $q$ is associated with a ground-truth evidence set $E(q)=\{r_1,\dots,r_m\}\subseteq D$, where $m=2\dots 4$, and an answer function $a(\cdot)$ that maps queries to a short string such as yes, no, an entity, a temporal word, or “insufficient information.” The benchmark explicitly includes null cases: $\forall q\in Q,\ E(q)\subseteq D,\ |E(q)|\ge 2$ or $=0$ for null queries, with the idealized condition $LLM^*(q,E(q))\Rightarrow a(q)$ (Tang et al., 2024).

The defining property of the query set is that answers cannot be found in any single chunk but require linking information across two to four documents. This places the benchmark at the intersection of retrieval quality and compositional reasoning quality. A common simplification in earlier RAG evaluation was to treat retrieval and answer synthesis largely independently; MultiHop-RAG instead operationalizes their coupling through explicitly annotated evidence sets and short-answer supervision (Tang et al., 2024).

2. Corpus design and query synthesis pipeline

The underlying knowledge base was built from English-language news articles harvested via the Mediastack API over a three-month window from 2023-09-26 to 2023-12-26, “ensuring minimal overlap with LLM pre-training data.” The corpus contains 609 distinct articles with average length 2,046 tokens. Category counts are technology 172, entertainment 114, sports 211, science 21, business 81, and health 10. Each article was split into 256-token chunks and stored in a vector index (Tang et al., 2024).

Query construction proceeds through a five-step pipeline. First, a fact/opinion classifier extracts sentence-level evidence from each article, and only articles whose extracted sentences share keywords with other articles are retained. Second, GPT-4 rewrites each evidence sentence as a self-contained claim and identifies a bridge-entity and a bridge-topic. Third, claims sharing the same entity or topic are grouped into sets of size 2–4, and GPT-4 generates one multi-hop query for each group. Fourth, GPT-4 outputs the gold answer and the minimal evidence set whose union logically entails that answer. Fifth, quality assurance combines manual spot-checks with a second GPT-4 pass verifying that each query uses all evidence, is answerable only from that evidence, and matches its assigned type (Tang et al., 2024).

This construction strategy makes bridge structure central. Because claims are grouped by shared entity or topic, the benchmark does not merely test whether a retriever can find semantically similar passages; it tests whether the system can assemble a cross-document support chain from bridge-bearing evidence units. That design choice became important in later multihop RAG work that emphasized decomposition, explicit path building, or structured memories.

3. Query taxonomy and evaluation protocol

The benchmark contains $|Q|=2{,}556$ queries. The four query templates are inference, comparison, temporal, and null. Their distribution is as follows (Tang et al., 2024).

Query type	Count	Share
Inference	816	31.9%
Comparison	856	33.5%
Temporal	583	22.8%
Null	301	11.8%

Retrieval is evaluated on non-null queries using standard ranking metrics. For a model $D=\cup_i \text{chunks}(d_i)$0 and query set $D=\cup_i \text{chunks}(d_i)$1, the paper defines:

• $D=\cup_i \text{chunks}(d_i)$2
• $D=\cup_i \text{chunks}(d_i)$3
• $D=\cup_i \text{chunks}(d_i)$4 in standard form over relevant retrieved chunks

Generation is evaluated by answer accuracy, with an additional focus on null-query robustness, namely correct production of “insufficient information” when the evidence set is empty. The protocol therefore separates three linked questions: whether supporting evidence is retrieved at all, whether enough of the support chain is retrieved, and whether the reader model can synthesize the final answer from the provided context (Tang et al., 2024).

4. Retrieval benchmark results

The retrieval study compares multiple embedding models, including OpenAI text-embedding-ada-002, OpenAI text-search-ada-query-001, voyage-02, LLM-embedder, bge-large-en-v1.5, jina-embeddings-v2-base-en, intfloat/e5-base-v2, hkunlp/instructor-large, and a second-stage bge-reranker-large. Queries are embedded and matched to 256-token chunks by cosine similarity, with results reported for $D=\cup_i \text{chunks}(d_i)$5 and $D=\cup_i \text{chunks}(d_i)$6 (Tang et al., 2024).

Without reranking, the best model, bge-large, achieves approximately $D=\cup_i \text{chunks}(d_i)$7, $D=\cup_i \text{chunks}(d_i)$8, and $D=\cup_i \text{chunks}(d_i)$9. With reranking, the strongest reported pipeline is voyage-02 plus bge-reranker, with $q$0, $q$1, $q$2, and $q$3. Even this best pipeline fails to retrieve all $q$4 supporting chunks within the top-4, which the paper identifies as evidence of substantial room for improvement (Tang et al., 2024).

These results establish retrieval as the primary bottleneck in the benchmark’s setting. The issue is not merely first-hit retrieval, since Hit@K and MRR@K can remain moderate even when the full support chain is missing. For MultiHop-RAG, partial retrieval can still be structurally insufficient, because the final answer often depends on two or more bridge-linked chunks.

5. Generation benchmark and the retrieval–reasoning gap

The generation benchmark evaluates GPT-4, GPT-3.5-Turbo, Claude-2.1, Google-PaLM, Llama-2-70B-chat-HF, and Mixtral-8x7B under a common “retrieve-then-answer” prompt, with retrieved chunks at $q$5 and, separately, oracle evaluation using gold evidence (Tang et al., 2024).

Model	Accuracy with retrieved chunks	Accuracy with gold evidence
GPT-4	0.56	0.89
Claude-2.1	0.52	0.56
PaLM	0.47	0.74
GPT-3.5	0.44	0.57
Mixtral-8x7B	0.32	0.36
Llama-2-70B-chat-HF	0.28	0.32

Two benchmark conclusions follow directly. First, strong reader models do not erase retrieval errors: GPT-4 rises from 0.56 with retrieved chunks to 0.89 with gold evidence, so imperfect retrieval leaves a large performance gap. Second, reasoning difficulty remains nontrivial even with oracle support, because the best gold-evidence result still falls short of perfect accuracy. The paper’s qualitative analysis further reports that most models correctly refuse to answer null queries at roughly 90%+, while open-source models struggle on comparison and temporal questions, often flipping yes/no judgments or mis-ordering events (Tang et al., 2024).

A recurrent misconception is that multi-hop RAG is principally a reader-side reasoning problem. MultiHop-RAG weakens that claim. Its measurements indicate that retrieval quality and evidence completeness dominate end-to-end performance, and that answer synthesis quality only becomes visible once the support chain is adequately assembled.

6. Methodological developments in later multi-hop RAG systems

Subsequent work diversified the design space exposed by benchmarks such as MultiHop-RAG. One line of work removed repeated LLM rewriting from iterative retrieval. TreeHop replaces the “Retrieve-Rewrite-Vectorize-Retrieve” cycle with a “Retrieve-Embed-Retrieve” loop, updates query embeddings through an UpdateGate, reduces branching from $q$6 to $q$7 under its pruning rules, and on 2WikiMultiHop reports $q$8 with 0.022 s latency at two hops; the paper states that similar gains hold on MuSiQue and MultiHop-RAG sets (Li et al., 28 Apr 2025).

Another line moved retrieval signals inside the generator. Layer-wise RAG uses intermediate hidden states from middle layers rather than explicit query rewriting, reporting +4–6 EM and +3–5 F1 over vanilla one-hop RAG, +2–3 EM over a naïve multi-step baseline, and a 25% compute saving in its four-hop comparison setting (Lin et al., 2 Mar 2025). A related but distinct strategy transforms both questions and documents: the framework in “Transforming Questions and Documents for Semantically Aligned Retrieval-Augmented Generation” decomposes a multi-hop question into single-hop subquestions and retrieves document chunks through embeddings of generated answerable questions; with GPT-4o as generator, it reports average F1 = 63.89, beating MacRAG at 56.97 by 6.92 points (Lee, 13 Aug 2025).

Efficiency-oriented offline restructuring became another theme. CompactRAG converts the corpus into an atomic QA knowledge base and invokes the LLM only twice during inference, once for decomposition and once for final synthesis, regardless of the number of reasoning hops; it reports about 1.9K tokens per query versus 4.7K to 10.2K for iterative baselines (Yang et al., 5 Feb 2026). In a different compression-based design, “Knowledge Compression via Question Generation” stores 109 paper-cards plus 109 main queries as 218 vectors, reports approximately 80% reduction in vector storage demands, and reaches F1 = 0.52 with LLaMA2-Chat-7B on LongBench 2WikiMultihopQA (Eponon et al., 9 Jun 2025).

Structured retrieval also expanded beyond flat passage ranking. HyperRAG argues that binary relational graphs limit relational expressiveness and instead reasons over n-ary hypergraphs, with HyperRetriever reporting average gains of 2.95% in MRR and 1.23% in Hits@10 over the strongest baseline (Lien et al., 16 Feb 2026). HKVM-RAG treats answer-path hyperedges as retrieval keys and passage texts as values, improving over KG-PPR by +3.426 F1 on 2WikiMultiHopQA and +3.592 F1 on MuSiQue, while also showing that higher structured support coverage does not necessarily yield standalone answer-F1 gains on HotpotQA (Zhang et al., 5 Jun 2026). On the control side, Stop-RAG casts iterative RAG as a finite-horizon Markov decision process and reports 69.5 EM on HotpotQA with only 2.3 retrievals on average, compared with 68.9 EM and 4 retrievals for a fixed four-hop baseline (Park et al., 16 Oct 2025).

Taken together, these systems indicate that later multi-hop RAG research split into several technically distinct responses: embedding-space query evolution, generator-internal retrieval signals, question decomposition and representation alignment, offline knowledge restructuring, structured graph or hypergraph memories, and adaptive stopping policies. This suggests that MultiHop-RAG helped crystallize multi-hop QA as a systems problem rather than a single-model problem.

7. Benchmark legacy, limitations, and extensions

MultiHop-RAG established a text-only, news-based benchmark with explicit support annotations and a strong emphasis on bridge-linked retrieval failure. Its recommendations for future systems included query decomposition, hybrid indexes combining sparse and dense embeddings, learned multi-hop retrievers or graph-based retrieval, chain-of-thought or self-ask prompting, and RAG-agent frameworks (Tang et al., 2024). Much of the subsequent literature can be read as concrete instantiations of those directions.

At the same time, later evaluation work targeted dimensions that MultiHop-RAG did not explicitly control. MHTS introduced a difficulty-controlled synthesis framework with the score $q$9 and reported near-perfect linear correlation between win-rate and benchmark difficulty at $E(q)=\{r_1,\dots,r_m\}\subseteq D$0, arguing that hop count alone is not a sufficient measure of multi-hop complexity (Lee et al., 29 Mar 2025). MiRAGE extended evaluation to multimodal and domain-specific corpora, generating 1,000 QA pairs per domain across finance, regulations, science, and journalism, with average hop counts 2.84, 2.60, 2.55, and approximately 1.2 respectively, while also finding that visual grounding remains challenging (Sahu et al., 21 Jan 2026).

The enduring significance of MultiHop-RAG lies in its explicit separation of retrieval sufficiency from reader competence. Its central empirical pattern remains difficult to evade: even strong LLMs degrade sharply when the evidence chain is incomplete, and improvements in multi-hop RAG must therefore be judged not only by answer accuracy but by the fidelity, completeness, and structure of the retrieved support itself.