EfficientRAG: Iterative Multi-hop Retrieval

• EfficientRAG is a retrieval method for multi-hop QA that uses a Labeler and Filter to iteratively refine evidence without multiple LLM calls.
• It employs token-level relevance labeling and extractive query synthesis to control the retrieval loop and reduce irrelevant evidence.
• Empirical results show improved efficiency with fewer retrieved chunks and competitive answer accuracy on benchmarks like HotpotQA and 2Wiki-MultihopQA.

EfficientRAG is a retrieval method for multi-hop question answering that augments retrieval-augmented generation by iteratively generating follow-up queries and filtering irrelevant evidence without invoking a LLM at every retrieval step. Introduced for open-domain multi-hop QA, it replaces repeated LLM-based query rewriting with lightweight learned components, so that iterative retrieval remains available while the final answer is still generated with a single downstream LLM call (Zhuang et al., 2024). In later work, the same pipeline was adapted inside a broader RAG evaluation setting for HotpotQA, where its token-labeling and iterative refinement mechanisms were combined with hybrid retrieval strategies (Zhang et al., 26 Sep 2025).

1. Problem setting and design objective

EfficientRAG was proposed to address a specific weakness of standard RAG in multi-hop question answering. Single-round retrieval is often adequate for one-hop questions, but multi-hop QA typically requires gathering evidence across multiple passages and reasoning over intermediate entities or relations. Prior iterative retrieval methods improved this setting by generating new queries at each hop, yet they often depended on multiple LLM calls for query reformulation, which increased latency, monetary cost, and prompt complexity (Zhuang et al., 2024).

The core design premise is that the diversity of relation types involved in multi-hop questions is relatively small, and that identifying the next useful retrieval target does not necessarily require a frontier-scale generator. EfficientRAG therefore shifts most of the iterative control logic from LLM prompting to two smaller learned modules: a Labeler that marks useful information in retrieved chunks, and a Filter that constructs the next-hop query from the current question and accumulated labeled tokens (Zhuang et al., 2024).

This design places EfficientRAG between one-shot RAG and LLM-centric agentic retrieval. It preserves iterative evidence gathering, but confines expensive LLM inference to the final answer generation stage. A plausible implication is that EfficientRAG should be most attractive in settings where multi-hop structure is common and inference budgets are constrained.

2. Core architecture and retrieval loop

EfficientRAG extends the conventional retriever-generator pipeline with an explicit iterative retrieval controller. The process begins with a user query, which is passed to a dense retriever to obtain candidate knowledge chunks. Those chunks are then analyzed by the Labeler, which performs token-level relevance labeling and chunk-level continuation decisions. Labeled tokens judged useful for continued reasoning are passed to the Filter, which synthesizes the next query. The system repeats retrieval until the branches terminate or a maximum iteration limit is reached, after which the consolidated relevant chunks are supplied to the final LLM question-answering module (Zhuang et al., 2024).

The retrieval loop has two key decision primitives. First, the Labeler assigns a chunk-level tag of either <Continue> or <Terminate>. Chunks marked <Continue> are retained as potentially useful evidence and contribute labeled tokens to the next-hop query synthesis. Chunks marked <Terminate> are treated as irrelevant or already sufficient for the current branch. Second, the Filter uses the original question together with all currently labeled tokens to form a new query representing the information still needed for the next hop (Zhuang et al., 2024).

This organization differs from standard iterative RAG baselines in two ways. Query generation is not performed by an LLM at each step, and information filtering is performed before final answer generation rather than delegated to the answering LLM. The method therefore seeks to reduce both irrelevant retrieval accumulation and repeated high-cost model invocations.

3. Model components and supervision

Both the Labeler and the Filter are implemented as fine-tuned DeBERTa-v3-large models with 24 layers and 304M parameters (Zhuang et al., 2024). The Labeler consumes a [Query] + [Chunk] sequence and produces two outputs: a binary relevance decision for each token and a chunk-level <Continue>/<Terminate> tag. The Filter consumes [Query] + [All labeled tokens collected so far] and outputs the next-hop query, extractively composed from its inputs (Zhuang et al., 2024).

Because standard multi-hop QA corpora do not provide token-level supervision for iterative retrieval control, EfficientRAG uses synthetic supervision generated with Llama-3-70B. The supervision process decomposes multi-hop questions into single-hop sub-questions, labels supporting chunks at the word level for entities and relations, generates next-hop queries conditioned on current knowledge and dependencies, and includes hard negative samples for robust termination tagging (Zhuang et al., 2024).

The reported training set sizes were:

Dataset	Labeler instances	Filter instances
HotpotQA	357k	73k
MuSiQue	93k	25k
2Wiki-MultihopQA	70k	13k

These supervision choices make EfficientRAG dependent on LLM-generated annotations during training, but not during deployment. This separation is central to its efficiency claim: the expensive model is used offline to create training signals, while inference-time retrieval control is handled by smaller specialized models.

4. Retrieval behavior and empirical results

EfficientRAG was evaluated on HotpotQA, 2Wiki-MultihopQA, and MuSiQue (Zhuang et al., 2024). The reported results distinguish retrieval quality, answer accuracy, and runtime efficiency.

For retrieval, EfficientRAG achieved high recall with substantially fewer retrieved chunks than several baselines. On HotpotQA it reported recall 81.84 with 6.41 retrieved chunks; on 2Wiki-MultihopQA, recall 84.08 with 3.69 retrieved chunks; on MuSiQue, recall 49.51 with 6.1 retrieved chunks (Zhuang et al., 2024). The comparison given in the paper showed that Direct-R@30 retrieved 30 chunks for lower or comparable recall on HotpotQA and 2Wiki-MultihopQA, while Iter-RetGen iter3 retrieved 16–17 chunks for its reported recall values (Zhuang et al., 2024).

For end-to-end QA accuracy, EfficientRAG was reported as follows:

Method	HotpotQA Acc	MuSiQue Acc	2WikiMQA Acc
Direct (no retr.)	25.79	5.51	28.67
Direct-R@10	44.56	17.12	32.70
Iter-RetGen iter3	57.56	25.31	46.59
EfficientRAG	57.86	20.00	53.41

These figures show two distinct patterns. On HotpotQA and 2Wiki-MultihopQA, EfficientRAG matched or exceeded the stronger iterative baseline in the reported accuracy values. On MuSiQue, it remained competitive but below the top LLM-based iterative method, which the summary attributes to lower recall from the smaller retrieval pool (Zhuang et al., 2024).

Efficiency results were also explicit. EfficientRAG required one LLM call per query, whereas Iter-RetGen iter3 required three and SelfAsk required 7.18 on average (Zhuang et al., 2024). For 200 MuSiQue samples, the reported latency values were 2.16 seconds for Direct, 2.47 seconds for Direct-R, 9.68 seconds for Iter-RetGen iter3, 27.47 seconds for SelfAsk, and 3.62 seconds for EfficientRAG (Zhuang et al., 2024). These numbers place EfficientRAG close to direct retrieval in runtime while preserving iterative retrieval behavior.

5. Comparison with related RAG paradigms

EfficientRAG occupies a specific point in the design space of efficient retrieval-augmented generation. Relative to single-hop RAG, it adds iterative retrieval and explicit evidence filtering. Relative to LLM-driven iterative retrieval systems, it removes repeated LLM query generation from the loop and instead uses small learned control modules (Zhuang et al., 2024).

A concise comparison, using terminology reported in the source summary, is as follows:

Aspect	Single-hop RAG	Iterative RAG (LLM-based)	EfficientRAG
Query generation	Original question only	LLM-generated next-hop queries	Filter-generated next-hop queries
Retrieval	One-shot	Iterative	Iterative
Chunk filtering	None	LLM or heuristics	Labeler at token and chunk level
LLM calls per query	1	At least number of reasoning steps	1

Later work adapted the EfficientRAG pipeline in a broader HotpotQA evaluation of retrieval strategies. That study described EfficientRAG as using a Labeler and Tagger to identify salient tokens after initial retrieval, followed by a Filter that constructs next-hop queries without invoking a full LLM at every step. In that setting, the pipeline was combined with hybrid retrieval based on dense embeddings, keyword matching, and maximal marginal relevance reranking (Zhang et al., 26 Sep 2025). The reported HotpotQA results for that adaptation were EM 0.200 and F1 0.276 for the hybrid retrieval configuration with EfficientRAG, versus EM 0.133 and F1 0.188 for cosine similarity and EM 0.150 and F1 0.220 for MMR alone (Zhang et al., 26 Sep 2025).

This later reuse indicates that EfficientRAG is not merely a standalone retriever, but also a reusable retrieval-control pattern. This suggests that its token-labeling and iterative reformulation mechanisms can be composed with alternative first-stage retrievers and rerankers.

6. Limitations, transferability, and significance

EfficientRAG’s principal limitation in the reported experiments is that its retrieval savings can come at the cost of reduced recall on more adversarial multi-hop settings. The MuSiQue results are the clearest example: although EfficientRAG remained competitive, it underperformed the strongest LLM-based iterative baseline in answer accuracy, and the source explicitly links this to a smaller retrieval pool (Zhuang et al., 2024). More generally, the method depends on the quality of synthetic supervision for token labeling, termination decisions, and next-hop query generation.

At the same time, the paper reports that cross-dataset transfer did not degrade performance substantially and sometimes improved it, which was attributed to overlap in relation types across datasets (Zhuang et al., 2024). That observation is consistent with the method’s original motivation that multi-hop retrieval control may rely on a comparatively compact set of reusable relation patterns.

The broader significance of EfficientRAG lies in its decomposition of multi-hop retrieval into smaller, trainable control tasks. Instead of treating iterative retrieval as an LLM-orchestration problem, it frames it as token labeling, branch termination, and extractive query synthesis. Within the reported experiments, this decomposition yielded competitive or state-of-the-art answer quality on two of three benchmarks while materially reducing retrieved chunk counts, LLM calls, and latency (Zhuang et al., 2024). In subsequent work, the same design was used as a retrieval optimization component inside a more general multi-hop QA evaluation framework, further indicating its influence on efficient RAG design for compositional question answering (Zhang et al., 26 Sep 2025).