Dissecting Agentic RAG: A Component Ablation for Multi-Hop QA with a Local 7B Model

Abstract: Agentic retrieval-augmented generation (RAG) systems combine iterative reasoning loops, query decomposition, and adaptive retrieval to tackle multi-hop question answering. However, the contribution of each component remains poorly understood, particularly under resource-constrained settings using only local LLMs. Many agentic designs add adaptive retrieval routing and deeper retrieval loops on the assumption that the added complexity helps. To test whether it does, we run a controlled ablation study of a full agentic RAG pipeline evaluated on 5,000 questions from the HotpotQA distractor development set using a local 7B parameter model (Qwen2.5-7B-Instruct). Our full pipeline achieves EM=53.2% and F1=61.6%, compared to a single-pass dense-retrieval baseline of EM=43.1% and F1=54.0%. Across eight ablation conditions, we find that: (1) fixed hybrid retrieval via reciprocal rank fusion consistently outperforms rule-based adaptive routing (+1.8 EM, +1.9 F1), as the routing heuristic over-routes to BM25 by firing on named entities present in nearly all multi-hop sub-questions; (2) two retrieval iterations over the decomposed sub-questions capture 95% of the gains of five, with no meaningful benefit from deeper loops; and (3) query decomposition and cross-encoder reranking each contribute statistically significant but smaller gains (p<0.01 and p<0.001 respectively). Taken together, on a fixed local-model budget, the simpler and fixed choices turn out to be competitive with or better than their adaptive versions: most of the gain comes from running a short retrieval loop, not from adaptive routing or from many iterations. We use no proprietary APIs or large-scale compute.

• The paper systematically ablates eight agentic RAG configurations on 5,000 HotpotQA questions, finding that a fixed hybrid retriever achieves the best results at 55.0% EM and 63.5% F1.
• The study shows that two retrieval iterations capture 95% of the five-step gain, while one step causes the largest performance drop and decomposition and reranking provide smaller improvements.
• The results suggest local 7B deployments should prioritize short iterative retrieval with fixed hybrid search over costly adaptive routing, although findings may not generalize beyond HotpotQA or Qwen2.5-7B.

This paper presents a controlled component-level ablation of an agentic retrieval-augmented generation (RAG) pipeline for multi-hop question answering, conducted entirely with a locally served Qwen2.5-7B-Instruct model and no proprietary APIs (2606.21553). The study evaluates eight ablation conditions on a fixed sample of 5,000 HotpotQA distractor development questions, using paired significance tests and bootstrap confidence intervals. Its central finding is that, under a fixed local-model budget, simpler and fixed design choices match or outperform their adaptive counterparts: fixed hybrid retrieval beats rule-based adaptive routing, and two retrieval iterations capture 95% of the gain of five.

Motivation and scope

Agentic RAG systems combine iterative reasoning loops, query decomposition, and adaptive retrieval to address the failure of single-pass RAG on multi-hop QA, where one retrieval step rarely surfaces the full evidence chain (2606.21553). Prior work such as IRCoT, Self-Ask, ReAct, and FLARE demonstrates end-to-end gains, but component contributions are rarely isolated, and existing ablations typically use large proprietary models. The paper's contribution is explicitly empirical rather than architectural: it asks, for a resource-constrained deployment, which added components actually help.

System architecture

The pipeline has four stages: query decomposition, an iterative retrieval loop, adaptive retrieval with cross-encoder reranking, and answer synthesis. A decomposer prompt splits the input question into sub-questions up front, following a plan-and-execute variant of ReAct rather than interleaving fresh reasoning before each retrieval. The loop then processes up to $T=5$ sub-questions; at each step it selects a retrieval strategy, retrieves, reranks with a cross-encoder, and appends evidence to a shared context. A single synthesis call produces the final answer; the LLM is not queried between retrieval steps. This design deliberately isolates iterative evidence accumulation from per-step reasoning.

Retrieval options include dense bi-encoder search (BGE-small-en-v1.5 over Qdrant), sparse BM25, and hybrid reciprocal rank fusion (RRF). A rule-based router sends entity-, date-, or number-containing queries to BM25, short or conceptual queries to dense, and everything else to hybrid. The top-20 candidates are reranked by ms-marco-MiniLM-L-6-v2 down to top-5. All LLM calls run via Ollama on a single RTX A6000.

Experimental setup

Evaluation uses a fixed-seed random sample of 5,000 HotpotQA distractor questions (10 candidate paragraphs each: 2 gold, 8 distractors), scored with the official EM and token-level F1 script. The baseline is single-pass dense retrieval (top-10) with one synthesis step—no decomposition, loop, or reranking. Eight ablation conditions each modify one component: loop depth (1, 2, 3 steps), no decomposition, no reranker, and fixed dense-only, sparse-only, or hybrid-only retrieval replacing the router. Because all conditions run on the same questions, the paper uses paired tests—McNemar's exact test for EM, Wilcoxon signed-rank for F1—with one-sided p-values that survive Holm-Bonferroni correction across the five key comparisons.

Main results

The full agentic pipeline reaches EM 53.2% and F1 61.6% versus 43.1%/54.0% for the single-pass baseline (+10.1 EM, +7.6 F1, $p<0.001$), at a latency of 5,642 ms per question versus 546 ms. Notably, the best variant—hybrid-only, which removes adaptive routing entirely—reaches EM 55.0% and F1 63.5%, outperforming the full adaptive pipeline by +1.8 EM and +1.9 F1 ($p<0.001$). The authors are careful to note that the baseline gap conflates several changes at once, so it measures the value of the complete pipeline, not any single component; the ablations isolate individual effects. They also caution against direct comparison with purpose-built systems such as the Hierarchical Graph Network (EM 69.2, F1 82.2 on the full distractor test set), since those use different architectures and evaluation sets.

Ablation findings

Fixed hybrid retrieval beats adaptive routing. The routing analysis explains the result: 79.2% of questions had their top-ranked evidence retrieved via the BM25 route, versus 16.3% hybrid and 4.5% dense, because the entity-firing heuristic triggers on nearly every multi-hop sub-question. Although per-stratum EM superficially favors BM25 (53.9 vs. 50.9 for hybrid), the authors attribute this to selection bias—the heuristic routes simpler, entity-anchored queries to BM25 and leaves harder queries elsewhere. Applied uniformly, hybrid RRF's complementary dense signal improves even entity-heavy questions, where multi-hop evidence chains involve paraphrase and conceptual overlap that lexical matching misses.

Two retrieval iterations suffice. Capping the loop at one step costs 7.1 EM—the largest single-component drop in the study ($p<0.001$)—confirming that iterative evidence accumulation, not any routing or reranking decision, is the primary driver over the baseline. The two-step variant recovers to within 0.3 EM of the full five-step pipeline, and three steps are statistically indistinguishable from five. Qualitative examples show the second step resolving missing evidence links that the first surfaced only partially. The latency savings from fewer steps are modest, however, since the two LLM calls (decomposition and synthesis) dominate per-question cost.

Decomposition and reranking help, with different cost profiles. Removing decomposition costs 1.4 EM ($p=0.004$); removing reranking costs 1.7 EM ($p<0.001$). Both are smaller than the iteration effect because later loop iterations partially compensate. The cost asymmetry is sharp: decomposition roughly doubles latency (2,546 ms without it vs. 5,642 ms with), while reranking is nearly free. The authors flag an interpretive caveat: because disabling decomposition also collapses the loop to a single retrieval, the no-decomp figure conflates loss of decomposition guidance with loss of multi-step retrieval, so it is not a clean isolation of decomposition alone.

Per-step strategy choice matters little inside the loop. Dense-only and sparse-only both score EM 53.0%, effectively matching the full adaptive pipeline. Repeated retrieval across multiple sub-question formulations compensates for any single strategy's blind spots, making routing decisions largely inconsequential once iteration is in place.

Limitations

The paper states its constraints plainly. Results come from a 5,000-question sample of the development set, not the full set, so absolute numbers may shift. All experiments use a single model family (Qwen2.5-7B-Instruct); generalization to other scales or architectures is untested. HotpotQA's two-hop structure means tasks with longer evidence chains or numerical reasoning may rank components differently. Latency figures are hardware- and serving-stack-specific (RTX A6000, Ollama). Each condition is a single run with greedy decoding at temperature 0, so the confidence intervals capture question-sampling variation but not residual run-to-run variation in generation.

Conclusion

This ablation study provides evidence that, for multi-hop QA with a local 7B model, the dominant source of gain over single-pass RAG is running a short retrieval loop at all: two iterations capture 95% of the benefit of five, fixed hybrid RRF retrieval outperforms a rule-based adaptive router ($p<0.001$), and decomposition and reranking contribute smaller but statistically significant improvements. The best configuration achieves EM 55.0% and F1 63.5% with no proprietary APIs. An open question the paper leaves is whether these component rankings hold for benchmarks requiring longer evidence chains than HotpotQA's two-hop structure, and whether the failure mode of entity-triggered routing generalizes to other adaptive routing heuristics.