---
title: 'MARAG-R1: Multi-Tool RAG Model'
url: https://www.emergentmind.com/topics/marag-r1
type: topic
---

# MARAG-R1: Multi-Tool RAG Model

Searching arXiv for MARAG-R1 and closely related agentic/multi-tool RAG work to ground the article in current literature.
MARAG-R1 is a reinforcement-learned multi-tool retrieval-augmented generation framework designed to overcome the single-retriever, fixed top-\(k\) bottleneck in conventional RAG. It equips a large language model with four retrieval tools—semantic search, keyword search, filtering, and aggregation—and trains the model to decide both how and when to use them through supervised fine-tuning followed by reinforcement learning. The framework is motivated by corpus-level reasoning tasks, where answering requires broader and more precise information access than a static retrieval pass can provide, and it is evaluated on GlobalQA, HotpotQA, and 2WikiMultiHopQA, where it substantially outperforms strong baselines and achieves new state-of-the-art results in corpus-level reasoning tasks [2510.27569].

## 1. Concept and scope

MARAG-R1 is introduced against the background that large language models are limited by static pretraining knowledge, while standard RAG pipelines typically rely on a single retriever with fixed top-\(k\) selection. The paper argues that this single-retriever paradigm restricts the model to a “narrow and static subset of the corpus,” which becomes the primary bottleneck for comprehensive external information acquisition, especially in tasks requiring corpus-level reasoning [2510.27569].

The framework addresses this bottleneck by replacing one-shot retrieval with multi-step, agentic retrieval. Rather than retrieving once and answering from a fixed evidence set, the model alternates between intermediate reasoning, tool invocation, and evidence accumulation. This design targets settings in which the system must gather sufficient evidence progressively, including Count, Sort, MinMax, TopK, and multi-hop question answering [2510.27569].

This placement distinguishes MARAG-R1 from conventional workflow RAG and from graph-based alternatives. The paper argues that graph-based RAG may improve global awareness but can lose document-level identifiers and fine-grained factual details during graph construction, whereas MARAG-R1 preserves direct access to original documents and their identifiers [2510.27569]. A plausible implication is that the framework is intended not only to improve recall, but also to maintain provenance needed for set operations and exact aggregation.

## 2. Retrieval model and tool set

The defining feature of MARAG-R1 is its multi-tool retrieval environment. The model is equipped with four retrieval tools:

| Tool | Paper notation | Role |
|---|---|---|
| Semantic Retriever | \(F_{\mathrm{DR}}\) | Dense semantic retrieval for broad contextual exploration |
| Keyword Retriever | \(F_{\mathrm{KR}}\) | Keyword-based matching for precise entity and attribute retrieval |
| Document Filter | \(F_{\mathrm{DF}}\) | Filtering documents by metadata attributes or logical constraints |
| Aggregation Tool | \(F_{\mathrm{AG}}\) | Statistical or structural synthesis such as counting, ranking, and set aggregation |

The paper presents these tools as complementary. Semantic retrieval is used for broad contextual exploration, keyword retrieval for precise lexical matching, document filtering for reducing candidate sets under explicit constraints, and aggregation for operations such as counting, ranking, and set aggregation [2510.27569]. The aggregation tool is particularly important for corpus-level reasoning tasks, because the answer may depend on a global operation over many documents rather than on a single supporting passage.

The framework’s broader design can be situated alongside recent training-free multi-agent RAG work, which also treats retrieval as a staged reasoning process rather than a one-shot module, although MARAG-R1 differs by learning tool coordination with reinforcement learning [2505.20096]. It also contrasts with multimodal RAG systems that focus on reranking or evidence pruning after initial retrieval rather than expanding the retrieval action space itself [2505.24073].

## 3. Trajectory formulation and inference process

The paper formalizes training and inference around multi-step trajectories. For each query-answer pair \((Q, A^*)\), the model produces a trajectory
\[
\mathcal{T} = \{S_1, S_2, \ldots, S_{|\mathcal{T}|}\},
\]
where each intermediate step is
\[
S_t = (R_t, C_t, D_t), \quad t < |\mathcal{T}|.
\]
Here \(R_t\) is intermediate reasoning, \(C_t\) is a tool invocation, and \(D_t\) is the resulting retrieved document output [2510.27569]. The final step replaces the tool call with the predicted answer \(A\).

Operationally, the model begins from the user query, reasons about what information is missing, selects a tool, observes the retrieved output, and either continues retrieving or answers once sufficient information has been obtained [2510.27569]. The paper does not provide a separate halting classifier or stopping threshold. Instead, the continue-versus-answer behavior is learned implicitly through sequence modeling and reinforced through the trajectory reward.

The framework therefore treats retrieval as a sequential decision problem over reasoning, actions, and observations. The paper does not explicitly formalize the setting as an MDP or POMDP, but the trajectory definition serves that role in practice. This suggests a retrieval-control view of RAG rather than a prompt-construction view.

## 4. Two-stage training procedure

MARAG-R1 is trained in two stages: supervised fine-tuning followed by reinforcement learning [2510.27569].

### 4.1 Supervised fine-tuning

The first stage constructs high-quality expert trajectories using GPT-4 with task instructions and examples, followed by rejection sampling to remove low-quality or redundant traces and ensure consistency and factual correctness. The resulting dataset is
\[
\mathcal{D} = \{(Q_i, \mathcal{T}_i^*, A_i^*)\}_{i=1}^N,
\]
where each example contains a query, an expert trajectory, and a gold answer [2510.27569].

The supervised objective is standard autoregressive next-token prediction over trajectory steps:
\[
\mathcal{L}_{\text{SFT}} = -\sum_{i=1}^{N} \sum_{j=1}^{|\mathcal{T}_i|} \log P_\theta(S_j \mid Q_i, S_{<j}).
\]
This stage teaches the model to imitate expert multi-step retrieval behavior. The paper refers to the SFT-only model as MARAG-CS, for “cold start” [2510.27569].

### 4.2 Reinforcement learning

After imitation learning, the policy is further optimized by reinforcement learning in the retrieval environment. The goal is to improve tool coordination beyond what is present in demonstrations, especially because answer-only supervision is too sparse for effective learning [2510.27569].

The paper uses Reinforcement Learning with Leave-One-Out baseline (RLOO). For a batch of \(K\) sampled trajectories, the gradient estimator is
\[
\begin{aligned}
\nabla_\theta J(\theta) = \frac{1}{K} \sum_{i=1}^{K} \Big( R(\mathcal{T}_i) - \tfrac{1}{K-1} \sum_{j \neq i} R(\mathcal{T}_j) \Big) \nabla_\theta \log P_\theta(\mathcal{T}_i).
\end{aligned}
\]
The paper specifies that RL uses 5 sampled rollouts per query [2510.27569].

A central claim is that SFT initialization is necessary for stable exploration, while the leave-one-out baseline reduces variance without requiring a reference model or KL regularization [2510.27569]. This places MARAG-R1 in a family of R1-style systems that use RL to refine multi-step reasoning behavior after supervised initialization, although its domain is retrieval control rather than pure reasoning [2505.23621][2506.00070].

## 5. Reward design

The RL stage optimizes a trajectory-level reward with three components:
\[
R(\mathcal{T}) = R_A + R_E + R_T.
\]

### 5.1 Answer reward

The answer reward is token-level F1 between the predicted answer and the reference answer:
\[
R_A = \text{F1}(A, A^*).
\]
This provides partial credit instead of a binary signal [2510.27569].

### 5.2 Document coverage reward

The document coverage reward measures how well the retrieved document set matches the gold supporting documents:
\[
R_E = \text{F1}(\mathcal{D}_{\text{pred}}, \mathcal{D}^*),
\]
with precision and recall over document identifiers. The paper gives the corresponding precision and recall definitions and computes an F1 score over predicted versus gold document sets [2510.27569]. This component encourages both completeness and relevance of retrieval.

### 5.3 Tool exploration reward

The tool exploration reward encourages strategic but not excessive tool use. Let \(N_{\text{call}}\) be the number of tool calls in the sampled trajectory and \(N_{\text{call}^*}\) the number in the expert trajectory. The reward is defined piecewise:
\[
R_T = \begin{cases}
1, & \text{if } N_{\text{call}} \leq N_{\text{call}^*}, \\
\max\!\left(0,\, 1 - \frac{N_{\text{call}} - N_{\text{call}^*}}{N_{\text{call}^*}}\right), & \text{otherwise.}
\end{cases}
\]
This gives maximal reward when the model uses no more tool calls than the expert and penalizes overuse thereafter [2510.27569].

The three-part reward is one of the paper’s core design decisions. It makes retrieval quality a first-class training target rather than treating retrieval as latent infrastructure. This aligns MARAG-R1 with other recent systems that train evidence selection or retrieval control directly rather than relying on final-answer supervision alone [2604.24564][2510.27569].

## 6. Experimental setting

The main evaluation is conducted on GlobalQA, a corpus-level reasoning benchmark containing four task types: TopK, Count, Sort, and MinMax [2510.27569]. These tasks are designed to stress comprehensive retrieval and explicit aggregation rather than local passage retrieval.

The paper also evaluates generalization on 2WikiMultiHopQA and HotpotQA [2510.27569]. This choice connects MARAG-R1 to the broader multi-hop QA literature and shows whether a framework optimized for global retrieval transfers to more standard compositional QA tasks.

The experiments use Qwen2.5-Instruct models at 3B, 7B, and 14B scale as backbones. Retrieval uses BGE as the retriever, Qwen3-4B as the filtering model, a chunk-free retrieval strategy, and TopK \(= 20\) [2510.27569]. Evaluation reports answer F1 and document F1@20, denoted D-F1@20, with the latter computed from precision and recall over retrieved versus gold evidence sets [2510.27569].

The baselines include StandardRAG, ITER-RETGEN, IRCoT, HyperGraphRAG, Search-R1, and ReCall [2510.27569]. The ReCall comparison is especially important because it uses the same multi-tool retrieval environment but optimizes only with answer-level rewards, isolating the effect of MARAG-R1’s process-aware training design.

## 7. Empirical results

### 7.1 GlobalQA

On GlobalQA, MARAG-R1 substantially outperforms all reported baselines across model sizes [2510.27569].

For Qwen2.5-14B, MARAG-R1 achieves an average **31.22 F1** and **42.11 D-F1@20**. In the same setting, ReCall achieves **14.25 F1** and **20.00 D-F1@20**, MARAG-CS achieves **28.92 F1** and **39.83 D-F1@20**, and StandardRAG achieves **1.51 F1** and **8.09 D-F1@20** [2510.27569].

For Qwen2.5-7B, MARAG-R1 reaches **28.60 F1** and **38.44 D-F1@20**, compared with ReCall at **10.84 F1** and **17.61 D-F1@20** [2510.27569].

For Qwen2.5-3B, MARAG-R1 reaches **26.40 F1** and **37.05 D-F1@20**, while ReCall reaches **6.41 F1** and **12.68 D-F1@20** [2510.27569].

The largest task-level strengths appear on Sort and MinMax, with Count remaining more difficult. For Qwen2.5-14B, the paper reports: TopK **32.65 / 38.81**, Count **6.05 / 42.38**, Sort **40.5 / 40.92**, and MinMax **39.20 / 45.81** in F1 / D-F1@20 [2510.27569]. This suggests that the framework is particularly effective when the problem requires structured aggregation over a sufficiently retrieved set, though counting remains comparatively challenging.

### 7.2 Generalization to multi-hop QA

On 2WikiMultiHopQA, MARAG-R1 achieves **22.00 EM / 26.93 F1**, compared with StandardRAG at **12.00 / 18.24**, ITER-RETGEN at **12.00 / 17.22**, IRCoT at **5.00 / 18.50**, and MARAG at **15.00 / 22.81** [2510.27569].

On HotpotQA, MARAG-R1 achieves **31.00 EM / 39.16 F1**, compared with StandardRAG at **19.00 / 26.36**, ITER-RETGEN at **17.00 / 25.56**, IRCoT at **11.00 / 23.92**, and MARAG at **22.00 / 34.05** [2510.27569].

These results indicate that the framework’s learned multi-tool retrieval strategy transfers beyond the corpus-level operations of GlobalQA to more conventional multi-hop QA tasks.

## 8. Ablations and interpretation

The ablation studies isolate the contributions of both training and tool design [2510.27569].

### 8.1 Training ablations

On Qwen2.5-14B GlobalQA, full MARAG-R1 achieves **31.22 / 42.11**. Removing the tool-calls reward reduces this to **29.45 / 39.54**. Removing the document reward reduces it to **28.87 / 39.53**. Removing the answer reward causes a larger drop to **13.78 / 24.48**. Removing SFT reduces performance to **14.25 / 20.00** [2510.27569].

This shows that SFT initialization and answer reward are indispensable, while document coverage and tool-calls reward provide consistent further gains. The pattern supports the paper’s interpretation that the three reward components are complementary.

### 8.2 Tool ablation

On Qwen2.5-7B, the full system achieves **27.35 / 38.19**. Removing the document filter yields **22.57 / 32.14**. Removing the semantic retriever yields **20.21 / 25.13**. Removing the keyword retriever yields **13.03 / 24.08**. Removing the aggregation function yields **10.48 / 19.54** [2510.27569].

Aggregation is therefore the most critical tool in the reported ablations, followed by keyword and semantic retrieval. This is consistent with the framework’s intended use for corpus-level reasoning, where retrieving documents is necessary but not sufficient unless the system can explicitly combine them.

### 8.3 Tool-usage analysis

The paper also compares answer quality, evidence coverage, and average number of tool calls. On Qwen2.5-14B, ReCall averages **4.53** calls, MARAG **2.81**, MARAG-CS **6.27**, and MARAG-R1 **6.32** [2510.27569]. The strongest systems make more tool calls, which the paper interprets as evidence that better performance depends on broader external information acquisition. Because MARAG-R1 and MARAG-CS make a similar number of calls but MARAG-R1 performs better, the improvement cannot be attributed to call volume alone; it reflects improved coordination.

### 8.4 Retrieval depth

Appendix analysis shows that performance rises quickly as the maximum retrieval steps increase from 2 to 5, and remains stable beyond 10 steps [2510.27569]. This suggests that multi-step retrieval is essential, but that the RL-trained policy is robust to larger search depth.

## 9. Representative behavior

The paper’s most illustrative case study is an OR-query requiring union and min aggregation over document IDs [2510.27569]. MARAG-R1 issues multiple searches for different subconditions, unions the resulting document sets, performs a keyword search for an additional clause, unions again to obtain **44 unique documents**, and then applies a Min operation to identify the smallest document ID, yielding the correct answer **21** [2510.27569].

This example is important because it demonstrates behavior not reducible to “better retrieval.” The model is not merely finding better passages; it is executing a retrieval-and-aggregation program over document sets. This suggests that MARAG-R1 is best understood as a retrieval-control architecture with explicit evidence operations, not merely an enhanced retriever.

## 10. Relation to adjacent work

MARAG-R1 belongs to a broader trend of moving retrieval from a fixed preprocessing stage into the reasoning loop. Training-free multi-agent RAG systems similarly decompose retrieval and synthesis into multiple coordinated subtasks, but rely on prompting rather than reinforcement learning [2505.20096]. Multimodal RAG work has likewise emphasized that evidence quality depends on reranking, grounding, and pruning, though those systems generally optimize evidence selection after initial retrieval rather than learning a richer retrieval action space [2505.24073][2604.24564].

This suggests that MARAG-R1 occupies a specific niche: it is a text-domain, corpus-level RAG framework that expands the model’s retrieval affordances and trains the policy over those affordances directly. A plausible implication is that its main contribution is architectural and procedural rather than representational.

## 11. Limitations and significance

The paper does not present a separate limitations section, but several constraints are evident from the design [2510.27569]. MARAG-R1 incurs higher inference cost than one-shot RAG because it averages around 6 tool calls in strong settings. It depends on the quality of its semantic retriever, keyword retriever, filtering model, and aggregation implementation. It also requires GPT-4-generated expert trajectories, rejection sampling, and reinforcement learning in a retrieval environment, making training substantially more complex than conventional RAG.

Even so, the reported evidence supports the paper’s central claim: the single-retriever top-\(k\) paradigm is a major bottleneck for corpus-level reasoning, and MARAG-R1 overcomes that bottleneck by learning to coordinate multiple retrieval tools dynamically [2510.27569]. Its strongest empirical value lies in tasks where answers depend on comprehensive evidence acquisition and explicit aggregation rather than on retrieving one relevant passage.

In that sense, MARAG-R1 can be understood as a transition point from retrieval-augmented generation to retrieval-governed reasoning: the model is not only grounded by retrieved evidence, but also trained to decide how evidence should be acquired, filtered, and combined to support final synthesis [2510.27569].

Source: https://www.emergentmind.com/topics/marag-r1