--- title: 'R-Search: Agentic Search-Enhanced Reasoning' url: https://www.emergentmind.com/topics/r-search type: topic --- # R-Search: Agentic Search-Enhanced Reasoning In contemporary large language model research, **R-Search** denotes a class of reasoning–search frameworks that replace one-shot retrieve-then-generate pipelines with explicit, multi-step interaction between inference and external information access. The label was used in two distinct 2025 systems: **"R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning"** [2506.04185], which casts reasoning–search interaction as a reinforcement-learning problem over interleaved reasoning and retrieval, and **"Reinforcement Fine-Tuning for Reasoning towards Multi-Step Multi-Source Search in Large Language Models"** [2506.08352], which uses **Reasoning-Search (R-Search)** for a single-LLM framework that emits reasoning, a natural-language search DAG, retrieved results, and a final answer in one structured trace. Both formulations target the same failure mode: parametric reasoning is strong, but factual accuracy, timeliness, and multi-hop evidence integration remain bottlenecks unless retrieval behavior itself becomes a learned or explicitly planned component of the inference policy. ## 1. Scope and named variants The term **R-Search** is not yet a single standardized architecture. In the 2025 literature it refers to at least two closely related but technically distinct formulations, both centered on search-augmented reasoning rather than static retrieval augmentation [2506.04185] [2506.08352]. | Variant | Core structure | Distinctive mechanism | |---|---|---| | R-Search (multi-reward RL) | Interleaved reasoning and search trajectory | Evidence block plus answer/evidence/format rewards | | R-Search (Reasoning-Search) | ``, ``, ``, `` | Natural-language DAG with multi-source execution | | Later related development | Retrieval–reasoning boundary calibration | Reasoning regions, reflection, and tree-based RL | The first system treats advanced retrieval-augmented generation as a sequential **Reason \(\bowtie\) Search** process in which the model learns when to retrieve, when to continue internal reasoning, and how to integrate retrieved material globally. The second system embeds the entire search workflow inside a single structured generation trace, with the model itself outputting a natural-language directed acyclic graph over heterogeneous sources. This suggests that **R-Search** is best understood as a research direction rather than a single immutable implementation. ## 2. Problem formulation: from retrieve-then-generate to search-conditioned reasoning The multi-reward RL formulation models advanced RAG as an interleaved sequence \[ a_1 \rightsquigarrow a_2 \rightsquigarrow \cdots \rightsquigarrow a_T,\qquad a_t \in \{\mathcal{S}, \mathcal{R}\}, \] where \(\mathcal{S}\) denotes a search action and \(\mathcal{R}\) denotes a reasoning action [2506.04185]. The entire pipeline is treated as a **partially observable Markov decision process**, with state composed of the question, prior generated reasoning, and retrieved documents; the hidden environment is the external corpus. The technical motivation is that multi-hop tasks such as HotpotQA, MuSiQue, and Bamboogle require not merely access to documents, but learned control over **when** to retrieve, **what** to retrieve, and **how** to propagate evidence across multiple steps. The single-LLM Reasoning-Search formulation uses a closely related MDP abstraction, \[ \mathcal{M} = (\mathcal{S}, \mathcal{A}, \mathcal{F}, \mathcal{O}, \mathbb{P}, R, L), \] but operationalizes it through a structured textual trace rather than tool-call tags [2506.08352]. Its state is the full textual trajectory; its action space includes reasoning, DAG construction, and answer synthesis; and its tools are external search functions spanning news, arXiv, and general web search. Here the key critique of prior systems is that multi-agent search frameworks incur substantial computational overhead, while single-LLM tool-calling baselines usually restrict themselves to sequential, single-query, single-source search. Taken together, these formulations treat **search as part of the policy**, not as a preprocessing stage. This suggests a conceptual shift from **retrieval augmentation** to **agentic search control**, in which the model’s search behavior becomes a first-class optimization target. ## 3. Multi-reward reinforcement-learning R-Search The framework in [2506.04185] uses a policy model \(\pi_\theta\) based on **Qwen-2.5-Instruct** in 3B and 7B variants, a dense retriever based on **E5**, a frozen reference model \(\pi_{\text{ref}}\) for KL regularization, and a frozen cross-family evaluator \(\pi_{\text{cf}}\) implemented as **Llama-3.2-3B-Instruct** for evidence scoring. Search interaction is encoded directly in the generated text: search queries are enclosed in `cyan{ }`, retrieved observations are returned in `brown{ }`, globally integrated evidence is emitted in `blue{ }`, and the final answer is emitted in `purple{ }`. Its RL objective is \[ \mathcal{J}(\theta ) = \mathbb{E}_{q\sim \mathcal{D},\,o\sim \pi_{\theta}(\cdot \mid q;\mathcal{S})} \left[ r_{\phi }(q,o) \right] -\beta \,\mathbb{D}_{\text{KL}} \!\left[ \pi_{\theta}(o \mid q;\mathcal{S}) \parallel \pi_{\text{ref}}(o \mid q;\mathcal{S}) \right]. \] The central design choice is the **multi-stage, multi-type reward**. Final answer quality is scored by token-level F1 between the predicted answer in `purple{ }` and the gold answer. Evidence quality is scored indirectly: the policy generates an evidence block \(e\) in `blue{ }`, then the cross-family evaluator answers the question using only \(q\) and \(e\), and the resulting F1 becomes the evidence reward. A format reward encourages exactly one evidence block when search occurs, exactly one answer block, and valid tag structure. The total reward is \[ r_\phi = r^\alpha_\phi + r^e_\phi + r^f_\phi, \] with \(r^\alpha_\phi, r^e_\phi \in [0,1]\) and \(r^f_\phi\) determined by \(\gamma_e = \gamma_\alpha = 0.2\) [2506.04185]. Two implementation details are especially consequential. First, retrieved-document tokens are **masked** during policy optimization, preventing gradient contamination from environment-generated text. Second, evidence tokens are **not masked**, because evidence generation is itself part of the learned reasoning policy. This makes the evidence block a trainable intermediate representation rather than a post hoc summary. Training uses only **2WikiMultiHopQA** for RL, yet evaluation spans seven datasets. The system samples five rollouts per question, uses top-\(k=3\) retrieval during training and top-\(k=5\) at evaluation, and is trained on **8 \(\times\) A100 80GB**, with maximum training steps 195, batch size 256, learning rate \(10^{-6}\), warm-up ratio 0.95, and KL coefficient \(\beta=0.001\) [2506.04185]. PPO and GRPO are both evaluated, but GRPO converges faster and to higher reward ceilings. ## 4. Reasoning-Search as a single-LLM multi-source search trace The second R-Search formulation, in [2506.08352], uses a single model—**DeepSeek-R1-Distill-Qwen-7B**—to emit the entire search-and-answer workflow as one structured output: \[ \begin{aligned} y = &\ \underbrace{\ r\ }_{\text{reasoning}} \underbrace{\ G\ }_{\text{planning}} \underbrace{\ R\ }_{\text{execution}} \underbrace{\ a\ }_{\text{synthesis}} . \end{aligned} \] Here \(G=(V,E)\) is a **natural-language DAG**. Nodes are query–tool pairs such as “A: Price fluctuation of coke ... (News)”, and edges encode dependencies such as `A -> C; B -> C`. The environment parses the DAG, validates acyclicity and tool names, topologically sorts the graph, executes independent nodes in parallel, and injects the retrieved results into `` before the model generates ``. This formulation is explicitly **multi-source**. The tool set \(\mathcal{T}\) includes **GNews** with **Serper** fallback for news, **arXiv REST API** for academic search, and **Serper GoogleSearch** for general web search. Each node retrieves the top-2 passages. Because tools are specified at the node level, the plan can distribute subproblems across heterogeneous sources without requiring separate planner and synthesizer models. Training uses **Reinforcement Fine-Tuning (ReFT)** based on **GRPO**, implemented in **TRL 0.18.0** with a modified GRPO trainer. The reward is \[ \mathcal{R}(y,a)= \alpha_{\text{fmt}}\mathcal{F}_{\text{fmt}}(y)+ \alpha_{\text{dag}}\mathcal{F}_{\text{dag}}(G)+ \alpha_{\text{ans}}\mathcal{F}_{\text{ans}}(y,a), \] with \[ \alpha_{\text{fmt}}:\alpha_{\text{dag}}:\alpha_{\text{ans}} = 0.25:0.25:0.5. \] \(\mathcal{F}_{\text{fmt}}\) rewards exact structural compliance with the four tags; \(\mathcal{F}_{\text{dag}}\) checks acyclicity, node syntax, and tool validity; and \(\mathcal{F}_{\text{ans}}\) uses **GPT-4.1-Mini** as a judge for open-ended tasks and F1 for multiple-choice tasks [2506.08352]. The training corpus is synthetic but explicitly time-sensitive and multi-source: about **2000 news articles** and **1500 arXiv AI/ML abstracts** from March–May 2025 are clustered with **Sentence-BERT**, **UMAP** reduction, hierarchical clustering, and recursive K-means, then used by **GPT-4.1-Mini** to generate about **1000** multi-hop QA pairs [2506.08352]. This design directly targets stale-knowledge failure modes rather than static Wikipedia-only reasoning. ## 5. Empirical performance and ablations The multi-reward RL system reports results on **seven datasets**—HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle, NQ, TriviaQA, and PopQA—and states that it **outperforms advanced RAG baselines by up to 32.2% (in-domain) and 25.1% (out-of-domain)** [2506.04185]. For **Qwen-2.5-7B-Instruct**, its overall average F1 is **53.6**, compared with **52.3** for Search-R1, **38.8** for Adaptive-RAG, and **37.8** for Vanilla RAG. On multi-hop subsets the gap is larger: average F1 is **54.6** for R-Search versus **52.4** for Search-R1 and **31.6** for both IRCoT and Adaptive-RAG. Evidence ablation confirms that the explicit evidence stage matters: for the 7B model, HotpotQA F1 drops from **64.4** to **61.9**, MuSiQue from **41.6** to **39.6**, and TriviaQA from **71.7** to **70.2** when evidence is removed. GRPO also dominates PPO: on the 7B model, multi-hop average F1 rises from **49.1** to **54.6**, and single-hop average from **50.2** to **52.1** [2506.04185]. The single-LLM Reasoning-Search system is evaluated on **FinSearchBench-24**, **SearchExpertBench-25**, and seven QA benchmarks [2506.08352]. On FinSearchBench-24 it reports **78.13 \(\pm\) 1.97%** accuracy, **1535.55 \(\pm\) 167.18** tokens, and **8.58 \(\pm\) 0.49 s** latency; on SearchExpertBench-25 it reports **73.00 \(\pm\) 3.46%** accuracy, **1811.34 \(\pm\) 139.54** tokens, and **9.60 \(\pm\) 0.68 s** latency. Relative to multi-agent baselines such as FinSearch and SearchExpert, it reports a **70% reduction in context token usage** and an **approximately 50% decrease in execution latency** while also improving accuracy [2506.08352]. On the seven QA benchmarks, its average EM is **41.87%**. Ablation results are sharp: on FinSearchBench-24, disabling multi-source search while keeping ReFT reduces accuracy to **54.63%**, whereas enabling multi-source search but removing ReFT reduces it further to **49.28%**; the full system reaches **78.13%** [2506.08352]. This indicates that explicit multi-source planning and RL-based structural optimization are both load-bearing components. These results support two distinct empirical claims. First, learned control over retrieval timing and evidence representation improves complex multi-hop QA. Second, explicit search plans over heterogeneous sources can outperform heavier multi-agent orchestration while using fewer tokens and less wall-clock time. ## 6. Position in agentic search research and later developments Both R-Search systems are positioned against three baselines: **standard RAG**, **heuristic multi-step RAG**, and **multi-agent search orchestration**. The multi-reward RL variant compares against Iter-RetGen, IRCoT, FLARE, Adaptive-RAG, and Search-R1, arguing that prompt-engineered retrieval heuristics misalign retrieval timing with actual information need [2506.04185]. The single-LLM Reasoning-Search variant compares against SearchAgent, MindSearch, FinSearch, SearchExpert, Search-o1, Search-R1, and ZeroSearch, arguing that multi-agent planners incur substantial overhead while sequential single-query tool-calling methods fail to represent parallel or source-specialized search plans [2506.08352]. A subsequent line of work, **"R\(^2\)-Searcher: Calibrating Retrieval and Reasoning Boundaries for Agentic Search"** [2606.28566], makes explicit a failure mode already implicit in both R-Search variants: the **retrieval–reasoning boundary shift**. It defines a retrieval boundary by whether the observation contains the minimal required factual elements for the current query and a reasoning boundary by whether the model’s effective reasoning region actually uses those elements. Its solution adds query-token-guided reasoning regions, retrieval reflection, and a tree-based RL algorithm called **R\(^2\)PO**, and reports consistent gains over AutoRefine and other agentic baselines on seven complex QA benchmarks. This suggests that the R-Search agenda has already evolved from learning **when** to search and **how** to plan search into learning **which retrieved facts deserve reasoning attention**. In that sense, R-Search marks a transition point in search-augmented language modeling. It replaces retrieval as a passive context-provider with search as an explicit, optimized component of the reasoning policy. The two 2025 systems differ in interface—tagged interleaving versus natural-language DAGs—and in supervision—multi-reward evidence-centric RL versus structural GRPO with DAG validity—but both move the field away from shallow retrieve-then-generate pipelines toward fully agentic search-conditioned reasoning [2506.04185] [2506.08352]. Source: https://www.emergentmind.com/topics/r-search