R-Search: Agentic Search-Enhanced Reasoning

• R-Search is a search-augmented reasoning framework that integrates agentic, multi-step retrieval with internal reasoning to enhance LLM performance.
• It employs reinforcement learning to optimize the interplay between reasoning and retrieval, addressing factual accuracy and multi-hop challenge bottlenecks.
• Two variants—multi-reward RL and natural-language DAG outputs—demonstrate reduced latency and token usage while significantly improving multi-source evidence integration.

In contemporary LLM research, R-Search denotes a class of reasoning–search frameworks that replace one-shot retrieve-then-generate pipelines with explicit, multi-step interaction between inference and external information access. The label was used in two distinct 2025 systems: "R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning" (Zhao et al., 4 Jun 2025), which casts reasoning–search interaction as a reinforcement-learning problem over interleaved reasoning and retrieval, and "Reinforcement Fine-Tuning for Reasoning towards Multi-Step Multi-Source Search in LLMs" (Shi et al., 10 Jun 2025), which uses Reasoning-Search (R-Search) for a single-LLM framework that emits reasoning, a natural-language search DAG, retrieved results, and a final answer in one structured trace. Both formulations target the same failure mode: parametric reasoning is strong, but factual accuracy, timeliness, and multi-hop evidence integration remain bottlenecks unless retrieval behavior itself becomes a learned or explicitly planned component of the inference policy.

1. Scope and named variants

The term R-Search is not yet a single standardized architecture. In the 2025 literature it refers to at least two closely related but technically distinct formulations, both centered on search-augmented reasoning rather than static retrieval augmentation (Zhao et al., 4 Jun 2025, Shi et al., 10 Jun 2025).

Variant	Core structure	Distinctive mechanism
R-Search (multi-reward RL)	Interleaved reasoning and search trajectory	Evidence block plus answer/evidence/format rewards
R-Search (Reasoning-Search)	>, <search>, <result>, <answer>	Natural-language DAG with multi-source execution
>	Later related development	Retrieval–reasoning boundary calibration

The first system treats advanced retrieval-augmented generation as a sequential Reason $\bowtie$ Search process in which the model learns when to retrieve, when to continue internal reasoning, and how to integrate retrieved material globally. The second system embeds the entire search workflow inside a single structured generation trace, with the model itself outputting a natural-language directed acyclic graph over heterogeneous sources. This suggests that R-Search is best understood as a research direction rather than a single immutable implementation.

2. Problem formulation: from retrieve-then-generate to search-conditioned reasoning

The multi-reward RL formulation models advanced RAG as an interleaved sequence

$$a_1 \rightsquigarrow a_2 \rightsquigarrow \cdots \rightsquigarrow a_T,\qquad a_t \in \{\mathcal{S}, \mathcal{R}\},$$

where $\mathcal{S}$ denotes a search action and $\mathcal{R}$ denotes a reasoning action (Zhao et al., 4 Jun 2025). The entire pipeline is treated as a partially observable Markov decision process, with state composed of the question, prior generated reasoning, and retrieved documents; the hidden environment is the external corpus. The technical motivation is that multi-hop tasks such as HotpotQA, MuSiQue, and Bamboogle require not merely access to documents, but learned control over when to retrieve, what to retrieve, and how to propagate evidence across multiple steps.

The single-LLM Reasoning-Search formulation uses a closely related MDP abstraction,

$$\mathcal{M} = (\mathcal{S}, \mathcal{A}, \mathcal{F}, \mathcal{O}, \mathbb{P}, R, L),$$

but operationalizes it through a structured textual trace rather than tool-call tags (Shi et al., 10 Jun 2025). Its state is the full textual trajectory; its action space includes reasoning, DAG construction, and answer synthesis; and its tools are external search functions spanning news, arXiv, and general web search. Here the key critique of prior systems is that multi-agent search frameworks incur substantial computational overhead, while single-LLM tool-calling baselines usually restrict themselves to sequential, single-query, single-source search.

Taken together, these formulations treat search as part of the policy, not as a preprocessing stage. This suggests a conceptual shift from retrieval augmentation to agentic search control, in which the model’s search behavior becomes a first-class optimization target.

3. Multi-reward reinforcement-learning R-Search

The framework in (Zhao et al., 4 Jun 2025) uses a policy model $\pi_\theta$ based on Qwen-2.5-Instruct in 3B and 7B variants, a dense retriever based on E5, a frozen reference model $\pi_{\text{ref}}$ for KL regularization, and a frozen cross-family evaluator $\pi_{\text{cf}}$ implemented as Llama-3.2-3B-Instruct for evidence scoring. Search interaction is encoded directly in the generated text: search queries are enclosed in cyan{ }, retrieved observations are returned in brown{ }, globally integrated evidence is emitted in blue{ }, and the final answer is emitted in purple{ }.

Its RL objective is

$$\mathcal{J}(\theta ) = \mathbb{E}_{q\sim \mathcal{D},\,o\sim \pi_{\theta}(\cdot \mid q;\mathcal{S})} \left[ r_{\phi }(q,o) \right] -\beta \,\mathbb{D}_{\text{KL}} \!\left[ \pi_{\theta}(o \mid q;\mathcal{S}) \parallel \pi_{\text{ref}}(o \mid q;\mathcal{S}) \right].$$

The central design choice is the multi-stage, multi-type reward. Final answer quality is scored by token-level F1 between the predicted answer in purple{ } and the gold answer. Evidence quality is scored indirectly: the policy generates an evidence block $e$ in blue{ }, then the cross-family evaluator answers the question using only $$a_1 \rightsquigarrow a_2 \rightsquigarrow \cdots \rightsquigarrow a_T,\qquad a_t \in \{\mathcal{S}, \mathcal{R}\},$$0 and $$a_1 \rightsquigarrow a_2 \rightsquigarrow \cdots \rightsquigarrow a_T,\qquad a_t \in \{\mathcal{S}, \mathcal{R}\},$$1, and the resulting F1 becomes the evidence reward. A format reward encourages exactly one evidence block when search occurs, exactly one answer block, and valid tag structure. The total reward is

$$a_1 \rightsquigarrow a_2 \rightsquigarrow \cdots \rightsquigarrow a_T,\qquad a_t \in \{\mathcal{S}, \mathcal{R}\},$$2

with $$a_1 \rightsquigarrow a_2 \rightsquigarrow \cdots \rightsquigarrow a_T,\qquad a_t \in \{\mathcal{S}, \mathcal{R}\},$$3 and $$a_1 \rightsquigarrow a_2 \rightsquigarrow \cdots \rightsquigarrow a_T,\qquad a_t \in \{\mathcal{S}, \mathcal{R}\},$$4 determined by $$a_1 \rightsquigarrow a_2 \rightsquigarrow \cdots \rightsquigarrow a_T,\qquad a_t \in \{\mathcal{S}, \mathcal{R}\},$$5 (Zhao et al., 4 Jun 2025).

Two implementation details are especially consequential. First, retrieved-document tokens are masked during policy optimization, preventing gradient contamination from environment-generated text. Second, evidence tokens are not masked, because evidence generation is itself part of the learned reasoning policy. This makes the evidence block a trainable intermediate representation rather than a post hoc summary.

Training uses only 2WikiMultiHopQA for RL, yet evaluation spans seven datasets. The system samples five rollouts per question, uses top-$$a_1 \rightsquigarrow a_2 \rightsquigarrow \cdots \rightsquigarrow a_T,\qquad a_t \in \{\mathcal{S}, \mathcal{R}\},$$6 retrieval during training and top-$$a_1 \rightsquigarrow a_2 \rightsquigarrow \cdots \rightsquigarrow a_T,\qquad a_t \in \{\mathcal{S}, \mathcal{R}\},$$7 at evaluation, and is trained on 8 $$a_1 \rightsquigarrow a_2 \rightsquigarrow \cdots \rightsquigarrow a_T,\qquad a_t \in \{\mathcal{S}, \mathcal{R}\},$$8 A100 80GB, with maximum training steps 195, batch size 256, learning rate $$a_1 \rightsquigarrow a_2 \rightsquigarrow \cdots \rightsquigarrow a_T,\qquad a_t \in \{\mathcal{S}, \mathcal{R}\},$$9, warm-up ratio 0.95, and KL coefficient $\mathcal{S}$0 (Zhao et al., 4 Jun 2025). PPO and GRPO are both evaluated, but GRPO converges faster and to higher reward ceilings.

4. Reasoning-Search as a single-LLM multi-source search trace

The second R-Search formulation, in (Shi et al., 10 Jun 2025), uses a single model—DeepSeek-R1-Distill-Qwen-7B—to emit the entire search-and-answer workflow as one structured output: $\mathcal{S}$1 Here $\mathcal{S}$2 is a natural-language DAG. Nodes are query–tool pairs such as “A: Price fluctuation of coke ... (News)”, and edges encode dependencies such as A -> C; B -> C. The environment parses the DAG, validates acyclicity and tool names, topologically sorts the graph, executes independent nodes in parallel, and injects the retrieved results into <result> before the model generates <answer>.

This formulation is explicitly multi-source. The tool set $\mathcal{S}$3 includes GNews with Serper fallback for news, arXiv REST API for academic search, and Serper GoogleSearch for general web search. Each node retrieves the top-2 passages. Because tools are specified at the node level, the plan can distribute subproblems across heterogeneous sources without requiring separate planner and synthesizer models.

Training uses Reinforcement Fine-Tuning (ReFT) based on GRPO, implemented in TRL 0.18.0 with a modified GRPO trainer. The reward is

$\mathcal{S}$4

with

$\mathcal{S}$5

$\mathcal{S}$6 rewards exact structural compliance with the four tags; $\mathcal{S}$7 checks acyclicity, node syntax, and tool validity; and $\mathcal{S}$8 uses GPT-4.1-Mini as a judge for open-ended tasks and F1 for multiple-choice tasks (Shi et al., 10 Jun 2025).

The training corpus is synthetic but explicitly time-sensitive and multi-source: about 2000 news articles and 1500 arXiv AI/ML abstracts from March–May 2025 are clustered with Sentence-BERT, UMAP reduction, hierarchical clustering, and recursive K-means, then used by GPT-4.1-Mini to generate about 1000 multi-hop QA pairs (Shi et al., 10 Jun 2025). This design directly targets stale-knowledge failure modes rather than static Wikipedia-only reasoning.

5. Empirical performance and ablations

The multi-reward RL system reports results on seven datasets—HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle, NQ, TriviaQA, and PopQA—and states that it outperforms advanced RAG baselines by up to 32.2% (in-domain) and 25.1% (out-of-domain) (Zhao et al., 4 Jun 2025). For Qwen-2.5-7B-Instruct, its overall average F1 is 53.6, compared with 52.3 for Search-R1, 38.8 for Adaptive-RAG, and 37.8 for Vanilla RAG. On multi-hop subsets the gap is larger: average F1 is 54.6 for R-Search versus 52.4 for Search-R1 and 31.6 for both IRCoT and Adaptive-RAG. Evidence ablation confirms that the explicit evidence stage matters: for the 7B model, HotpotQA F1 drops from 64.4 to 61.9, MuSiQue from 41.6 to 39.6, and TriviaQA from 71.7 to 70.2 when evidence is removed. GRPO also dominates PPO: on the 7B model, multi-hop average F1 rises from 49.1 to 54.6, and single-hop average from 50.2 to 52.1 (Zhao et al., 4 Jun 2025).

The single-LLM Reasoning-Search system is evaluated on FinSearchBench-24, SearchExpertBench-25, and seven QA benchmarks (Shi et al., 10 Jun 2025). On FinSearchBench-24 it reports 78.13 $\mathcal{S}$9 1.97% accuracy, 1535.55 $\mathcal{R}$0 167.18 tokens, and 8.58 $\mathcal{R}$1 0.49 s latency; on SearchExpertBench-25 it reports 73.00 $\mathcal{R}$2 3.46% accuracy, 1811.34 $\mathcal{R}$3 139.54 tokens, and 9.60 $\mathcal{R}$4 0.68 s latency. Relative to multi-agent baselines such as FinSearch and SearchExpert, it reports a 70% reduction in context token usage and an approximately 50% decrease in execution latency while also improving accuracy (Shi et al., 10 Jun 2025). On the seven QA benchmarks, its average EM is 41.87%. Ablation results are sharp: on FinSearchBench-24, disabling multi-source search while keeping ReFT reduces accuracy to 54.63%, whereas enabling multi-source search but removing ReFT reduces it further to 49.28%; the full system reaches 78.13% (Shi et al., 10 Jun 2025). This indicates that explicit multi-source planning and RL-based structural optimization are both load-bearing components.

These results support two distinct empirical claims. First, learned control over retrieval timing and evidence representation improves complex multi-hop QA. Second, explicit search plans over heterogeneous sources can outperform heavier multi-agent orchestration while using fewer tokens and less wall-clock time.

6. Position in agentic search research and later developments

Both R-Search systems are positioned against three baselines: standard RAG, heuristic multi-step RAG, and multi-agent search orchestration. The multi-reward RL variant compares against Iter-RetGen, IRCoT, FLARE, Adaptive-RAG, and Search-R1, arguing that prompt-engineered retrieval heuristics misalign retrieval timing with actual information need (Zhao et al., 4 Jun 2025). The single-LLM Reasoning-Search variant compares against SearchAgent, MindSearch, FinSearch, SearchExpert, Search-o1, Search-R1, and ZeroSearch, arguing that multi-agent planners incur substantial overhead while sequential single-query tool-calling methods fail to represent parallel or source-specialized search plans (Shi et al., 10 Jun 2025).

A subsequent line of work, "R$\mathcal{R}$5-Searcher: Calibrating Retrieval and Reasoning Boundaries for Agentic Search" (Zhang et al., 26 Jun 2026), makes explicit a failure mode already implicit in both R-Search variants: the retrieval–reasoning boundary shift. It defines a retrieval boundary by whether the observation contains the minimal required factual elements for the current query and a reasoning boundary by whether the model’s effective reasoning region actually uses those elements. Its solution adds query-token-guided reasoning regions, retrieval reflection, and a tree-based RL algorithm called R$\mathcal{R}$6PO, and reports consistent gains over AutoRefine and other agentic baselines on seven complex QA benchmarks. This suggests that the R-Search agenda has already evolved from learning when to search and how to plan search into learning which retrieved facts deserve reasoning attention.

In that sense, R-Search marks a transition point in search-augmented language modeling. It replaces retrieval as a passive context-provider with search as an explicit, optimized component of the reasoning policy. The two 2025 systems differ in interface—tagged interleaving versus natural-language DAGs—and in supervision—multi-reward evidence-centric RL versus structural GRPO with DAG validity—but both move the field away from shallow retrieve-then-generate pipelines toward fully agentic search-conditioned reasoning (Zhao et al., 4 Jun 2025, Shi et al., 10 Jun 2025).