R1-Searcher++: Retrieval-Augmented LLM Framework

• R1-Searcher++ is a retrieval-augmented language model framework that combines two-stage supervised and reinforcement learning with a memorization mechanism.
• The framework uses outcome-based reinforcement learning to enable the model to dynamically choose between internal knowledge and external retrieval for improved efficiency.
• It achieves state-of-the-art accuracy on multi-hop QA benchmarks while significantly decreasing the number of external retrieval calls compared to previous methods.

R1-Searcher++ is a retrieval-augmented LLM framework that extends standard Retrieval-Augmented Generation (RAG) by enabling LLMs to dynamically and efficiently leverage both their internal knowledge and external retrieval sources, using a two-stage training pipeline based on supervised fine-tuning (SFT) and reinforcement learning (RL) with a memorization mechanism. Its design specifically addresses the limitations of static-knowledge LLMs and conventional RAG—namely, hallucinations, inefficient over-reliance on retrieval, poor generalization, and lack of dynamic knowledge assimilation—by incentivizing strategic internal/external knowledge selection and continual updating of the model’s knowledge base with newly retrieved information.

1. Framework Structure and Dynamic Knowledge Utilization

R1-Searcher++ operates in a two-stage training paradigm:

• SFT Cold-start: The model is first fine-tuned on synthetic data to learn the formatting and basic control required for tool usage, using explicit tags such as <internal> (to signal use of internal knowledge), <external> (to initiate external retrieval), and <document> (to denote content from retrieved sources). Document tokens from retrieval are masked during the loss computation to avoid overfitting to retrieval content.
• RL for Dynamic Knowledge Acquisition: This stage employs a policy-gradient RL method (REINFORCE++) for outcome-based optimization, rewarding the model for correct answers, proper format usage, and minimizing unnecessary retrieval calls. At each reasoning step, the model autonomously chooses whether to attempt solving with internal knowledge or to query external resources.

The operational decision loop:

• Generate a step with <internal> or <external> tag.
• If <internal>, rely on parametric knowledge for reasoning.
• If <external>, formulate a query, integrate the retrieved document as <document>, and proceed with updated context.

This design ensures R1-Searcher++ uses external retrieval judiciously and learns to internalize information over time.

2. Training and Reward Mechanisms

The RL phase is guided entirely by outcome-based supervision, which consists of:

• Format Reward ($R_{\text{format}}$): Enforces correct tool call structure.

$$R_{\text{format}}(q, o_i) = \begin{cases} 0, & \text{if format correct} \ -2, & \text{if incorrect} \end{cases}$$

• Answer Reward ($R_{\text{answer}}$): Evaluates answer correctness using Cover Exact Match (CEM) and conciseness.

$R_{\text{answer}}(q, o_i) = \begin{cases} 1, & \text{\text{CEM true} and answer $\leq$ 10 words} \ 0, & \text{otherwise} \end{cases}$

• Group Reward ($R_{\text{group}}$): Incentivizes correct answers with minimal retrievals, proportional to standard deviation in retrieval counts:

$$R'_{\text{group}}(q,o_i) = \begin{cases} 2 \times \sigma^2, & R_{\text{answer}}(q,o_i) = 1 \wedge t_i = t_{\min} \ 0, & \text{otherwise} \end{cases}$$

$$R_{\text{group}}(q, o_i) = \min(R'_{\text{group}}(q, o_i), \eta)$$

where $\eta$ is a predefined clip.

• Aggregate Reward:

$$R(q, o_i) = R_{\text{format}}(q, o_i) + R_{\text{answer}}(q, o_i) + R_{\text{group}}(q, o_i)$$

Masking is applied during policy optimization so that the loss function only updates model-generated tokens and not tokens copied from external retrievals.

3. Memorization and Self-Improvement Mechanism

Unlike traditional retrieval-augmented models where external information is not retained, R1-Searcher++ incorporates a memorization step. After RL exploration, it:

• Identifies trajectories where external retrieval contributed to a correct answer.
• Uses a separately fine-tuned rewrite model to convert retrieval-augmented reasoning steps into an internal-knowledge-only version.
• Forms a dataset $\mathcal{T}$ of these "internalized" examples and adds their negative log-likelihood to the training objective:

$$\mathcal{L}_{\text{M}}(\theta) = -\frac{1}{\sum_{o_i \in \mathcal{T}} |o_i|} \sum_{o_i \in \mathcal{T}} \sum_{t=1}^{|o_i|} \log \pi_\theta(o_{i, t} \mid q, o_{i, <t})$$

• The final model loss is

$$\mathcal{L}(\theta) = -\mathcal{J}_{\text{Mask}}(\theta) + \mu \cdot \mathcal{L}_{\text{M}}(\theta)$$

with $\mu$ controlling tradeoff between policy RL and memorization.

This approach enables the continual enhancement of internal knowledge, reducing repeated external retrieval for similar problems.

4. Experimental Results and Comparative Performance

R1-Searcher++ demonstrates substantial improvements on multi-hop, open-domain QA benchmarks, including HotpotQA, 2WikiMultiHopQA (in-domain) and Musique, Bamboogle (out-of-domain). The model outperforms strong RL-based baselines such as R1-Searcher and Search-R1, not only in accuracy but in efficiency:

Model	Avg LLM-as-Judge (%)	Avg F1 (%)	Avg Retrieval Calls
R1-Searcher	52.9	45.6	2.30
Search-o1	43.9	36.6	1.47
R1-Searcher++	55.2	45.3	1.61

• R1-Searcher++ increases LLM-judged answer accuracy by up to 4.3% over RL baselines.
• Mean retrieval calls are decreased by up to 42.9%, indicating more efficient use of external resources.
• The model sustains strong performance in online settings (e.g., with Google API), indicating practical generalization.

Ablation studies confirm that both the group reward (discouraging unnecessary retrievals) and memorization steps are critical to achieving these efficiency and correctness improvements.

5. Implementation and Practical Considerations

R1-Searcher++ is implemented using Qwen-2.5-7B-Instruct as foundation, with BGE-large-en-v1.5 as the dense retriever and FlashRAG for retrieval integration. Training leverages DeepSpeed Zero-3 for efficient memory scaling. All components, including code and data pipelines, are made publicly available:

• https://github.com/RUCAIBox/R1-Searcher-plus

Requirements include access to the backbone model, a compatible retrieval toolkit (FlashRAG), and suitable hardware for distributed RL. Both local and online retrieval are supported, with the model demonstrating robust adaptability across settings.

6. Significance and Broader Impact

R1-Searcher++ represents a significant advance in retrieval-augmented LLMs. Its key contributions include:

• Efficient integration of internal and external knowledge with minimal redundancy.
• Outcome-supervised, RL-driven training for dynamic knowledge selection.
• Explicit mechanisms to internalize external knowledge, promoting continual self-improvement.
• Demonstrated state-of-the-art multi-hop reasoning accuracy with fewer retrievals and strong generalization to new domains and live web data.

A plausible implication is that this approach makes efficient, human-like reasoning feasible for LLMs in open, rapidly changing knowledge environments, while controlling resource consumption and supporting domain transfer.

7. Summary Table

Feature	R1-Searcher++ Implementation	Performance/Impact
Internal/External Decision	Explicit tags and outcome-supervised RL	Reduces unnecessary retrieval calls, increases accuracy
Memorization	Post-hoc internalization and NLL on transformed data	Internal knowledge base grows with retrieved information
Efficiency	Group reward and masking in policy optimization	Fewer retrievals, lower latency, higher sample efficiency
Generalization	Evaluated on in-domain and out-of-domain tasks	Robust accuracy across QA datasets and online retrieval

R1-Searcher++ is positioned as a generalizable, extensible framework for research and deployment of retrieval-augmented LLMs where dynamic and efficient knowledge acquisition is required.