Search-R1++: Dynamic Retrieval Integration

• Search-R1++ is a family of search-integrated reasoning systems that interleaves token generation with explicit search actions, using reinforcement learning to decide when to retrieve information.
• It refines earlier models like R1-Searcher by employing a dual-phase training strategy—including SFT cold-start and RL—to standardize control tokens and enhance dynamic knowledge acquisition.
• Variants such as Fast Thinking, LLDS, and ExpandSearch demonstrate practical advances in prompt design, reward shaping, and query expansion that improve answer accuracy while reducing retrieval cost.

Search-R1++ denotes a family of Search-R1-derived systems for search-integrated reasoning in which a LLM, and in some variants a large multimodal model, interleaves token generation with explicit search actions and is post-trained with reinforcement learning to decide when to search, how to query, and when to answer. In recent literature, the label is used both for the named framework "R1-Searcher++: Incentivizing the Dynamic Knowledge Acquisition of LLMs via Reinforcement Learning" and for several later refinements that alter prompt templates, reward shaping, policy optimization, memorization, query expansion, and trajectory repair (Song et al., 22 May 2025, Xu et al., 23 Feb 2026). A common feature across these systems is that reasoning plus retrieval is treated as a sequential decision problem rather than as a fixed retrieval-augmented generation pipeline.

1. Genealogy and terminological scope

The immediate precursor is "R1-Searcher," which models the interaction between an LLM and an external retrieval engine as an episodic Markov decision process. Its state is the concatenation of the original question, previously generated reasoning tokens, and retrieved documents; its action space is an extended vocabulary containing normal tokens together with retrieval-control tags such as <begin_of_query>, <end_of_query>, <begin_of_documents>, and <end_of_documents>; and its reward is strictly outcome-based, with no intermediate process rewards (Song et al., 7 Mar 2025).

"R1-Searcher++" then reorients the framework around what it calls Dynamic Knowledge Acquisition. It introduces a two-stage pipeline consisting of an SFT Cold-start phase for learning a standardized <internal>/<external>/<document> format and an RL phase that rewards correct answers, penalizes malformed traces, discourages excessive retrieval, and adds a memorization mechanism that assimilates retrieved information into model parameters (Song et al., 22 May 2025).

A later line of work uses "Search-R1++" as the name of a refined baseline derived from ablation over prompt template, reward function, and optimizer. In that usage, Search-R1++ is not a new search protocol so much as a recipe: Fast Thinking prompts, F1+ reward shaping, and REINFORCE with a small KL constraint (Xu et al., 23 Feb 2026).

A common misconception is that Search-R1++ denotes one standardized architecture. The cited literature instead uses the name for several closely related extensions of Search-R1-style agents. This suggests that Search-R1++ is best understood as a research family centered on end-to-end learning of search behavior, with different papers instantiating that family through different control tokens, reward functions, and stabilization mechanisms.

2. Agent architecture and search-control interfaces

In the original R1-Searcher formulation, generation and retrieval are unified at the token level. Whenever the policy emits <begin_of_query>, it continues generation until <end_of_query> is produced; the resulting query string is submitted to a search API; returned documents are appended between <begin_of_documents> and <end_of_documents>; and auto-regressive decoding resumes. The model therefore learns retrieval invocation as part of its ordinary next-token policy rather than through an external router (Song et al., 7 Mar 2025).

R1-Searcher++ generalizes the interface to distinguish internal and external knowledge explicitly. The backbone policy $\pi_\theta$ generates traces interleaving <internal> ... </internal> blocks, where the model uses parametric memory, and <external> query </external> blocks, after which the retriever returns top-$K$ documents and inserts them as <document> ... </document>. A separate memory module stores pairs of the form $(q,\text{reasoning path without retrieval})$ for later fine-tuning, reflecting the paper’s emphasis on converting externally acquired knowledge into internal knowledge over time (Song et al., 22 May 2025).

The 2026 Search-R1++ baseline simplifies this interface further through the Fast Thinking template. It removes mandatory > ... </think> blocks and instructs the model only to emit <search>query</search> when external knowledge is needed and <answer> ... </answer> when ready to respond. Returned evidence is inserted in <information> ... </information> blocks. The explicit goal is to constrain exploration to search and answer decisions rather than to long intermediate reasoning traces (Xu et al., 23 Feb 2026).

ExpandSearch extends the action space from a single query to a set of query variants. At each </search> step, the policy emits a block of the form query₁ ## query₂ ## … ## queryₙ; these queries are searched in parallel; and a frozen "squeezer" model condenses the retrieved chunks into a short summary injected back into the agent context. In this variant, the agent is optimized primarily for high-recall query generation, while document condensation is delegated to the auxiliary model (Zhao et al., 11 Oct 2025).

3. Reinforcement-learning objectives and training recipes

R1-Searcher uses a two-stage, outcome-based RL curriculum. Stage 1 is a tool-invocation warm-up with

$R_1 = R_{retrieval} + R_{format},$

where $R_{retrieval}=0.5$ if at least one retrieval occurs and $0$ otherwise, and $R_{format}=0.5$ if the invocation format is correct and $0$ otherwise. Stage 2 replaces this with

$R_2 = R_{answer} + R'_{format},$

where $R_{answer}=\mathrm{F1}(\hat y,y^*)$ and $K$0 for correct format and $K$1 for incorrect format. The objective is

$K$2

optimized with a policy-gradient objective using KL regularization and a moving-average baseline. Retrieved-document tokens are masked out in the loss so that the model is not trained to imitate tool output (Song et al., 7 Mar 2025).

R1-Searcher++ replaces this with a reward that explicitly balances correctness and retrieval economy:

$K$3

Here, $K$4 for correct tags and boxed answer format and $K$5 otherwise; $K$6 when Cover Exact Match is true and the answer has at most 10 words, else $K$7; and $K$8 rewards the correct rollout that uses the minimum number of <external> calls among correct rollouts in the same batch, with variance-based clipping. The RL loss is paired with a memorization loss

$K$9

where $(q,\text{reasoning path without retrieval})$0 trains on internal-only rewritten traces derived from successful externally grounded trajectories (Song et al., 22 May 2025).

The ablation-based Search-R1++ baseline studies three reward functions: Exact Match, token-level F1, and F1+ with action penalties. Its key shaped reward is

$(q,\text{reasoning path without retrieval})$1

with $(q,\text{reasoning path without retrieval})$2, where $(q,\text{reasoning path without retrieval})$3 is the number of <search> actions and $(q,\text{reasoning path without retrieval})$4 the number of <answer> actions. The paper reports that pure F1 underperforms EM because training collapses through answer avoidance, whereas F1+ arrests this behavior and ultimately surpasses EM. Within that study, REINFORCE outperforms PPO and GRPO, while requiring fewer search actions (Xu et al., 23 Feb 2026).

Taken together, these formulations show a progression from sparse answer-only supervision toward reward designs that encode retrieval formatting, participation in answering, search minimization, and knowledge transfer from external evidence into the model’s internal state.

4. Balancing internal knowledge, external retrieval, and search cost

A central design question in Search-R1++ systems is not merely how to retrieve, but when retrieval should be avoided. R1-Searcher++ addresses this directly through two mechanisms. First, the batch-level group reward gives bonus only to correct traces with the minimum number of external calls among correct rollouts, so over-retrieval is penalized indirectly through relative comparison. Second, the memorization loss converts successful retrieval-grounded traces into internal-only traces using a rewriting model, thereby encouraging the model to answer similar questions from internal memory in later episodes (Song et al., 22 May 2025).

ExpandSearch addresses the same issue from the opposite direction. It assumes that single-query search is a bottleneck for compound and multi-hop questions, so it trains the agent to emit several query variants in parallel. Because such expansion greatly increases retrieved context, the frozen squeezer compresses all retrieved chunks into a reasoning-focused summary. The reported analysis distinguishes syntax expansions, which constitute 63.35% of generated queries, from semantic expansions, which constitute 36.65%, and ablations show that removing the squeezer reduces average EM from 0.446 to 0.364 (Zhao et al., 11 Oct 2025).

In multimodal settings, MMSearch-R1 makes search cost explicit in the terminal reward. Its reward combines exact-match accuracy and format compliance with a search penalty:

$(q,\text{reasoning path without retrieval})$5

with $(q,\text{reasoning path without retrieval})$6 and $(q,\text{reasoning path without retrieval})$7, where $(q,\text{reasoning path without retrieval})$8 indicates whether any search was invoked. The model can choose between an image-search action and a text-search action, and the paper argues that balanced training data containing both search-required and search-free samples is essential; without search penalty or balanced data, the model converges to near-100% search usage (Wu et al., 25 Jun 2025).

These mechanisms all instantiate the same principle: external retrieval is useful only when it complements, rather than supplants, internal knowledge. Search-R1++ systems therefore tend to couple correctness rewards with either explicit cost signals or implicit competitive pressures that favor minimal tool use.

5. Failure modes, stabilization, and fine-grained credit assignment

One recurring failure mode concerns prompt-induced degeneracy. In the Search-R1 ablation study, the Slow Thinking template forces every reasoning step into <think> ... </think> blocks. The paper reports that the policy learns to stack empty <think> blocks because each tag becomes spuriously correlated with positive return; this leads to exploding reasoning length and collapse in answer quality. The Fast Thinking template removes these mandatory tags and yields more stable learning curves by restricting the token-level action space to search and answer decisions (Xu et al., 23 Feb 2026).

A second failure mode is optimizer instability. The same study reports that REINFORCE achieves the best final accuracy, PPO remains stable but tends toward near-constant search behavior, and GRPO shows the poorest stability. A more detailed diagnosis is given in "On GRPO Collapse in Search-R1," which identifies Lazy Likelihood Displacement (LLD) as the core mechanism. The paper defines per-step log-likelihood displacement

$(q,\text{reasoning path without retrieval})$9

and describes a three-phase trajectory of early stagnation, steady decay, and accelerated collapse. To prevent this, it introduces LLDS, a rollout-gated and token-selective likelihood-preserving regularizer, and LLDS-MA, which masks final answer spans inside that regularizer. Vanilla GRPO collapses by approximately step 200 in all four tested model variants, whereas LLDS-augmented training remains stable (Deng et al., 3 Dec 2025).

A third limitation is sparse credit assignment. Search-R2 argues that trajectory-level outcome rewards cannot distinguish solid evidence-driven reasoning from lucky guesses and that a single bad search step can invalidate an otherwise good trajectory. Its solution is an Actor–Meta-Refiner decomposition: the Actor generates an initial search-integrated trajectory, a Discriminator decides whether to accept it, and a Trimmer selects a cut-point if repair is needed; the suffix is then regenerated from that prefix. Search-R2 supplements exact-match outcome reward with a dense process reward measuring the information density of retrieved chunks, using the gated hybrid form

$R_1 = R_{retrieval} + R_{format},$0

This is presented as a way to repair local errors without discarding the entire rollout (He et al., 3 Feb 2026).

Across these works, stabilization progresses along three axes: removing degenerate action channels, reshaping rewards to prevent refusal or over-search, and replacing undifferentiated trajectory credit with token-, step-, or suffix-level corrective signals.

6. Empirical profile, broader variants, and open directions

Empirical comparison is complicated by metric heterogeneity. R1-Searcher reports Cover-Exact-Match ($R_1 = R_{retrieval} + R_{format},$1) and LLM-as-Judge ($R_1 = R_{retrieval} + R_{format},$2) on HotpotQA, 2WikiMultiHopQA, Bamboogle, and Musique; R1-Searcher++ reports average F1, LasJ, and Retrieval Count; the ablation-defined Search-R1++ reports Exact Match; LLDS reports EM on NQ+Hotpot training setups; and multimodal work reports average VQA accuracy and search ratio. Cross-paper comparisons therefore require attention to both evaluation metric and retrieval setting (Song et al., 7 Mar 2025, Song et al., 22 May 2025, Xu et al., 23 Feb 2026).

Variant	Distinctive mechanism	Representative reported outcome
R1-Searcher	Two-stage pure RL with autonomous query invocation	Qwen-2.5-7B reaches 0.654/0.750 on HotpotQA and 0.636/0.650 on 2Wiki for $R_1 = R_{retrieval} + R_{format},$3 (Song et al., 7 Mar 2025)
R1-Searcher++	SFT cold-start, dynamic knowledge acquisition, memorization	Average over four datasets: F1 45.3, LasJ 55.2, RC 1.61; 30% fewer retrievals than R1-Searcher (Song et al., 22 May 2025)
Search-R1++ baseline	Fast Thinking, F1+, REINFORCE	Avg. EM improves from 0.403 to 0.442 on Qwen2.5-7B and from 0.289 to 0.331 on Qwen2.5-3B (Xu et al., 23 Feb 2026)
LLDS-augmented Search-R1++	Likelihood-preserving regularization for GRPO	On Qwen2.5-3B-Base, NQ+Hotpot: 31.2 EM to 43.0 with LLDS-MA; on Qwen2.5-7B-Base: 35.0 to 46.2 (Deng et al., 3 Dec 2025)
Search-R2	Actor–Refiner collaboration and dense process reward	Average EM over seven QA benchmarks rises from 35.0 to 40.4 on Qwen2.5-7B and from 45.6 to 50.8 on Qwen2.5-32B (He et al., 3 Feb 2026)
ExpandSearch	Query expansion plus frozen squeezer	Average improvement of 4.4% over state-of-the-art baselines; 3B-Instruct reaches avg EM 0.457 (Zhao et al., 11 Oct 2025)

The empirical profile is therefore not one of a single dominant recipe, but of recurring gains from several orthogonal modifications. R1-Searcher establishes that search invocation can be learned from scratch via RL; R1-Searcher++ shows that internal/external balancing and memorization can reduce retrieval count without sacrificing judged quality; the ablation-defined Search-R1++ identifies prompt simplification and reward shaping as major contributors to stability; LLDS demonstrates that GRPO collapse is not incidental but structurally tied to likelihood dynamics; Search-R2 shows that trajectory repair and process reward can improve multi-hop QA further (Song et al., 22 May 2025, Xu et al., 23 Feb 2026, Deng et al., 3 Dec 2025, He et al., 3 Feb 2026).

The label also extends beyond text-only QA. MMSearch-R1 applies the same search-integrated RL pattern to multimodal VQA with separate image-search and text-search actions, reporting 54.6% average accuracy against 51.6% for an equal-size RAG baseline while reducing search ratio from 100% to 67.1% (Wu et al., 25 Jun 2025). In a different reinterpretation, R1-Compress uses chunk compression plus inter-chunk search to shorten Long Chain-of-Thought traces; on MATH500 with Qwen2.5-32B it reports 92.4% accuracy, a 0.6 percentage-point drop from Long-CoT, with $R_1 = R_{retrieval} + R_{format},$4 versus $R_1 = R_{retrieval} + R_{format},$5, corresponding to a 15.3% token reduction and an approximately 19% valid-token reduction (Wang et al., 22 May 2025). These uses indicate that "Search-R1++" can also denote search-mediated control over internal reasoning artifacts, not only web retrieval.

Open directions are similarly diverse. The cited works explicitly propose actor-critic or Q-learning variants, learned cost penalties for dynamic retrieval budgets, latent memory modules, hierarchical RL, richer reward models beyond EM, joint training of auxiliary components such as squeezers or summarizers, and stronger source tracking and verification mechanisms (Song et al., 7 Mar 2025, Wu et al., 25 Jun 2025). The unifying agenda is consistent: to turn search from a rigid external add-on into a learned, budgeted, and increasingly self-correcting component of reasoning.