OPDSearch+

• OPDSearch+ combines live-search interaction with forward-KL distillation from a frozen, off-the-shelf instruct model, enhancing search and reasoning behaviors in small language models (Qwen2.5-3B) without task-specific fine-tuning.
• This framework transfers complex reasoning, query formation, and evidence integration through two stages: live-search on-policy distillation and GRPO-based reinforcement learning, with stage one focusing on behavior initialization and stage two on outcome optimization.
• OPDSearch+ achieved notable improvements in multi-hop retrieval benchmarks, with significant gains reported on datasets like 2WikiMultihopQA and HotpotQA, demonstrating effective transfer of online search-augmented reasoning capacities and techniques.

OPDSearch+ is an on-policy distillation and reinforcement-learning framework for search-augmented reasoning by small LLMs. It combines live-search interaction with per-position forward-KL distillation from a frozen, off-the-shelf instruct model, followed by GRPO-based refinement using answer-correctness and formatting rewards. The framework is designed to transfer reasoning decomposition, query formulation, and evidence integration without fine-tuning a task-specific teacher, while allowing reinforcement learning to improve beyond the teacher’s task performance ceiling (Ye et al., 25 Aug 2026).

1. Problem setting and motivation

Search-augmented reasoning requires a LLM to perform more than answer generation. For a multi-hop question, the model must determine whether search is necessary, identify entities and intermediate relations, formulate useful queries, interpret retrieved passages, decide whether additional searches are required, and synthesize evidence into a final answer. The interaction can be represented as a sequence in which generated queries alter subsequent retrieved contexts:

$$\text{query}_1 \rightarrow \text{retrieved passages}_1 \rightarrow \text{query}_2 \rightarrow \text{retrieved passages}_2 \rightarrow \cdots$$

This dynamic environment creates difficulties for both supervised fine-tuning and reinforcement learning. Offline teacher trajectories are generated under a particular retriever and corpus; changes in the student’s first query can produce different passages and consequently different later reasoning states. The resulting distribution shift compounds over multiple search turns.

Reinforcement learning from scratch addresses the student’s own interaction distribution but provides sparse credit assignment. Search-R1-style methods use answer-level rewards such as exact match, which assigns essentially the same outcome signal to reasoning and query tokens even when only some search actions were useful. A small model may also be unlikely to discover entity-specific decomposition or effective multi-hop querying, resulting in few successful trajectories from which to learn.

Directly applying on-policy distillation with an off-the-shelf teacher removes the cost of task-specific teacher training but introduces a different limitation: the student may imitate a teacher whose search behavior is imperfect. OPDSearch+ therefore treats the teacher as a policy reshaper rather than as the final policy. The teacher first moves the student toward a richer behavioral region; subsequent RL adapts that policy to the benchmark’s verifiable objective.

The student is a Qwen2.5-3B model, and the teacher is a frozen Qwen2.5-14B-Instruct model. The teacher is not fine-tuned on the search task. The complete training recipe is:

$$\boxed{\text{live-search on-policy distillation}\;\longrightarrow\;\text{GRPO answer refinement}}$$

2. Interactive search formulation

For a question $x$, the student generates an interactive trajectory

$o=(o_1,o_2,\ldots,o_T).$

A typical trajectory interleaves reasoning, tool calls, retrieved information, further reasoning, and the final answer:

$T_{\max}=4$7

The search engine is treated as part of the environment. It receives a generated query and returns passages that are appended to the context. Retrieved passage tokens are not sampled actions of the student policy and are excluded from the optimized token set.

During training, each question produces $G=8$ trajectories, and the maximum search horizon is $T_{\max}=4$ search turns. The student therefore determines the prefixes and retrieved evidence on which the teacher operates. This distinguishes OPDSearch+ from teacher-generated offline trajectories: the teacher evaluates the contexts that the student actually encounters under the live retriever.

The generated positions are partitioned into reasoning, query, and answer tokens:

$$\mathcal T_R,\qquad \mathcal T_Q,\qquad \mathcal T_A,$$

with

$$\mathcal T_R\cup\mathcal T_Q\cup\mathcal T_A=\mathcal M,$$

where $\mathcal M$ denotes all model-generated token positions. The stage-one objective applies to reasoning tokens, search-query tokens, and answer tokens, but not to inserted <information> tokens. No separate hand-designed reward is assigned to the three generated-token categories in stage one.

This masking strategy permits the teacher to supervise query construction and reasoning without treating retrieved documents as actions that the policy should imitate. A query token favored by the teacher but under-produced by the student receives a larger update coefficient, as do teacher-favored reasoning or answer tokens.

3. Two-stage training architecture

Stage one: live-search OPD

For each training question, the student samples a trajectory from its current policy. Each generated search query is sent to the live retriever, and the returned passages are appended to the student’s context. The frozen teacher then evaluates the resulting trajectory, including the actual retrieved passages.

The procedure is:

1. Sample a trajectory from the current student policy.
2. Send every generated search query to the live retriever.
3. Append retrieved passages to the context.
4. Continue generation until an answer is produced or the search horizon is reached.
5. Run the frozen teacher on the student-generated trajectory and context.
6. Compute a forward-KL distillation update over model-generated tokens.

The best OPD checkpoint is reported as step 150 in the experimental configuration. Stage one transfers reasoning decomposition, search-query formation, and evidence integration while preserving the student’s interaction with the current retrieval environment.

Stage two: RL refinement

Stage two initializes GRPO from the best stage-one checkpoint. The teacher is no longer used. The student again produces live-search trajectories, and GRPO optimizes a task reward combining exact-match correctness and answer formatting:

$R=0.9R_{\mathrm{EM}}+0.1R_{\mathrm{format}}.$

Here, $$\boxed{\text{live-search on-policy distillation}\;\longrightarrow\;\text{GRPO answer refinement}}$$0 is the verifiable exact-match reward and $$\boxed{\text{live-search on-policy distillation}\;\longrightarrow\;\text{GRPO answer refinement}}$$1 measures compliance with the required answer format. The paper does not report an additional evidence-verification reward, query-quality reward, or separately defined process reward in the final OPDSearch+ stage.

The RL procedure is:

$T_{\max}=4$8

The RL stage uses batch size 1024 and is stopped when collapse is observed. The reported framework therefore separates dense behavioral initialization from sparse outcome optimization: stage one makes useful search behaviors more probable, while stage two adapts them to answer correctness.

Sequential rather than joint optimization

A joint RL-plus-OPD objective was tested but performed substantially worse than sequential OPD followed by RL. The reported interpretation is objective interference: simultaneous teacher imitation and sparse outcome optimization do not provide the clean behavioral warmup produced by stage-one distillation followed by task-specific refinement.

4. Forward-KL distillation objective

Let the student log-probability of generated token $$\boxed{\text{live-search on-policy distillation}\;\longrightarrow\;\text{GRPO answer refinement}}$$2 be

$$\boxed{\text{live-search on-policy distillation}\;\longrightarrow\;\text{GRPO answer refinement}}$$3

and let the frozen teacher’s log-probability for the same token, in the same student-generated context, be

$$\boxed{\text{live-search on-policy distillation}\;\longrightarrow\;\text{GRPO answer refinement}}$$4

The teacher-to-student importance ratio is

$$\boxed{\text{live-search on-policy distillation}\;\longrightarrow\;\text{GRPO answer refinement}}$$5

OPDSearch+ clips this ratio:

$$\boxed{\text{live-search on-policy distillation}\;\longrightarrow\;\text{GRPO answer refinement}}$$6

using

$$\boxed{\text{live-search on-policy distillation}\;\longrightarrow\;\text{GRPO answer refinement}}$$7

The stage-one forward-KL loss is implemented as teacher-weighted cross-entropy:

$$\boxed{\text{live-search on-policy distillation}\;\longrightarrow\;\text{GRPO answer refinement}}$$8

where $$\boxed{\text{live-search on-policy distillation}\;\longrightarrow\;\text{GRPO answer refinement}}$$9 denotes stop-gradient. The ratio is detached, so the student is updated through its token log-probability rather than through the weighting coefficient.

Without clipping, taking the expectation over student-generated tokens yields a gradient equivalent to minimizing the teacher-to-student forward KL at each prefix:

$x$0

Clipping introduces controlled bias but bounds the importance weights. The complete stage-one objective adds a KL penalty relative to the frozen initial student checkpoint:

$x$1

where

$x$2

and the reported OPD coefficient is

$x$3

The KL term limits catastrophic drift from the original student policy.

Implicit supervision of search behavior

The gradient of the forward-KL loss is

$x$4

Separating token categories gives

$x$5

where $x$6.

Consequently, OPDSearch+ does not require a manually designed query-quality reward in stage one. The teacher’s token distribution supplies dense supervision for query generation, reasoning, and answer production. The same mechanism can promote an entity-specific query, a useful intermediate inference step, or an answer token that the teacher considers more probable than the student’s alternative.

Reverse-KL comparison

The paper also evaluates a reverse-KL surrogate:

$x$7

with teacher log-probabilities detached.

The reported explanation for reverse-KL instability is that it can assign large coefficients to actions already favored by the student. Because these actions are sampled frequently, a positive-feedback loop may develop, producing entropy collapse. Forward KL instead emphasizes teacher-supported actions that the student under-produces, while ratio clipping bounds their influence.

Objective	Stability	Best validation EM	Gradient norm
Forward KL	Stable for 400+ steps	0.493	0.73–1.19
Reverse-KL adaptive	Stable for 160+ steps	0.470	0.23–0.32
Reverse-KL clipped	Degrades by about step 60	0.136	2.2–4.5
Reverse-KL JSD	Collapses around step 20	—	$x$8
Reverse-KL log-ratio	Collapses around step 20	—	$x$9

The authors caution that reverse-KL failures might be mitigated through objective-specific tuning. Under shared settings, however, forward KL is reported as both more stable and more effective.

5. Policy reshaping and optimization dynamics

OPDSearch+ is motivated by the observation that a 3B model may assign insufficient probability to effective search behaviors. Examples include identifying an entity in a question, generating a focused subquery, using one retrieved fact to form the next query, and terminating after sufficient evidence has been collected.

The stage-one teacher does not merely provide a sequence for imitation. It changes the student’s policy distribution over search and reasoning behaviors. The reported entropy increases from

$o=(o_1,o_2,\ldots,o_T).$0

to

$o=(o_1,o_2,\ldots,o_T).$1

This increase is interpreted as a broader behavioral distribution and greater exploration capacity rather than simple mode copying.

RL from scratch must discover multi-hop decomposition through successful trajectories. When such trajectories are initially rare, answer-level rewards provide little useful direction. OPDSearch+ increases the probability of useful decompositions, allowing GRPO to observe successful trajectory groups and refine them.

The paper also describes the process geometrically: the base student and the OPD-initialized student enter RL from different regions of policy space. RL from the base model may become trapped in behaviors such as excessive searching, copying the original question into queries, or answering without adequate evidence. OPD moves the student toward a region containing better decompositions, allowing RL to focus on answer correctness instead of discovering the entire search strategy.

Reported training observations include:

• forward-KL OPD policy drift of approximately KL $o=(o_1,o_2,\ldots,o_T).$2 from the reference;
• pure RL drift exceeding KL $o=(o_1,o_2,\ldots,o_T).$3 before collapse;
• average search turns decreasing from $o=(o_1,o_2,\ldots,o_T).$4 to $o=(o_1,o_2,\ldots,o_T).$5;
• OPD gradient norms decreasing from approximately $o=(o_1,o_2,\ldots,o_T).$6 to $o=(o_1,o_2,\ldots,o_T).$7 over 418 steps;
• RL after OPD beginning with gradient norms around $o=(o_1,o_2,\ldots,o_T).$8;
• OPD remaining stable for more than 400 steps and reportedly through 501 steps.

These observations support the interpretation that OPD supplies a dense and comparatively stable warm start. They do not establish a universal guarantee that forward-KL distillation or subsequent RL will converge reliably under other teachers, retrievers, model sizes, or search horizons.

6. Experimental evaluation and limitations

Experimental configuration

The student is Qwen2.5-3B base, and the primary teacher is frozen Qwen2.5-14B-Instruct. A Qwen2.5-7B-Instruct teacher is used in a teacher-scale ablation.

Training uses approximately 170,000 QA pairs from the NQ and HotpotQA training splits. Evaluation covers:

• single-hop retrieval: NQ, TriviaQA, and PopQA;
• multi-hop retrieval: HotpotQA, 2WikiMultihopQA, MuSiQue, and Bamboogle.

The retriever is E5-base-v2 over a December 2018 Wikipedia dump, with the top three passages returned per search. The maximum search horizon is four turns, and each question generates eight rollouts. Training uses veRL on four H200 GPUs with a learning rate of $o=(o_1,o_2,\ldots,o_T).$9. Stage-one OPD uses batch size 512 and more than 420 planned updates, with reported curves extending to 501 steps. The evaluation metric is exact match, with single-hop and overall averages.

Main results

Method	NQ	TriviaQA	PopQA	SH-Avg	HotpotQA	2Wiki	MuSiQue	Bamboogle	Avg
Direct Generation	.106	.288	.108	.167	.149	.244	.020	.024	.134
SFT	.249	.292	.104	.215	.186	.248	.044	.112	.176
Naive RAG	.348	.544	.387	.426	.255	.226	.047	.080	.270
Search-R1-Base	.421	.583	.413	.472	.297	.274	.066	.128	.312
AutoRefine-Base	.467	.620	.450	.512	.405	.393	.157	.344	.405
GiGPO-Instruct	.420	.595	.424	.480	.369	.370	.126	.641	.421
OPDSearch+	.4852	.6226	.4513	.520	.4580	.4264	.1819	.4560	.4402

OPDSearch+ obtains a mean exact match of $G=8$0, compared with $G=8$1 for GiGPO-Instruct. The paper reports:

• HotpotQA: $G=8$2, a reported $G=8$3 gain over AutoRefine-Base;
• 2WikiMultihopQA: $G=8$4, a reported $G=8$5 gain over AutoRefine-Base;
• MuSiQue: $G=8$6, the best listed 3B result;
• Bamboogle: $G=8$7, below GiGPO’s $G=8$8.

The largest gains occur on multi-hop benchmarks, consistent with the proposed transfer of decomposition and evidence-chaining behavior.

Ablations

The teacher-scale ablation is:

Method	Avg	HotpotQA	2Wiki	MuSiQue	Bamboogle
OPD only, 7B teacher	.3311	.2872	.2875	.1150	.3920
OPD only, 14B teacher	.3655	.3602	.3281	.1419	.4000
OPD $G=8$9 RL, 7B teacher	.4205	.4357	.3944	.1910	.4240
OPD $T_{\max}=4$0 RL, 14B teacher	.4402	.4580	.4264	.1819	.4560

A larger teacher improves OPD-only performance, but RL refinement remains important for both teacher sizes.

The distillation-versus-RL comparison is:

Method	Avg	HotpotQA	2Wiki	Bamboogle
Reproduced pure RL	.3524	.3467	.3142	.1760
OPD only, 14B $T_{\max}=4$1 3B	.3655	.3602	.3281	.4000
OPD $T_{\max}=4$2 RL	.4402	.4580	.4264	.4560

The paper reports that OPD-only improves over offline SFT, while RL refinement is essential for the final result. The difference between $T_{\max}=4$3 and $T_{\max}=4$4 is reported as a $T_{\max}=4$5 relative improvement.

Sequential optimization also outperforms joint optimization:

Method	Avg
Joint RL + OPD	.3660
OPD $T_{\max}=4$6 RL	.4402

Limitations

The reported results depend on a particular experimental environment: E5-base-v2, a December 2018 Wikipedia dump, three retrieved passages per search, and a maximum of four search turns. Robustness to other retrievers, changing web corpora, noisy search APIs, or substantially longer trajectories is not established.

Teacher fine-tuning is eliminated, but teacher inference remains necessary during stage one. The 14B teacher must score student-generated trajectories token by token, creating substantial computational cost. The approach also uses eight rollouts per question and long interactive contexts.

Stage two relies on exact-match and formatting rewards rather than direct measures of retrieval quality, evidence faithfulness, or reasoning validity. Correct answers may therefore result from accidental retrieval or unsupported reasoning, while reward hacking remains possible. The paper does not report citation correctness, human evaluation, retrieval precision and recall, tool-call cost, or search latency.

Other reproducibility limitations include missing details about the relative size of OPD and search batches, the weighting between objectives, total compute, teacher-server throughput, GRPO clipping parameters, decoding settings, passage formatting, and search-corpus splits. The report also does not provide a numerical live-search-versus-offline-search ablation under otherwise identical conditions.

Bamboogle contains only 125 samples, making its unusually high GiGPO result and comparative differences less reliable. The RL stage is stopped when collapse is observed, which makes best-checkpoint selection and stability comparisons sensitive to the stopping protocol.

OPDSearch+ is therefore best characterized as a two-stage behavioral-transfer framework rather than a universally optimal search-training algorithm. Its central contribution is the use of live-search, student-generated trajectories for forward-KL distillation, followed by RL refinement. The reported evidence indicates that this combination can improve small-model multi-hop search reasoning beyond both offline SFT and RL from scratch, while avoiding the need to fine-tune a task-specific teacher.