Search-E1: Self-Distillation Drives Self-Evolution in Search-Augmented Reasoning

Abstract: Post-training has become the dominant recipe for turning a LLM into a competent search-augmented reasoning agent. A line of recent work pushes its performance further by adding elaborate machinery on top of this standard pipeline. These augmentations import external supervision from stronger external systems, attach auxiliary modules such as process reward models or retrospective critics, restructure the rollout itself with tree search or multi-stage curricula, or shape the reward with hand-crafted bonuses and penalties. Each addition delivers a measurable gain, but each also inflates the training pipeline and ties the recipe to resources or designs that may not always be available. We take a step back and ask whether any of this machinery is actually necessary, and propose Search-E1, a self-evolution method that lets a search-augmented agent improve through only vanilla GRPO interleaved with offline self-distillation (OFSD). After each GRPO round, the policy rolls out on its own training questions. A token-level forward KL objective then aligns the policy's inference-time distribution to its own distribution under a privileged context that exposes a more efficient sibling trajectory. Despite this simplicity, the procedure naturally provides dense per-step supervision. On seven QA benchmarks, Search-E1 reaches $0.440$ average EM with Qwen2.5-3B, surpassing all open-source baselines at both scales. Code and complete version will be made public soon.

• The paper presents a novel self-distillation method that leverages internal model rollouts to provide dense, per-decision supervision for search-augmented reasoning.
• It integrates a standard GRPO reinforcement learning baseline with an offline self-distillation step to enhance QA performance, especially on multi-hop datasets.
• Empirical results demonstrate that internal self-supervision can outperform external methods, closing performance gaps without relying on additional annotations.

Search-E1: Self-Distillation-Driven Self-Evolution in Search-Augmented Reasoning

Overview and Motivation

Search-E1 presents a streamlined approach to post-training LLMs for knowledge-intensive, search-augmented reasoning tasks. In contrast to a recent trend of progressively complex training pipelines—such as those involving external supervision, auxiliary modules, step-level critics, or rich reward shaping—this work introduces a minimalist two-stage alternation: standard Grouped Rollout Policy Optimization (GRPO) and a novel, fully internal, offline self-distillation (OFSD) step. The core claim is that dense, per-timestep supervision can be derived solely from the model’s own rollouts, with no recourse to external annotations or modules, leading to measurable gains in question answering (QA) benchmarks over all comparative open-source systems at the 3B and 7B scale.

Methodological Framework

GRPO Baseline

The primary training backbone mirrors GRPO (Shao et al., 2024), a reinforcement learning protocol where multiple trajectories per question are sampled, each receiving trajectory-wide scalar rewards based on exact-match answer correctness. This approach, while robust and practically verifiable, is limited by uniform credit assignment within a trajectory—providing no differentiated supervision for specific action steps (e.g., search query formation).

Offline Self-Distillation (OFSD)

To address the sparse and imprecise nature of trajectory-level supervision, Search-E1 introduces OFSD as an offline, interleaved procedure:

• Pair Mining: After a GRPO round, for each question, trajectories are grouped, and a pair is selected—a reference trajectory ($T_{\text{ref}}$), typically the shortest correct one, and a student trajectory ($T_{\text{stu}}$), typically the most divergent incorrect one.
• Asymmetric Conditioning: The policy model acts as both student and teacher. The student receives a standard prompt; the teacher receives a privileged context with $T_{\text{ref}}$ attached.
• Token-Level Alignment: A forward KL-divergence loss aligns the student's token-level distribution to the teacher’s across positions generated by the policy, with per-token clipping to ensure gradient stability.

This design guarantees dense, per-decision credit assignment derived strictly from model-internal information, yielding an integrated, modular training pipeline that maintains compatibility with future developments in retrieval, backbone design, or RL strategies.

Alternating Self-Evolution Pipeline

The training alternates GRPO (exploratory RL on full trajectories) and OFSD (consolidation of per-step best practices). This alternation is modular—neither stage introduces any per-timestep complexity to the other or imposes auxiliary training burdens or dependencies.

Empirical Results

On seven leading QA benchmarks (NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultihopQA, MuSiQue, and Bamboogle), Search-E1 with Qwen2.5-3B-Instruct achieves an average EM of 0.440, outperforming all open-source baselines at this model size. Notably, the gains are most pronounced on multi-hop datasets (HotpotQA +2.2, 2Wiki +4.3, MuSiQue +3.6 over AutoRefine), confirming that per-step supervision via self-distillation confers the greatest benefit when complex, multi-stage reasoning is required.

A particularly important claim is that, unlike StepSearch or GiGPO, which depend on external signals (GPT-4o-generated annotations or anchor state grouping), Search-E1 achieves comparable or superior results relying exclusively on QA pairs. The approach also closes (and slightly inverts) the typical performance gap between “Base” and “Instruct” variants, indicating that per-step self-supervision can compensate for, and even surpass, curated pre-training.

Theoretical and Practical Implications

The strong numerical gains of Search-E1 suggest that much of the incremental benefit from complex, externally-supervised process reward models or hand-crafted reward shaping in prior work may be principally accessible via internal model self-comparison, provided an appropriate offline distillation protocol is established. The procedure’s independence from teacher signals makes it especially promising for domains lacking high-quality step-level annotations or where external supervision is infeasible.

By not modifying the GRPO RL regime or introducing trajectory format changes, the method is future-proof—compatible with advances in retrieval, model scaling, and RL paradigms. The implication is that further technique development in the RL or retrieval component can be integrated directly, continuing to benefit from dense internal per-step alignment.

Potential and Future Directions

The authors point toward several natural extensions:

• Expanded Privileged Context: Conditioning the OFSD teacher on sets of correct sibling trajectories or leveraging transfer from similar questions could provide a richer per-step supervision signal without violating the on-policy character of GRPO.
• Extended Alternation Schedules: Empirical evidence shows that gains persist across multiple GRPO+OFSD cycles, although diminishing returns eventually set in as backbone model capacity ceilings are reached.
• Transfer to Other Tool-Augmented Paradigms: The OFSD procedure is not limited to search retrieval augmentation and may generalize to other tool-augmented or structured action spaces.

Conclusion

Search-E1 demonstrates that an alternating schedule of standard GRPO and fully internal, offline self-distillation suffices for state-of-the-art search-augmented reasoning without external supervision, auxiliary models, or complicated reward shaping. Per-step supervision can be efficiently extracted from sibling rollout contrasts, providing dense guidance that closes or reverses traditional gaps observed in post-training paradigms. This approach is likely to inform future directions in scalable, annotation-light, tool-augmented LLM development.

For complete methodological and empirical details, refer to "Search-E1: Self-Distillation Drives Self-Evolution in Search-Augmented Reasoning" (2605.22511).