DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment

Abstract: Training tool-use agents to improve from their own experience remains challenging, as supervised fine-tuning relies on fixed teacher-distilled trajectories, while sparse-reward reinforcement learning provides weak supervision for long-horizon interactions. We present DeepSearch-Evolve, a self-distillation framework for web agents built on DeepSearch-World, a deterministic and verifiable environment with reproducible search and page-reading tools. DeepSearch-World contains 420K multi-hop QA tasks constructed from entity-level random walks and supports key agentic cognitive behaviors useful for self-evolving, including progress verification, grounded reflection, and failure recovery. DeepSearch-Evolve iteratively performs trajectory generation, filtering, data mixing, and fine-tuning to train stronger agents. Without distillation from more capable models, DeepSearch-World-9B achieves competitive performance compared with open-source agents, reaching 31.2% on BrowseComp, 61.5% on GAIA, and 93.4% on HotpotQA, showing that verifiable environments enable scalable self-evolution for long-horizon web agents. We will release the environment, 420K training pool, validation set, model, and code to facilitate future research on self-improving deep search agents.

• The paper introduces a deterministic, verifiable environment (DeepSearch-World) built on a 10M-entry Wikipedia corpus for generating complex multi-hop QA tasks.
• It proposes DeepSearch-Evolve, a scaffolded self-distillation framework integrating a structured teacher agent, trajectory verification, and rejection sampling.
• Experimental results demonstrate significant performance gains in tool usage and evidence aggregation, outperforming baselines on key benchmarks.

DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment

Background and Motivation

Effective training of long-horizon tool-using agents remains constrained by limitations inherent in current learning paradigms: supervised fine-tuning (SFT) saturates quickly due to fixed teacher data and limited trajectory diversity, whereas RL-style self-improvement suffers from reward sparsity and lacks actionable supervision for intermediate steps, especially in long and complex operations. On-policy self-distillation (OPSD) addresses reward sparsity by distilling dense token-level policies, but in the context of agentic tool use, its effectiveness is hindered by environmental indeterminacy and distributional shift. The authors introduce DeepSearch-World, a deterministic and verifiable virtual environment targeting these bottlenecks, and propose DeepSearch-Evolve, a framework for scalable self-improvement via environment-grounded, scalable self-distillation.

Figure 1: Schematic comparison of self-evolving paradigms—DeepSearch-Evolve derives advantage from verified, process-level supervision in a deterministic world supporting planning, memory, error correction, and query rewriting.

DeepSearch-World: Design and Implementation

DeepSearch-World is instantiated as a fully deterministic Wikipedia-based environment comprising two main offline tools: a BM25-based document search interface and a document retrieval ("visit") interface. The environment is underpinned by a 10M-entry local Wikipedia corpus, enabling verifiable retrieval and page access. Task instances are rigorously constructed: 420K multi-hop QA tasks are generated using entity-level random walks in the Wikipedia knowledge graph, then “fuzzified” to ensure non-triviality, requiring cross-document reasoning and factual synthesis inaccessible via pattern matching. Each QA instance is equipped with ground-truth entity sets, allowing fine-grained, step-level progress verification and automated, order-free assignment of tool-use success or failure.

Crucially, the design enables environment-grounded reflection: unsuccessful tool calls trigger staged, rule-based recovery signals rather than generic error messages. These verifiable, interpretable signals support the formation of robust failure recovery and query reformulation policies, which are systematically used during scaffold teacher rollouts.

Figure 2: DeepSearch-World enables asynchronous self-evolving SFT via verified trajectory generation, data filtering, and importance-sampled replay, with a strict separation of deterministic, verifiable environment interactions and iterative student model optimization.

Scaffolded Self-Distillation with DeepSearch-Evolve

DeepSearch-Evolve integrates (i) a structured scaffolded teacher agent, (ii) trajectory verification and quality filtering, and (iii) supervised distillation into a single iterative framework. The teacher agent operates in three explicit phases—Plan, Act, End—maintaining a structured progress state for subgoals, learned experiences, and evidence tracking. Interactions are mediated by strict policies on tool invocation sequences and memory management. Each action-observation pair is annotated with fine-grained rationales and decision traces, which are converted into ReAct-format supervision through systematic removal of environment-specific prompts and internal states, instead synthesizing concise self-correcting reasoning blocks.

Trajectory data is further filtered via rejection sampling (ensuring final task correctness) and trajectory quality filtering (removing redundant or illogical steps). This process is asynchronous and importance-sampled across evolving rounds, maintaining data diversity and mitigating catastrophic forgetting.

Figure 3: The scaffolded teacher's Plan–Act–End paradigm ensures explicit decomposition, evidence collection, and systematic failure recovery.

Figure 4: Multi-hop QA is systematically decomposed, with evidence checkpoints, error handling, and grounded self-correction at each stage.

Experimental Results

DeepSearch-World-9B (a 9B-parameter instantiation of Qwen3.5) is evaluated against a comprehensive suite of benchmarks (BrowseComp, GAIA, HotpotQA, xbench, HLE, BrowseComp-ZH) and competitive proprietary/open-source search agents. Unlike many open-source baselines, which are frequently bootstrapped from stronger teacher policies or multi-agent synthetic pipelines, DeepSearch-World-9B is strictly self-improved via its own verified rollouts without external teacher data.

Key results include:

• BrowseComp: 31.2 (vs. 7.4 Qwen3.5-9B-Instruct, +23.8 absolute improvement)
• GAIA: 61.5 (vs. 23.9 baseline, +37.6)
• HotpotQA: 93.4 (vs. 45.3, +48.1)

Performance is competitive with recent state-of-the-art open-source models and in certain metrics even approaches proprietary "foundation" research agents.

Figure 5: Increasing QA pool size (100K → 420K) consistently raises validation plateau, reduces format errors, and improves tool-use generalization, demonstrating strong data-scale benefits.

On the DeepSearch-Val behavioral evaluation split, DeepSearch-World-9B sustains significantly longer tool-usage trajectories (18.0 vs 4.7 turns), makes more evidence-gathering calls, and achieves a 70% advanced capability score on a multi-dimensional LLM rubric (vs 19% for Qwen3.5-9B-Instruct).

Figure 6: Tool-use analytics showing that DeepSearch-World sustains both longer turn counts and richer evidence aggregation than vanilla Qwen3.5-9B.

Ablation studies confirm that state internalization, reflection rewriting, and rigorous rejection sampling all contribute strong additive improvements to final performance, both in absolute accuracy and advanced reasoning dimensions.

Theoretical and Practical Implications

The results provide empirical evidence for several critical design hypotheses:

• Deterministic, verifiable environments are essential to enable stable, scalable self-distillation in long-horizon agentic settings, by systematically exposing agents to grounded, process-level supervision and sharp failure states.
• Evolving SFT, when paired with rigorous trajectory filtering rather than reliance on soft distributional matching (OPSD), provides robust and efficient learning signals that avoid instability arising from noisy rollouts and teacher drift.

Practically, the environment and framework design enable reproducible, cost-efficient, high-throughput agent training without live web access or dependence on externally generated synthetic demonstration pools. This imposes reproducibility and interpretability constraints critical for advanced tool-use evaluation. It also provides a scalable path to long-horizon agent improvement strictly via verified experience, sidestepping reliance on proprietary teacher policies.

Limitations and Future Directions

The current instantiation is restricted to Wikipedia as a knowledge substrate, limiting both domain diversity and environmental realism. Expanding the verifiable environment to other knowledge graphs, noisy web, or multimodal sources is a necessary next step for generalization. Moreover, while the evolving SFT update rule is shown to be robust, integration with advanced reinforcement learning objectives or hybrid distributional/distilled policy matching may further strengthen process-level skill acquisition, particularly in error recovery and high-level goal decomposition.

Conclusion

DeepSearch-World demonstrates that verifiable, deterministic environments can systematically address core bottlenecks in long-horizon, tool-using agent training via scalable self-distillation. Through rigorous environment design, scaffold process supervision, and stable evolving SFT, DeepSearch-Evolve enables agents to autonomously acquire transferable skills in planning, grounded reflection, error recovery, and evidence-based synthesis. These results set new standards for reproducible agent development, transfer learning, and controllable evaluation, and lay the groundwork for broader research in verifiable self-evolving artificial agents.