---
title: 'SynPlanResearch-R1: Deep-Research Agent Framework'
url: https://www.emergentmind.com/topics/synplanresearch-r1
type: topic
---

# SynPlanResearch-R1: Deep-Research Agent Framework

SynPlanResearch-R1 is a framework for training deep-research agents that use external tools to answer complex, knowledge-intensive questions through multi-turn interaction. It was introduced as a response to a specific failure mode in reinforcement learning from verifiable rewards (RLVR): agents trained end-to-end often terminate too early and overuse a single familiar tool, which limits evidence gathering and constrains multi-step exploration. The framework addresses this cold-start bottleneck by synthesizing plan-guided tool-use trajectories for supervised fine-tuning (SFT), then refining the resulting policy with RLVR using Group Relative Policy Optimization (GRPO). On seven multi-hop and open-web benchmarks, it improves average F1 to 0.580 on Qwen3-8B and 0.529 on Qwen3-4B, corresponding to relative gains of up to 6.0% and 5.8% over state-of-the-art baselines [2603.07853].

## 1. Research setting and motivation

SynPlanResearch-R1 is situated in the setting of research agents: large language models equipped with tools such as `web_search` and `crawl_webpage`. These systems must dynamically interleave internal reasoning with tool use in order to collect evidence from the web and synthesize answers to multi-hop or open-domain questions [2603.07853].

The framework is motivated by two exploration failures observed under pure RLVR. The first is **premature termination**, in which the agent makes only a small number of tool calls before producing a final answer. The second is **biased tool usage**, in which an agent in a multi-tool environment relies heavily on one tool while neglecting others that may provide richer information. Because RLVR is on-policy, weak initialization can trap the policy in these behaviors: the agent remains near its initial strategy and does not reliably discover longer or more varied tool-use sequences [2603.07853].

This formulation places exploration quality, rather than answer generation alone, at the center of training. A plausible implication is that deep-research performance depends not only on whether a model can call tools, but also on whether its initial policy encourages sufficiently broad search trajectories for RL to improve.

## 2. Framework architecture

SynPlanResearch-R1 has two stages. The first stage is **cold-start SFT with plan-guided synthetic data**. The second stage is **RLVR using GRPO** to further refine the policy [2603.07853].

The synthetic-data pipeline begins by sampling a random tool plan
$$
P=(a_1,\dots,a_L),
$$
where \(L\in[L_{\min},L_{\max}]\), \(a_1=\text{web\_search}\), and later actions are sampled uniformly from \(\{\text{web\_search},\text{crawl\_webpage}\}\). During rollout, the system injects action-specific cues such as “Let me think of the best query for web_search” at the start of each `<think>` block, thereby softly steering a large reasoning model to follow the sampled plan. The trajectory alternates among reasoning, tool calls, and tool responses for up to \(L\) steps, and then emits a final `<answer>` [2603.07853].

Only trajectories that satisfy strict filtering criteria are retained. The JSON format must be valid, and the final answer must either exactly match the gold answer or match it under F1-overlap. Each retained trajectory is then passed through a thought-rewriting step using Claude-3.7-Sonnet in order to remove cue artifacts while preserving the tool-use structure. The filtered and rewritten trajectories form the corpus \(\mathcal{D}_{\mathrm{sft}}\) used for cold-start imitation learning [2603.07853].

The supervised objective is
$$
L_{\text{sup}} = -\sum_{(q,y)\in\mathcal{D}_{\mathrm{sft}}} \log \pi_\theta(y\mid q).
$$
The paper explicitly notes that there is no separate exploration-bonus term in SFT; instead, exploration is shaped implicitly by conditioning on diverse plans and cue-guided trajectories [2603.07853].

## 3. Training procedure and optimization

The cold-start SFT stage initializes from a pre-trained backbone such as Qwen3-8B, fine-tunes for 2.5 epochs with learning rate \(5\times 10^{-6}\), and uses a maximum sequence length of 32K tokens. Synthetic data is produced by a fixed large reasoning model, Qwen3-32B. For each question, \(N=16\) plan-guided rollouts are generated using tool-plan lengths in \([3,8]\). The SFT corpus itself is built from 8K questions mixed from multi-hop QA, SuperGPQA, and WebWalker-Silver [2603.07853].

After SFT converges, the framework switches to RLVR with GRPO. For each prompt, the model samples \(G=16\) on-policy rollouts with temperature \(=1.0\) and top-\(p=1.0\). Rollouts allow a maximum of 8 tool-calling turns, input length 16K tokens, and response length 4K tokens. The reward function combines answer overlap with format validity: valid answers with positive F1 receive \(\text{F1}(y,\text{gold})+\alpha\), valid answers with zero F1 receive \(0\), and invalid-format trajectories or those with F1 below \(\tau\) receive \(-\alpha\), with \(\alpha=0.2\) and \(\tau=0.8\) [2603.07853].

The GRPO objective is given as
$$
\mathcal{L}_{\text{GRPO}}(\theta)
=
-\mathbb{E}_{(x,y)\sim\pi_{\theta_{\mathrm{old}}}}
\Bigg[
\sum_{t=1}^{|y|}
m_t
\min\big(\rho_t A_t,\,
\mathrm{clip}(\rho_t,1-\epsilon,1+\epsilon)A_t\big)
\Bigg],
$$
where \(\rho_t=\pi_\theta(y_t\mid c_t)/\pi_{\theta_{\mathrm{old}}}(y_t\mid c_t)\), \(A_t\) is the group-normalized advantage, and \(m_t\) masks tool-response tokens and void trajectories. RL is run for 80 steps at learning rate \(1\times10^{-6}\). Invalid or truncated trajectories are masked from the policy gradient, but their rewards are still used in group-relative advantage computation [2603.07853].

The paper notes that a joint objective of the form
$$
L(\theta)=L_{\mathrm{GRPO}}(\theta)+\lambda L_{\sup}(\theta)
$$
is possible in principle, but the reported experiments apply SFT and RL sequentially rather than jointly [2603.07853].

## 4. Benchmarks and empirical performance

Evaluation covers seven tasks spanning multi-hop QA and advanced open-web QA [2603.07853].

| Benchmark family | Tasks | Scale |
|---|---|---|
| Multi-Hop QA | HotpotQA, 2WikiMultihopQA, MuSiQue, Bamboogle | HotpotQA and 2WikiMultihopQA downsampled to 5K; MuSiQue and Bamboogle full |
| Advanced Open-Web QA | GPQA, WebWalkerQA, GAIA | As reported |

The model backbones are Qwen3-8B and Qwen3-4B. Baselines include Direct Inference, RAG, Search-o1, Rejection Sampling, Search-R1, and SimpleDeepSearcher. On the seven-benchmark average, SynPlanResearch-R1 attains 0.580 F1 on Qwen3-8B, compared with 0.547 for SimpleDeepSearcher, and 0.529 F1 on Qwen3-4B, compared with 0.500. These correspond to relative gains of 6.0% and 5.8%, respectively. On multi-hop QA alone, the reported improvement reaches up to +5.1% absolute; on advanced QA, up to +8.7% [2603.07853].

These results are presented as evidence that cold-start exploration shaping matters at both model scales. The reported gains are not confined to a single dataset family, but extend across both canonical multi-hop QA and more demanding open-web settings.

## 5. Tool-use behavior, ablations, and training dynamics

A central empirical claim of SynPlanResearch-R1 is that improved answers are associated with deeper exploration rather than merely with longer outputs. Tracking checkpoints at 20, 40, 60, and 80 RL steps shows that the framework maintains both higher average tool calls and higher scores than baselines, which the paper interprets as confirmation that deeper exploration leads to better answers [2603.07853].

Training dynamics further support this view. The framework retains higher policy entropy throughout RL, enabling continued exploration. Although its early reward is lower, it climbs to a higher plateau, whereas naïve RL from a shallow SFT baseline converges prematurely. This makes exploration quality a measurable part of optimization, not simply a by-product [2603.07853].

The ablation results isolate several components. Removing cue injection drops multi-hop QA by approximately 0.31 and GAIA by approximately 0.38. Restricting the system to `web_search` only degrades GAIA by 0.80, highlighting the importance of `crawl_webpage`. Shortening the maximum tool budget or lowering rollout temperature also hurts performance significantly [2603.07853].

Pre-training statistics on tool-use structure make the same point numerically. Standard ReAct yields 1.40 tool calls per question with no plan adherence. Adding a tool plan raises this to 2.05 tool calls with 28.3% plan adherence. Adding both tool plan and cue injection yields 4.36 tool calls with 76.9% plan adherence. This suggests that the synthetic-planning mechanism primarily acts by changing the agent’s exploration prior before RL begins [2603.07853].

## 6. Relation to AI-powered research planning

SynPlanResearch-R1 belongs to a broader line of work that treats planning as a first-class problem for scientific or research-oriented LLM systems, but it targets a different object of planning than several adjacent efforts.

“Idea2Plan: Exploring AI-Powered Research Planning” defines research planning as the transformation from a concise scientific idea into a five-section structured research plan, and introduces Idea2Plan Bench and Idea2Plan JudgeEval to measure that capability [2510.24891]. In that setting, the output is a document-like plan containing an introduction, key literatures, methods, experimental design, and resources, compliance, and ethical considerations. By contrast, SynPlanResearch-R1 operates at the level of **tool-use trajectories** for answering multi-hop and open-web questions rather than at the level of abstract research proposals [2603.07853].

“Planner-Centric Reinforcement Learning for Deep Research with Structure-Aware Reward” proposes DecomposeR, which represents research plans as typed directed acyclic graphs and assigns explicit planner rewards to graph structure, search quality, and synthesis behavior [2605.30824]. SynPlanResearch-R1 differs in its mechanism: instead of rewarding an explicit plan object such as a typed DAG, it synthesizes trajectories that encourage longer and more diverse sequences of tool calls and uses those trajectories to shape the initialization for RL [2603.07853].

Taken together, these works outline distinct but complementary formulations of research planning. Idea2Plan evaluates the transition from idea to executable plan [2510.24891]; DecomposeR makes planning explicit and rewardable through graph structure [2605.30824]; SynPlanResearch-R1 addresses exploration failure in tool-using research agents through synthetic plan-guided imitation and subsequent RL refinement [2603.07853].

## 7. Limitations and future directions

The paper’s analysis emphasizes that RLVR alone yields only marginal gains over imitation learning when the starting policy explores poorly. SynPlanResearch-R1 is therefore presented not as a replacement for RL, but as a mechanism for improving the initial exploration distribution that RL can subsequently refine [2603.07853].

The reported future directions remain closely tied to that premise. They include scaling to a greater variety of tools such as databases and calculators, automating cue and plan generation, integrating off-policy RL methods to better exploit synthetic trajectories, and applying the framework to non-QA domains including code synthesis, data analysis, and multimodal research [2603.07853]. A plausible implication is that the framework’s main contribution is methodological rather than domain-specific: it provides a way to control exploration structure before on-policy RL begins.

The code and data are publicly available at the project repository, reflecting the framework’s emphasis on reproducible agent-training pipelines [2603.07853].

Source: https://www.emergentmind.com/topics/synplanresearch-r1