CoSearch: Joint RL for Agentic Search

• CoSearch is a reinforcement-learning framework that jointly trains a multi-step reasoning agent and a generative document ranker to optimize agentic search.
• It introduces semantic grouping for GRPO-based ranking and a composite reward signal that blends immediate ranking and long-term answer quality.
• Empirical results show that CoSearch improves average F1 scores and reduces retrieval turns, with significant gains on multi-hop QA benchmarks using both 7B and 3B agents.

CoSearch is a reinforcement-learning framework for agentic search in which a multi-step reasoning agent and a generative document ranker are trained jointly rather than coupling a learned reasoner to a fixed retrieval tool. In the formulation introduced in "CoSearch: Joint Training of Reasoning and Document Ranking via Reinforcement Learning for Agentic Search" (Zeng et al., 19 Apr 2026), the agent follows a ReAct-style loop of thought generation, sub-query issuance, document observation, and answer synthesis. The framework is motivated by the claim that retrieval quality is a principal bottleneck in iterative search systems, and it addresses that bottleneck through joint optimization, a semantic grouping strategy for GRPO-based ranker training, and a composite reward that combines ranking signals with trajectory-level answer outcomes.

1. Problem setting and motivation

CoSearch studies agentic search as a sequential decision process in which a system reasons, searches, and then reasons again. The trajectory is written as

$$y = \bigl(q_0,\; \tau_1, q_1, o_1,\; \tau_2, q_2, o_2,\; \dots,\; \tau_m, q_m, o_m,\; \tau_{m+1}, a\bigr),$$

where $q_0$ is the user query, $\tau_t$ is the thought at step $t$, $q_t$ is the sub-query, $o_t$ is the retrieved observation, and $a$ is the final answer (Zeng et al., 19 Apr 2026).

The central criticism of earlier RL-based agentic search systems is that they optimize only the reasoning policy while treating retrieval as immutable. CoSearch quantifies this limitation through an oracle retrieval experiment in which documents containing the gold answer are promoted to top positions if they are already present in the candidate set. Under this intervention, a 7B main agent improves average F1 from 0.546 to 0.626, a +14.7% relative gain, while a 3B main agent improves from 0.440 to 0.558, a +26.8% relative gain. These figures are used to argue that retrieval is not merely auxiliary infrastructure but an upper-bound constraint on end-task performance.

A common misconception is that better reasoning alone is sufficient to scale agentic QA. CoSearch directly contests that view. Its motivating evidence implies that weaker agents are especially sensitive to retrieval quality, because the relative gain under oracle retrieval is larger for the 3B main agent than for the 7B main agent.

2. Retrieval architecture and joint objective

CoSearch decomposes retrieval into a fixed first-stage recall component and a trainable second-stage ranking component (Zeng et al., 19 Apr 2026). At each search step, a fixed dense retriever recalls top-$N$ candidate passages $\mathcal{D}_N$, and a generative ranker selects the top-$K$ documents $q_0$0 that are shown to the reasoning agent. The observation is therefore

$q_0$1

The two-stage procedure is specified as:

$q_0$2

$q_0$3

$q_0$4

$q_0$5

The joint training target is

$q_0$6

so both the main reasoning policy and the ranking policy are optimized with respect to downstream answer quality rather than with separate, frozen objectives.

The reported implementation uses a fixed E5 base dense retriever over the 2018 Wikipedia dump, with $q_0$7 recalled candidate passages and $q_0$8 selected passages. Main-agent backbones are Qwen2.5-7B-Instruct and Qwen2.5-3B-Instruct, and the ranker backbone is Qwen2.5-7B-Instruct. The design therefore does not train retrieval end-to-end from the raw corpus; instead, it treats dense recall as fixed and makes the second-stage ranker the trainable retrieval component.

3. GRPO training and semantic grouping

Both the reasoning agent and the ranker are trained with Group Relative Policy Optimization (GRPO) (Zeng et al., 19 Apr 2026). For the main agent, GRPO is straightforward because multiple trajectories can be sampled from the same initial query. The difficulty arises for the ranker: its prompts depend on the sub-queries generated along the evolving reasoning trajectories, so different ranker calls do not naturally form valid optimization groups.

CoSearch addresses this with a semantic grouping strategy. For a fixed initial query $q_0$9, all ranker calls across sampled trajectories are pooled:

$\tau_t$0

Sub-queries are then greedily clustered by token-level F1 similarity. A query $\tau_t$1 is assigned to a group $\tau_t$2 if its token F1 with the group representative exceeds a threshold:

$\tau_t$3

The reported grouping procedure has two levels. First, ranker calls are separated into an easy split and a hard split according to whether the candidate set already contains a gold-answer document. Second, clustering is performed within each split using token-level F1. Groups smaller than $\tau_t$4 are discarded to stabilize advantage estimation. The hyperparameters are $\tau_t$5 and $\tau_t$6.

This component is the key technical device that makes listwise RL for the ranker tractable without extra rollouts. The ablation results identify it as the most important design choice among those tested: removing semantic grouping lowers average F1 from 0.568 to 0.549.

4. Composite reward and credit assignment

CoSearch defines ranker learning as a credit-assignment problem in which neither pure answer reward nor pure relevance supervision is sufficient (Zeng et al., 19 Apr 2026). A trajectory can end with a wrong answer despite good ranking, and a correct answer can occasionally be produced despite weak ranking. Conversely, documents that do not directly contain the final answer can still be useful for decomposition or intermediate reasoning.

To address this, the framework combines three signals.

The relevance reward uses pseudo-relevant documents defined as those containing the gold answer. It is based on Hit@$\tau_t$7:

$\tau_t$8

with $\tau_t$9.

The main-agent reward uses token-level F1 for valid ReAct trajectories and a fixed penalty otherwise:

$t$0

where $t$1 indicates whether the trajectory follows the required format and $t$2 is the format penalty.

The final ranker reward is conditional. If the format is invalid, the penalty is applied. If an answer-bearing document exists in the candidate set but ranking quality is poor, the reward is driven mainly by $t$3. If ranking quality exceeds a threshold, the reward becomes $t$4. If no answer-bearing document exists, the reward falls back to $t$5. The threshold is $t$6.

The reward design is deliberately rule-based and does not require LLM judges. In ablations, removing $t$7 lowers average F1 from 0.568 to 0.560; removing $t$8 lowers it further to 0.552. Replacing Hit@$t$9 with nDCG@$q_t$0 yields 0.556, and replacing the composite reward with an LLM-as-judge signal yields 0.558. These results indicate that both immediate ranking supervision and long-horizon answer feedback are necessary, with ranking-quality reward slightly more influential in the reported setting.

5. Benchmarks, baselines, and empirical results

CoSearch is evaluated on seven QA benchmarks: PopQA, Natural Questions (NQ), and TriviaQA for single-hop QA, and HotpotQA, 2WikiMultiHopQA, Musique, and Bamboogle for multi-hop QA (Zeng et al., 19 Apr 2026). Evaluation uses token-level F1. The RL training set contains 51,200 questions, drawn from NQ: 20,480, HotpotQA: 14,220, Musique: 9,000, and 2WikiMultiHopQA: 7,500. Training uses $q_t$1 rollouts per query, up to 6 search calls per trajectory, learning rate $q_t$2, rollout batch size 512, effective batch size 128, and sampling temperature 1.0.

The main comparison set includes Direct Inference, CoT, RAG, Search-o1, Search-R1, ZeroSearch, Retrieval Only, and Fixed Ranker. The Fixed Ranker baseline is trained on 92,160 query examples constructed from MS MARCO and Natural Questions, using the same listwise generative format and the same Hit@$q_t$3 reward.

System	7B average F1	3B average F1
Search-R1	0.533	0.428
Retrieval Only	0.546	0.440
Fixed Ranker	0.553	0.460
CoSearch	0.568	0.471

For the 7B main agent, CoSearch improves average F1 by +6.6% relative over Search-R1, +4.0% over Retrieval Only, and +2.7% over Fixed Ranker. For the 3B main agent, it improves from 0.428 under Search-R1 to 0.471. The reported qualitative summary is that CoSearch is best on all seven benchmarks or tied at the top depending on rounding, with especially strong gains on harder multi-hop tasks.

Additional analyses connect ranking behavior to end-task performance. Improved ranking quality, measured by Hit@5, tracks higher downstream F1 during training. CoSearch also reduces the average number of retrieval turns, with about 1.67 turns versus 2.62 for the fixed reranker. Sensitivity to the number of retained documents $q_t$4 is mild: $q_t$5 yields 0.538, $q_t$6 yields 0.561, $q_t$7 yields 0.568, and $q_t$8 yields 0.571, making $q_t$9 a practical trade-off in the reported system.

6. Scope, limitations, and related uses of the name

CoSearch, in this usage, denotes a framework for jointly training reasoning and document ranking in agentic QA rather than a general term for collaborative or code-oriented retrieval (Zeng et al., 19 Apr 2026). The name is adjacent to several distinct systems in the literature. "CO-Search" is a COVID-19 scientific search engine that combines SBERT semantic retrieval, TF-IDF, BM25 rank fusion, QA, and abstractive summarization for TREC-COVID (Esteva et al., 2020). "CoSearchAgent" is a Slack-based lightweight collaborative search agent that rewrites context-dependent multi-user queries, issues web searches, and produces grounded answers with citations (Gong et al., 2024). In code retrieval, recent work such as CoREB emphasizes multitask retrieve-then-rerank evaluation for developer-style code search rather than RL-trained reasoning agents (Xue et al., 6 May 2026).

The reported results also indicate several practical constraints of CoSearch as an agentic-search method. The framework still depends on a fixed first-stage dense retriever, so retrieval is not fully optimized from corpus access onward. Semantic grouping relies on heuristic token-F1 similarity and a minimum-group-size rule. Reward construction depends on pseudo-relevant documents defined by answer containment, which is an imperfect proxy for broader evidential utility. On the modeling side, a 3B ranker is reported to have diverged, a shared-backbone configuration underperformed the default separate-backbone setup with 0.559, and a cross-encoder ranker scored 0.551, suggesting that direct RL listwise optimization is sensitive to ranker capacity and architectural separation.

A plausible implication is that CoSearch represents a broader methodological shift in search research: retrieval is treated not as a static subsystem, but as a trainable component whose behavior co-evolves with reasoning. Within the agentic-search setting examined here, that shift is supported by both the oracle-retrieval gap and the consistent gains over fixed-ranker and retrieval-only baselines.