---
title: 'CoSearch: Joint RL for Agentic Search'
url: https://www.emergentmind.com/topics/cosearch
type: topic
---

# CoSearch: Joint RL for Agentic Search

CoSearch is a reinforcement-learning framework for **agentic search** in which a multi-step reasoning agent and a generative document ranker are trained jointly rather than coupling a learned reasoner to a fixed retrieval tool. In the formulation introduced in "CoSearch: Joint Training of Reasoning and Document Ranking via Reinforcement Learning for Agentic Search" [2604.17555], the agent follows a ReAct-style loop of thought generation, sub-query issuance, document observation, and answer synthesis. The framework is motivated by the claim that retrieval quality is a principal bottleneck in iterative search systems, and it addresses that bottleneck through joint optimization, a semantic grouping strategy for GRPO-based ranker training, and a composite reward that combines ranking signals with trajectory-level answer outcomes.

## 1. Problem setting and motivation

CoSearch studies agentic search as a sequential decision process in which a system reasons, searches, and then reasons again. The trajectory is written as
$$
y = \bigl(q_0,\; \tau_1, q_1, o_1,\; \tau_2, q_2, o_2,\; \dots,\; \tau_m, q_m, o_m,\; \tau_{m+1}, a\bigr),
$$
where \(q_0\) is the user query, \(\tau_t\) is the thought at step \(t\), \(q_t\) is the sub-query, \(o_t\) is the retrieved observation, and \(a\) is the final answer [2604.17555].

The central criticism of earlier RL-based agentic search systems is that they optimize only the reasoning policy while treating retrieval as immutable. CoSearch quantifies this limitation through an **oracle retrieval** experiment in which documents containing the gold answer are promoted to top positions if they are already present in the candidate set. Under this intervention, a **7B main agent** improves average F1 from **0.546** to **0.626**, a **+14.7%** relative gain, while a **3B main agent** improves from **0.440** to **0.558**, a **+26.8%** relative gain. These figures are used to argue that retrieval is not merely auxiliary infrastructure but an upper-bound constraint on end-task performance.

A common misconception is that better reasoning alone is sufficient to scale agentic QA. CoSearch directly contests that view. Its motivating evidence implies that weaker agents are especially sensitive to retrieval quality, because the relative gain under oracle retrieval is larger for the 3B main agent than for the 7B main agent.

## 2. Retrieval architecture and joint objective

CoSearch decomposes retrieval into a fixed first-stage recall component and a trainable second-stage ranking component [2604.17555]. At each search step, a **fixed dense retriever** recalls top-\(N\) candidate passages \(\mathcal{D}_N\), and a **generative ranker** selects the top-\(K\) documents \(\mathcal{D}_K\) that are shown to the reasoning agent. The observation is therefore
$$
o_t = \mathcal{D}_K.
$$

The two-stage procedure is specified as:
$$
\mathcal{D}_N \leftarrow \text{DenseRetrieve}(q_t,\mathcal{C},N),
$$
$$
\text{prompt} \leftarrow \text{BuildPrompt}(q_0,q_t,\mathcal{D}_N,K),
$$
$$
\text{output} \leftarrow \pi_{\theta_{\text{gr}}}(\text{prompt}),
$$
$$
\mathcal{D}_K \leftarrow \text{ParseRanking}(\text{output},\mathcal{D}_N,K).
$$

The joint training target is
$$
\max_{\theta_{\text{main}},\,\theta_{\text{gr}}}\;\mathbb{E}_{T\sim \pi_{\theta_{\text{main}}},\,\pi_{\theta_{\text{gr}}}}\bigl[r(T)\bigr],
$$
so both the main reasoning policy and the ranking policy are optimized with respect to downstream answer quality rather than with separate, frozen objectives.

The reported implementation uses a fixed **E5 base** dense retriever over the **2018 Wikipedia dump**, with **\(N=50\)** recalled candidate passages and **\(K=5\)** selected passages. Main-agent backbones are **Qwen2.5-7B-Instruct** and **Qwen2.5-3B-Instruct**, and the ranker backbone is **Qwen2.5-7B-Instruct**. The design therefore does not train retrieval end-to-end from the raw corpus; instead, it treats dense recall as fixed and makes the second-stage ranker the trainable retrieval component.

## 3. GRPO training and semantic grouping

Both the reasoning agent and the ranker are trained with **Group Relative Policy Optimization (GRPO)** [2604.17555]. For the main agent, GRPO is straightforward because multiple trajectories can be sampled from the same initial query. The difficulty arises for the ranker: its prompts depend on the sub-queries generated along the evolving reasoning trajectories, so different ranker calls do not naturally form valid optimization groups.

CoSearch addresses this with a **semantic grouping strategy**. For a fixed initial query \(q_0\), all ranker calls across sampled trajectories are pooled:
$$
\mathcal{Q}(q_0)=\{q_t^{(i)} \mid i\in[1,G],\; t\in[1,T_i]\}.
$$
Sub-queries are then greedily clustered by **token-level F1 similarity**. A query \(q\) is assigned to a group \(g_m\) if its token F1 with the group representative exceeds a threshold:
$$
q \in g_m \iff \text{F1}_{\text{token}}(q, q_m^{\text{rep}})\ge \delta.
$$

The reported grouping procedure has two levels. First, ranker calls are separated into an **easy** split and a **hard** split according to whether the candidate set already contains a gold-answer document. Second, clustering is performed within each split using token-level F1. Groups smaller than \(k_{\min}\) are discarded to stabilize advantage estimation. The hyperparameters are **\(\delta = 0.8\)** and **\(k_{\min} = 3\)**.

This component is the key technical device that makes listwise RL for the ranker tractable without extra rollouts. The ablation results identify it as the most important design choice among those tested: removing semantic grouping lowers average F1 from **0.568** to **0.549**.

## 4. Composite reward and credit assignment

CoSearch defines ranker learning as a credit-assignment problem in which neither pure answer reward nor pure relevance supervision is sufficient [2604.17555]. A trajectory can end with a wrong answer despite good ranking, and a correct answer can occasionally be produced despite weak ranking. Conversely, documents that do not directly contain the final answer can still be useful for decomposition or intermediate reasoning.

To address this, the framework combines three signals.

The **relevance reward** uses pseudo-relevant documents defined as those containing the gold answer. It is based on Hit@\(\!k\):
$$
r_{\text{rel}} = \frac{1}{|\mathcal{K}|}\sum_{k\in\mathcal{K}}\text{Hit@}k,
$$
with \(\mathcal{K}=\{1,3,5\}\).

The **main-agent reward** uses token-level F1 for valid ReAct trajectories and a fixed penalty otherwise:
$$
r_{\text{main}}(q_0, y_i)=
\begin{cases}
s_{\text{ans}}, & f=1,\\
-\alpha, & f=0,
\end{cases}
$$
where \(f\in\{0,1\}\) indicates whether the trajectory follows the required format and **\(\alpha=0.2\)** is the format penalty.

The **final ranker reward** is conditional. If the format is invalid, the penalty is applied. If an answer-bearing document exists in the candidate set but ranking quality is poor, the reward is driven mainly by \(r_{\text{rel}}\). If ranking quality exceeds a threshold, the reward becomes \(r_{\text{rel}} + r_{\text{main}}\). If no answer-bearing document exists, the reward falls back to \(r_{\text{main}}\). The threshold is **\(\gamma = 0.5\)**.

The reward design is deliberately rule-based and does not require LLM judges. In ablations, removing \(r_{\text{main}}\) lowers average F1 from **0.568** to **0.560**; removing \(r_{\text{rel}}\) lowers it further to **0.552**. Replacing Hit@\(\!k\) with nDCG@\(\!k\) yields **0.556**, and replacing the composite reward with an **LLM-as-judge** signal yields **0.558**. These results indicate that both immediate ranking supervision and long-horizon answer feedback are necessary, with ranking-quality reward slightly more influential in the reported setting.

## 5. Benchmarks, baselines, and empirical results

CoSearch is evaluated on **seven QA benchmarks**: **PopQA**, **Natural Questions (NQ)**, and **TriviaQA** for single-hop QA, and **HotpotQA**, **2WikiMultiHopQA**, **Musique**, and **Bamboogle** for multi-hop QA [2604.17555]. Evaluation uses token-level F1. The RL training set contains **51,200 questions**, drawn from **NQ: 20,480**, **HotpotQA: 14,220**, **Musique: 9,000**, and **2WikiMultiHopQA: 7,500**. Training uses **\(G=8\)** rollouts per query, up to **6 search calls** per trajectory, learning rate **\(1\times 10^{-6}\)**, rollout batch size **512**, effective batch size **128**, and sampling temperature **1.0**.

The main comparison set includes **Direct Inference**, **CoT**, **RAG**, **Search-o1**, **Search-R1**, **ZeroSearch**, **Retrieval Only**, and **Fixed Ranker**. The **Fixed Ranker** baseline is trained on **92,160 query examples** constructed from **MS MARCO** and **Natural Questions**, using the same listwise generative format and the same Hit@\(\{1,3,5\}\) reward.

| System | 7B average F1 | 3B average F1 |
|---|---:|---:|
| Search-R1 | 0.533 | 0.428 |
| Retrieval Only | 0.546 | 0.440 |
| Fixed Ranker | 0.553 | 0.460 |
| CoSearch | 0.568 | 0.471 |

For the **7B main agent**, CoSearch improves average F1 by **+6.6% relative** over **Search-R1**, **+4.0%** over **Retrieval Only**, and **+2.7%** over **Fixed Ranker**. For the **3B main agent**, it improves from **0.428** under **Search-R1** to **0.471**. The reported qualitative summary is that CoSearch is best on all seven benchmarks or tied at the top depending on rounding, with especially strong gains on harder multi-hop tasks.

Additional analyses connect ranking behavior to end-task performance. Improved ranking quality, measured by **Hit@5**, tracks higher downstream F1 during training. CoSearch also reduces the average number of retrieval turns, with about **1.67** turns versus **2.62** for the fixed reranker. Sensitivity to the number of retained documents \(K\) is mild: **\(K=1\)** yields **0.538**, **\(K=3\)** yields **0.561**, **\(K=5\)** yields **0.568**, and **\(K=10\)** yields **0.571**, making **\(K=5\)** a practical trade-off in the reported system.

## 6. Scope, limitations, and related uses of the name

CoSearch, in this usage, denotes a framework for **jointly training reasoning and document ranking in agentic QA** rather than a general term for collaborative or code-oriented retrieval [2604.17555]. The name is adjacent to several distinct systems in the literature. **"CO-Search"** is a COVID-19 scientific search engine that combines SBERT semantic retrieval, TF-IDF, BM25 rank fusion, QA, and abstractive summarization for TREC-COVID [2006.09595]. **"CoSearchAgent"** is a Slack-based lightweight collaborative search agent that rewrites context-dependent multi-user queries, issues web searches, and produces grounded answers with citations [2402.06360]. In code retrieval, recent work such as **CoREB** emphasizes multitask retrieve-then-rerank evaluation for developer-style code search rather than RL-trained reasoning agents [2605.04615].

The reported results also indicate several practical constraints of CoSearch as an agentic-search method. The framework still depends on a **fixed first-stage dense retriever**, so retrieval is not fully optimized from corpus access onward. Semantic grouping relies on **heuristic token-F1 similarity** and a minimum-group-size rule. Reward construction depends on **pseudo-relevant documents** defined by answer containment, which is an imperfect proxy for broader evidential utility. On the modeling side, a **3B ranker** is reported to have **diverged**, a **shared-backbone** configuration underperformed the default separate-backbone setup with **0.559**, and a **cross-encoder ranker** scored **0.551**, suggesting that direct RL listwise optimization is sensitive to ranker capacity and architectural separation.

A plausible implication is that CoSearch represents a broader methodological shift in search research: retrieval is treated not as a static subsystem, but as a trainable component whose behavior co-evolves with reasoning. Within the agentic-search setting examined here, that shift is supported by both the oracle-retrieval gap and the consistent gains over fixed-ranker and retrieval-only baselines.

Source: https://www.emergentmind.com/topics/cosearch