BrowseComp-Plus Benchmark

• BrowseComp-Plus is a large-scale, human-verified benchmark designed to assess deep-research agents that integrate LLMs with search tools.
• It employs a fixed, curated corpus of over 100,000 documents to enable reproducible experiments and clear separation of retrieval and reasoning capabilities.
• The benchmark utilizes rigorous evaluation metrics and ablation studies, offering actionable insights to improve the design and integration of deep-research systems.

BrowseComp-Plus is a large-scale, human-verified evaluation benchmark specifically designed for deep-research agents—systems that integrate LLMs with search tools to answer complex, multi-step queries through iterative retrieval and reasoning. In contrast to prior benchmarks such as BrowseComp, which relied on black-box, live web APIs and uncontrolled document corpora, BrowseComp-Plus introduces a fixed, curated corpus and standardized evaluation protocol. This enables reproducibility, fair comparison across systems, and fine-grained disentanglement of retrieval versus reasoning capabilities, providing the research community with a transparent and extensible testbed for deep-research systems (Chen et al., 8 Aug 2025).

1. Motivation and Conceptual Advances

BrowseComp-Plus was developed to address three central deficiencies in earlier deep-research agent evaluation frameworks:

• Fairness: Dynamic, opaque web APIs caused irreproducible results—each system operated on a shifting corpus, precluding causal separation of retrieval and reasoning effectiveness.
• Transparency: The lack of control over document content precluded targeted analysis of search tools and hampered diagnosis of system failures.
• Accessibility: Reliance on paid/commercial APIs increased barriers to experimentation and introduced usage-based variability.

BrowseComp-Plus grounds all evaluation in a fixed corpus of approximately 100,000 web pages. Each of its 830 complex queries is paired with a unique set of human-curated positive (evidence) documents and a set of challenging, “hard negative” distractors. This design allows for controlled, reproducible experiments with the ability to isolate the effects of LLM reasoning, retrieval algorithms, and context engineering strategies.

2. Dataset Construction and Curation

The dataset construction pipeline for BrowseComp-Plus comprises multiple stages:

2.1 Fixed Document Corpus

A two-phase approach was used to create a self-contained, deduplicated corpus of 100,195 documents:

1. Evidence Gathering: For each BrowseComp query, an LLM with web access (OpenAI o3) was tasked to propose URLs and snippets (“clues”) potentially justifying the answer. Only reachable and valid pages were retained (restricting the dataset from 1,266 to 1,005 candidates).
2. Human Verification: Fourteen trained annotators invested over 400 hours verifying that each clue was supported by a labeled document span, and that the union of clues sufficed to answer the question. Annotators flagged “gold documents” containing the answer. Queries lacking verifiable evidence were removed, yielding 830 final queries with >80% inter-annotator agreement.

2.2 Hard Negative Mining

To stress retrieval models, each query was decomposed into subqueries (mean: 7 per question) via GPT-4o. Each was submitted to a search API (SerpAPI), retrieving up to 100 high-rank documents. After deduplication, each query contained, on average: 6.1 positive, 76.3 hard negatives, and 2.9 gold documents.

3. Evaluation Metrics and Protocol

BrowseComp-Plus employs both agent-level and retrieval-level metrics to support multi-faceted system assessment:

3.1 End-to-End Agent Metrics

• Accuracy: Fraction of queries for which the agent’s final answer exactly matches the annotated ground truth:

$$\mathrm{Accuracy} = \frac{\# \text{correct answers}}{\# \text{total queries}}$$

• Recall (Document Coverage): Fraction of human-verified evidence documents retrieved at any search step.
• Search Calls: Mean number of retriever tool invocations per query.
• Calibration Error: Difference between self-reported model confidence and empirical accuracy.

3.2 Retrieval Effectiveness (Cranfield Paradigm)

• Recall@k: Fraction of positive (evidence) or gold documents retrieved in top $k$ results.
• nDCG@k: Normalized discounted cumulative gain at $k$, quantifying ranking quality.
• Additional IR metrics: Precision@k, mean average precision (MAP) as formally defined.

3.3 Protocol Details

Agents use a reasoning + retrieval loop, outputting an explanation, exact answer, and confidence. Retriever responses are truncated to the first 512 tokens. Agents are evaluated by sweeping combinations of front-end LLMs and backend retrievers.

4. Retrieval and Agent Architectures

BrowseComp-Plus supports a wide benchmarking grid:

• Retrievers: BM25 (Pyserini baseline), Qwen3-Embedding (dense; 0.6B, 4B, 8B via Tevatron), and ReasonIR-8B (dense, reasoning-oriented).
• Agents: 12 LLM-based search agents, ranging from open-source (Qwen3-32B, SearchR1-32B, gpt-oss-20B/120B) to closed (OpenAI gpt-4.1, o3, gpt-5; Anthropic Sonnet-4, Opus-4; Google Gemini-2.5 Flash/Pro).

5. Experimental Results and Key Findings

5.1 End-to-End Agent Performance

The following table summarizes representative accuracy results:

Agent/Retriever	Accuracy	Recall	Search Calls
SearchR1-32B + BM25	3.86%	2.61%	1.78
GPT-5 + BM25	55.90%	61.70%	23.23
GPT-5 + Qwen3-8B	70.12%	78.98%	21.74

Dense retrievers sharply outperform BM25, and agent-retriever synergy drives gains. The leading agent-retriever pair (GPT-5 + Qwen3-8B) achieves 70.12% accuracy, with corresponding jumps in recall and even fewer average search calls.

5.2 Retriever Effectiveness

Direct evidence retrieval (Recall@5, nDCG@10):

Retriever	Recall@5	nDCG@10
BM25	1.2%	1.6
Qwen3-Embed-8B	14.5%	20.3
ReasonIR-8B	12.2%	16.8

Dense retrievers provide order-of-magnitude improvements in recall and ranking quality over lexical baselines.

5.3 Context Engineering and Ablation

• Oracle retrieval (all positive docs as context): gpt-4.1 achieves 93.49% accuracy, Qwen3-32B 83.25%, demonstrating that retrieval—not reasoning—is the principal bottleneck.
• Full-document reading: Enabling access to complete documents (not just previews) via the “get-doc” tool boosts accuracy for gpt-4.1 from 35.42% to 43.61%.
• Deliberation modes: Raising “reasoning effort” in gpt-oss models increases accuracy by 20–30 points at a cost of 10–20× more searches, underscoring an inherent trade-off.

5.4 Citation and Attribution

Better retrievers (Qwen3-8B) yield higher precision and recall for evidence citations, with GPT-5 citing 83.4% of final answers from verified documents, up from 71.8% for BM25.

6. Implications for Agent Design and Future Directions

BrowseComp-Plus quantifies several core phenomena in current deep-research pipelines:

• Retriever quality is the dominant constraint on system accuracy, as evidenced by dramatic jumps when swapping in high-recall dense retrievers.
• Open-source LLM agents lag in tool orchestration and search integration, even when reasoning skills, tested in the oracle context, are competitive.
• Design of context encoding (e.g., full document access) and algorithmic control over reasoning depth can yield significant improvements, motivating continued research into context engineering and search pruning.

BrowseComp-Plus sets the stage for work on co-optimization of retriever–reasoner stacks, federated/multi-tool retrieval, and explicit study of retrieval generalization.

7. Position within the Evaluation Landscape

BrowseComp-Plus defines a new standard for research in deep-research agent evaluation. Its methodology ensures transparency (fixed corpus, human-verified evidence, hard negatives), fairness (identical evaluation targets for all systems), and extensibility (released queries, qrels, and scripts). The MM-BrowseComp benchmark extends this paradigm to the multimodal domain, requiring agents to reason over images and videos, and further exposing limitations of current architectures (Li et al., 14 Aug 2025).

By enabling systematic, repeatable ablations and providing dense, citation-level ground truth, BrowseComp-Plus facilitates diagnosis of retrieval, context, and reasoning weaknesses while fostering fair competition and progress in agent design. The benchmark’s construction and analytic framework are pivotal for advancing understanding and development of LLM-driven research systems (Chen et al., 8 Aug 2025).