BrowseComp+ Benchmark

• BrowseComp+ is a fixed-corpus benchmark that replaces dynamic web APIs with a curated 100K-document set and human-verified evidence for reproducible deep-research evaluations.
• It isolates key components such as retriever performance, citation fidelity, and reasoning quality through measurable metrics on 830 annotated queries.
• The framework supports detailed diagnostic analyses of retrieval ceilings, context engineering effects, and agent-specific sensitivities to guide future improvement.

BrowseComp-Plus, often abbreviated BrowseComp+, is a fixed-corpus, component-wise extension of the original BrowseComp benchmark for web-browsing LLM agents. It was introduced to address two limitations of evaluations built on black-box live web search APIs: fairness, because dynamic and opaque web APIs hinder fair comparisons and reproducibility, and transparency, because lack of control over the document corpus makes it difficult to isolate retriever contributions. BrowseComp-Plus replaces opaque, dynamic web APIs with a static, curated 100 K-document corpus and annotates each of 830 queries with human-verified supporting documents, explicitly labeled gold answer documents, and mined hard negatives, thereby enabling controlled experimentation on retrievers, deep-research agents, citation behavior, and context engineering (Chen et al., 8 Aug 2025).

1. Origins and motivating problem formulation

BrowseComp-Plus is derived from BrowseComp, a benchmark introduced as a simple yet challenging test of whether agents can browse the web to answer short, easily verifiable questions that nevertheless require persistence, creativity, and multi-hop evidence gathering. The original BrowseComp comprises 1,266 questions and was designed around “needle-in-haystack” facts that are hard to find but easy to grade once found (Wei et al., 16 Apr 2025). BrowseComp-Plus inherits that problem setting, but changes the evaluation substrate.

The central motivation for BrowseComp-Plus is that live-web evaluation conflates multiple components of a deep-research system. In the original live-API regime, search results change over time, black-box ranking makes the retriever’s contribution impossible to isolate, and commercial APIs impose usage limits and fees. BrowseComp-Plus addresses these issues with a fixed, versioned document collection, human-verified positives, hard negatives, and open release of data, qrels, and evaluation scripts (Chen et al., 8 Aug 2025).

This design shifts the benchmark from evaluation of a complete browsing system “at a given time” toward controlled experimentation on underlying capabilities. In particular, it permits apples-to-apples comparison of agent architectures, standalone evaluation of retrievers under the Cranfield paradigm, and plug-and-play testing of LLM agents under fixed retrieval conditions. A plausible implication is that BrowseComp-Plus occupies a distinct niche: it preserves the long-horizon, evidence-seeking character of BrowseComp while making the evaluation environment inspectable and reproducible.

2. Corpus construction and annotation pipeline

BrowseComp-Plus constructs positive evidence documents through a multi-stage pipeline. First, OpenAI’s o3 model was prompted with question-answer pairs to propose “clues” and corresponding URLs. Second, those URLs were scraped via Selenium+Trafilatura, with malformed or unreachable pages excluded. Third, 14 annotators, over 400+ hours, verified that each document justifies its clue, labeled exact text spans, and flagged the subset of documents that contains the final answer, called “gold documents.” Finally, of the 1,266 original questions, 830 passed verification; the resulting benchmark has on average 6.1 positives per query and 2.9 gold docs per query (Chen et al., 8 Aug 2025).

Hard negatives are mined separately. GPT-4o decomposed each complex query into approximately 7 self-contained subqueries; each subquery was issued to SerpAPI with up to 100 results; pages that were scraped and were not labeled positive became hard negatives. These hard negatives average 76.3 per query (Chen et al., 8 Aug 2025).

The final corpus statistics are as follows:

Component	Value
Documents	100,195
Average document length	5179 words
Queries	830
Positives per query	6.1
Negatives per query	76.3
Gold docs per query	2.9

These choices make the benchmark unusually explicit about evidence structure. Human-verified supporting documents define the evidence set, gold answer documents identify answer-bearing support, and mined challenging negatives create controlled distractors. This permits disentangled analysis of retrieval and reasoning in a way that live-web settings cannot provide.

3. Evaluation formalism and controlled agent setup

BrowseComp-Plus defines end-to-end answer accuracy over $N$ test queries by letting $A_i \in \{0,1\}$ indicate whether the agent’s final answer is judged correct, and then computing

$\mathrm{Accuracy} = \frac1N \sum_{i=1}^N A_i.$

For retrieval, if $E_i$ is the set of human-verified evidence documents for query $i$ and $R_i(k)$ is the set retrieved in top-$k$, then evidence recall is

$$\mathrm{Recall@}k = \frac{1}{N}\sum_{i=1}^N \frac{\lvert R_i(k)\cap E_i\rvert}{|E_i|}.$$

The benchmark also uses standard nDCG:

$$\mathrm{DCG@}k = \sum_{j=1}^k \frac{2^{\mathrm{rel}_j}-1}{\log_2(j+1)}, \qquad \mathrm{nDCG@}k = \frac{\mathrm{DCG@}k}{\mathrm{IDCG@}k}.$$

For citation evaluation, if $C_i$ is the set of documents cited by the agent and $A_i \in \{0,1\}$0, then citation precision and citation recall are

$A_i \in \{0,1\}$1

The experimental setup uses the 100,195-document corpus and 830 BrowseComp-Plus queries. Example search agents include closed-source models such as gpt-4.1, o3, gpt-5, claude-opus-4, claude-sonnet-4, gemini-2.5-flash, and gemini-2.5-pro, together with open-source models such as Qwen3-32B, Search-R1-32B, and gpt-oss-20B/120B in low, medium, and high reasoning modes. Retrievers include BM25 via Pyserini, Qwen3-Embedding-{0.6B, 4B, 8B}, and ReasonIR-8B. Each search call returns top-5 documents with the first 512 tokens of each; agents may invoke search arbitrarily, and average calls range from approximately 1.8 for Search-R1 up to approximately 25 for o3 and gpt-5 (Chen et al., 8 Aug 2025).

The significance of this formalism is that retrieval quality, citation fidelity, and downstream answer correctness become jointly measurable yet analytically separable. BrowseComp-Plus is therefore not only an agent benchmark but also an infrastructure for diagnosing where a deep-research system fails.

4. Quantitative performance and retriever sensitivity

BrowseComp-Plus was explicitly shown to distinguish the performance of deep-research systems. Three representative end-to-end configurations reported in the benchmark are summarized below (Chen et al., 8 Aug 2025).

Configuration	Accuracy
Search-R1-32B + BM25	3.86%
GPT-5 + BM25	55.90%
GPT-5 + Qwen3-Embedding-8B	70.12%

These numbers establish two benchmark properties. First, the fixed-corpus setting remains highly discriminative across model classes. Second, retriever choice substantially affects end-to-end answer quality. Switching GPT-5 from BM25 to Qwen3-Embedding-8B raises accuracy by approximately 14 points, from 55.9 to 70.1, while also reducing average search calls from 23.23 to 21.74 (Chen et al., 8 Aug 2025).

The same retriever shift improves citation behavior. In the reported analysis, GPT-5’s citation recall climbs from 51.3% to 62.3% when paired with the stronger retriever (Chen et al., 8 Aug 2025). This indicates that retriever improvements do not merely alter ranking metrics; they propagate into the evidence actually cited by the agent and into the final grounded answer.

A plausible implication is that BrowseComp-Plus makes retriever-agent coupling a first-class experimental object. In live-web evaluation, improvements in answer accuracy could be attributed ambiguously to model updates, retrieval updates, or API drift. In BrowseComp-Plus, by contrast, retrieval substitution can be studied directly under constant corpus conditions.

5. Diagnostic analyses: retrieval ceilings, context engineering, and reasoning effort

BrowseComp-Plus supports oracle-style analysis by feeding agents all positive documents. Under that condition, gpt-4.1 reaches 93.5% accuracy and Qwen3-32B reaches 83.3%. The benchmark authors interpret the remaining errors as reasoning faults rather than retrieval failures (Chen et al., 8 Aug 2025).

The benchmark also isolates context engineering effects. Allowing gpt-4.1 to use a get-document tool that fetches full documents beyond the default 512-token previews raises accuracy from 35.4% to 43.6%, with approximately 1.8 full reads per query (Chen et al., 8 Aug 2025). This result indicates that preview truncation is not merely a latency optimization; it materially shapes the reasoning regime in which an agent operates.

Reasoning effort can likewise be varied independently of the corpus. For gpt-oss models, moving from low to high reasoning increases accuracy substantially; for oss-20B, the reported increase is 13.4% to 34.6%, while average search calls rise from approximately 2 to approximately 24 (Chen et al., 8 Aug 2025). This exposes a direct trade-off between answer quality and search expenditure.

Taken together, these analyses make BrowseComp-Plus a benchmark for bottleneck diagnosis rather than only leaderboard comparison. Retrieval quality drives a large fraction of performance; context access patterns produce additional gains; and even with oracle evidence, substantial residual failure remains. This decomposition is one of the benchmark’s defining contributions.

6. Later extensions, saturation concerns, and terminological divergence

Later work extends the controlled-evaluation idea in several directions. XBCP, or Cross-lingual BrowseComp-Plus, preserves the English question-and-answer space of BrowseComp-Plus while varying the languages of the supporting documents. It defines a cross-lingual setting, where each query is paired with evidence in a single assigned language, and a multilingual setting, where the corpus is distributed equally and randomly across 12 languages. Reported results show substantial degradation in answer accuracy, evidence recall, calibration, and citation fidelity when evidence is translated, and show that accuracy remains lower even when all gold evidence is supplied directly, indicating both retrieval failures and an agent-side difficulty in integrating language-mismatched evidence (Lu et al., 13 Jun 2026).

A different later line of work argues that BrowseComp-style evaluation is approaching saturation. LoHoSearch reports that average model accuracy on BrowseComp rose from roughly 30 percent to over 90 percent within ten months and attributes the resulting difficulty ceiling to human-authored question construction, low-branching constraints, and limited structural complexity. Its proposed remedy is KG-driven subgraph sampling with formal uniqueness guarantees, and its presentation frames BrowseComp+ as an expanded version intended to overcome this ceiling (Zhao et al., 11 Jun 2026). This suggests a future in which BrowseComp-Plus-style benchmarks may be regenerated algorithmically to preserve discriminative power.

At the same time, the term “BrowseComp+” is not used uniformly across the literature. WebExplorer refers to “BrowseComp-Plus” as a recent extension that re-annotates and standardizes the original BrowseComp benchmarks for greater fairness and transparency (Liu et al., 8 Sep 2025). By contrast, DeepDive, OpenSeeker, InfoAgent, and AgentFounder use “BrowseComp+” or “BrowseComp⁺” to denote a combined or extended benchmark built from BrowseComp-en and BrowseComp-zh, rather than the fixed 100 K-document corpus introduced by Chen et al. (Lu et al., 12 Sep 2025, Du et al., 16 Mar 2026, Zhang et al., 29 Sep 2025, Su et al., 16 Sep 2025). This suggests that the name has evolved into a broader family label in subsequent agent papers, even though its original technical meaning is the fixed-corpus, evidence-annotated benchmark of 2025.

7. Limitations and prospective research directions

BrowseComp-Plus has explicit limitations. Its static corpus cannot capture very recent or dynamically updated content. Its corpus size, approximately 100 K documents, is smaller than web scale; although fine-web augmentations yielded consistent trends, false negatives may arise. Its focus is fact-seeking queries, so interactive, multimodal, or API-driven tasks such as maps and forms lie outside the benchmark (Chen et al., 8 Aug 2025).

The benchmark’s stated future directions include co-optimizing retriever and agent through joint training, studying out-of-distribution tool-use generalization such as switching retrievers at test time, orchestrating federated or multi-tool search, designing retrieval objectives tailored to downstream deep-research agents rather than human readers, and extending the setup to multimedia or real-time data sources (Chen et al., 8 Aug 2025).

Subsequent work makes these directions more concrete. Cross-lingual evaluation shows that multilingual retrieval and evidence integration are not solved by monolingual advances alone (Lu et al., 13 Jun 2026). Difficulty-ceiling analyses suggest that future BrowseComp-Plus variants may need larger knowledge-graph-backed construction pipelines and formal uniqueness checks to remain discriminative (Zhao et al., 11 Jun 2026). On that trajectory, BrowseComp-Plus functions less as a single static dataset than as a controlled benchmarking paradigm for deep-research agents: fixed corpora, verified evidence, disentangled retrieval analysis, and explicit measurement of grounded answer production.