EvoBrowseComp: Benchmarking Evolving Search Agents

• EvoBrowseComp is an evolving benchmark defining search agents by leveraging fresh live-web data to counter static tests and shortcut memorization.
• It employs a three-agent framework—QA synthesis, evidence filtering, and reasoning graph validation—to ensure robust, longitudinal assessment.
• The benchmark reveals empirical difficulties, illustrating that disciplined, tool-based retrieval and multi-source cross-verification are essential for genuine competence.

EvoBrowseComp is an evolving benchmark for evaluating search agents—LLMs augmented with search tools—under conditions intended to remain valid as world knowledge changes. It introduces 800 complex question–answer pairs, evenly split between English and Chinese, synthesized automatically from live-web traversal after a freshness timestamp and filtered to reduce contamination, parametric memorization, and reasoning shortcuts (Wang et al., 11 Jun 2026). Within the broader BrowseComp literature, the term also denotes a longitudinal view of browsing competence: BrowseComp-$V^3$ presents Visual, Vertical, and Verifiable evaluation for multimodal browsing, thereby operationalizing the measurement of how browsing competence evolves across domains, reasoning depth, and process quality (Zhang et al., 13 Feb 2026).

1. Origins in the BrowseComp line

BrowseComp established the basic benchmark template that EvoBrowseComp extends. It comprises 1,266 questions with short, semantically gradable answers, and was explicitly designed to measure whether agents can persistently and creatively navigate the open internet to locate hard-to-find, entangled information (Wei et al., 16 Apr 2025). Its construction emphasized difficulty, persistence, and verifiability: questions were created by human trainers, answers were short strings, and grading relied on semantic equivalence rather than exact matching.

EvoBrowseComp is motivated by limitations that arise when such benchmarks are static. The benchmark description identifies three failure modes of fixed datasets: test-set contamination, parametric memorization, and reasoning shortcuts. Static datasets such as BrowseComp, BrowseComp-ZH, BrowseComp-Plus, GAIA, and DeepSearchQA are described as being anchored to fixed knowledge snapshots; as pretraining corpora expand, benchmark content can leak into model parameters, making “solve-by-recall” possible and obscuring genuine retrieval competence (Wang et al., 11 Jun 2026). The paper also states that BrowseComp answers leaked into public data, which is presented as explicit evidence that contamination is not merely hypothetical.

The central design response is temporal freshness. EvoBrowseComp grounds questions in fresh knowledge discovered from the live web after a specified timestamp, requires the final answer to depend on fresh knowledge, and uses a fully automated synthesis pipeline so that the benchmark can be regularly updated. This makes temporal refresh a core property rather than a maintenance afterthought. A plausible implication is that benchmark validity is treated as a moving target tied to model training cutoffs, not as a one-time curation artifact.

2. Three-agent synthesis and shortcut control

EvoBrowseComp is produced by a three-agent collaborative framework. The first component, the QA synthesis agent, retrieves fresh knowledge from the live web and synthesizes question–answer pairs. Seed entities are collected from the live web across 9 core domains and 50 sub-domains using DeepSeek-V3.2 with search tools, producing approximately $50\text{K}$ seeds $E$ (Wang et al., 11 Jun 2026). For each seed $e$, synthesis proceeds iteratively: $$e \to \langle q_e^{(1)}, a_e^{(1)} \rangle \to \langle q_e^{(2)}, a_e^{(2)} \rangle \to \cdots \to \langle q_e^{(m)}, a_e^{(m)} \rangle.$$

The synthesis agent performs multi-turn interactions with two tools, a Google search tool (“Search”) and a page extraction tool (“Visit”), to collect an evidence list $\mathcal{E} = \{\epsilon_1,\ldots,\epsilon_n\}$. Fresh knowledge is defined as information available after timestamp $t$, set in the paper to January 1, 2026. Each evidence item is explicitly classified as fresh or non-fresh, and every evidence item must be backed by a visited source_url, not merely a search snippet. Difficulty is increased through obfuscation strategies such as vague time anchors and non-specific descriptors, while retrievability is preserved.

The second component, the information filtering agent, enforces credibility and anti-shortcut constraints. Fresh evidence is cross-validated across multiple authoritative sources and labeled “credible,” “not credible,” or “unclear”; only credible fresh evidence is retained. Non-fresh evidence is labeled “popular” or “non-popular” without tools, and only non-popular non-fresh evidence is retained. If the retained evidence list shrinks below the threshold $k=5$, the QA synthesis agent retries evidence gathering. This makes shortcut suppression operational rather than aspirational: fresh evidence blocks stale memorization, while popularity filtering reduces reliance on widely known facts.

The third component, the high-level guidance agent, formalizes each question into a reasoning graph $$\mathcal{G}_e^t = \{\mathcal{V}_e^t,\mathcal{R}_e^t\}$$ and iteratively removes redundancy and shortcut paths. The graph is built using three operations: $$v_i \cap v_j,\qquad \overline{v_i},\qquad v_j = \pi_r(v_i).$$ The paper defines these operations explicitly as intersection, complement, and projection. After graph construction, the agent detects isolated nodes or subgraphs unrelated to the answer and identifies structural bypasses that would let a model reach the answer without traversing the intended chain. These checks use NetworkX. Iteration stops only when no redundancy or shortcuts remain, at least five iterations have run, and the graph has at least five edges, that is, $50\text{K}$0.

3. Corpus structure, grounding, and quality assurance

The released benchmark contains 800 QA pairs: 400 English and 400 Chinese. The questions are evenly distributed across nine predefined domains (Wang et al., 11 Jun 2026). Average question lengths are 142.48 tokens for English and 162.33 tokens for Chinese, while the average reasoning-graph node counts are 8.62 and 8.07 respectively. Every question is grounded in visited webpages through recorded source_url fields, and the benchmark formalizes each item as a reasoning graph rather than a free-form text prompt.

A defining structural property is horizontal search breadth. Questions aggregate evidence from multiple independent root domains; the distribution is described as bell-shaped, with an average of 4.2 distinct root domains per question, and over 90% of questions require reasoning across at least three independent sources. This is intended to force cross-site collation and verification rather than single-source lookup. The benchmark description presents this as evidence that EvoBrowseComp requires broad horizontal search rather than narrow local retrieval.

Quality control proceeds in multiple stages. DeepSeek-V3.2 judges fluency, clarity, and ambiguity, and low-quality pairs are removed. Difficulty and uniqueness are stress-tested using six strong LLMs, each answering three times, for 18 total solutions per question. If more than 80% of solutions converge on the same incorrect answer, the question is discarded to avoid multiple valid answers; if at least 5 of 6 LLMs answer correctly, the question is also discarded to maintain difficulty. Human validation on a 100-sample subset reports that 93.0% of evidence lists are entirely correct, 90% of questions are consistent with their evidence lists and unambiguous, and 100.0% of answers can be inferred from the evidence; 87% pass all three checks (Wang et al., 11 Jun 2026).

These design choices jointly make provenance a first-class benchmark object. Evidence lists, source URLs, and reasoning-chain metadata are part of the benchmark artifact, not merely annotation residue. This suggests that EvoBrowseComp treats evaluation as inseparable from auditable retrieval structure.

4. Evaluation protocol and measurement regime

Evaluation uses a tool-based browsing setup with explicit resource constraints. Agents are given search (Google) and visit tools, a maximum of 40 tool calls, and a 128K context window; the reported decoding parameters are temperature=0.6 and top_p=0.95, and open-source deployments run on NVIDIA H20 GPUs via SGLang (Wang et al., 11 Jun 2026). The metric is accuracy based on decisions by GLM-5-Chat, used as an LLM-as-a-judge. The paper does not provide formal formula definitions for “Accuracy,” “F1,” “Freshness,” or $50\text{K}$1, and accuracy is reported as a percentage.

Judge calibration is itself evaluated. GLM-5-Chat is reported to have the best Spearman correlation with human judgments, at 0.864. This gives the benchmark a stated judge-reliability anchor, although it remains a final-answer evaluation rather than a trajectory evaluation.

The baseline set includes both tool-free and tool-based systems. Tool-based evaluations cover Claude-Opus-4.6, multiple Qwen3.5 variants, GLM-5, Kimi-K2.6, DeepSeek-V4-Pro/Flash, DeepSeek-V3.2, and Qwen3-235B. The benchmark also reports ablations on reasoning-effort configurations for DeepSeek-V4-Flash—Chat, High, and Max—and introduces “Exceed Ratio,” defined as the proportion of evaluation samples that exceeds the maximum allowed number of tool calls (Wang et al., 11 Jun 2026).

In the adjacent multimodal BrowseComp-$50\text{K}$2 setting, evaluation is expanded beyond final-answer accuracy to include a process-level metric. Success Rate is evaluated under Pass@1, and Process Score is defined as the proportion of ground-truth subgoals completed: $50\text{K}$3 BrowseComp-$50\text{K}$4 applies this over 300 multimodal questions with gold-standard trajectories, expert-validated subgoals, and tool-mediated multi-turn browsing, thereby extending EvoBrowseComp-style measurement from evolving knowledge into evolving multimodal browsing behavior (Zhang et al., 13 Feb 2026).

5. Empirical difficulty and observed capability boundaries

EvoBrowseComp is deliberately difficult in both tool-free and tool-based settings. Without tools, most evaluated models score below 5% accuracy; the best tool-free result reported is DeepSeek-V3.2 with 6.3% on English and 10.3% on Chinese (Wang et al., 11 Jun 2026). The paper presents this as confirmation that the benchmark prevents solve-by-recall and requires genuine retrieval.

With tools, accuracy improves but remains modest. In English, Claude-Opus-4.6 reaches 44.8% accuracy, Qwen3.5-397B reaches 42.0%, and GLM-5 reaches 39.2%. Chinese results are lower but similar in ordering: Claude-Opus-4.6 achieves 36.8%, Qwen3.5-397B 34.5%, and GLM-5 30.5%. The benchmark summary states that even top LLMs perform modestly with tools at below 45% accuracy and collapse without tools at below 11%, reinforcing the claim that broad horizontal search is necessary.

Comparison with BrowseComp and BrowseComp-ZH quantifies the increased difficulty. For Qwen3.5-397B, performance drops from 69.0% to 42.0% on English and from 70.3% to 34.5% on Chinese when moving from BrowseComp/BrowseComp-ZH to EvoBrowseComp (Wang et al., 11 Jun 2026). This is presented as evidence of both increased difficulty and greater contamination resistance.

Reasoning efficiency matters as much as raw search access. DeepSeek-V4’s Max setting produces high Exceed Ratios—75.5% on English and 82.5% on Chinese—and lower accuracy than the High setting, whereas Claude-Opus-4.6 maintains far lower Exceed Ratios, around 26%, alongside higher accuracy. The stated conclusion is that “more thinking” can be inefficient; disciplined tool usage and targeted reasoning are advantageous. This suggests that benchmark success depends not only on retrieving more information, but on using a bounded interaction budget coherently.

BrowseComp-$50\text{K}$5 reaches a related conclusion in the multimodal regime. There, human performance is $50\text{K}$6 Success Rate and $50\text{K}$7 Process Score, while no model exceeds 40% Success Rate; the best official tool-augmented result is GPT-5.2-Thinking at $50\text{K}$8 SR and $50\text{K}$9 PS, and OmniSeeker + GPT-5.2 reaches $E$0 SR and $E$1 PS (Zhang et al., 13 Feb 2026). The reported bottlenecks are “multimodal information integration and fine-grained perception,” with higher Process Score than Success Rate indicating frequent partial progress that fails to resolve into correct end-to-end answers.

6. Longitudinal significance, reproducibility, and limitations

EvoBrowseComp is explicitly designed to be auto-updatable. Because its synthesis pipeline is fully automated and tool-driven, the benchmark can be regenerated at low cost by resetting the freshness timestamp $E$2, retiring over-exposed items, and incorporating newly surfaced facts (Wang et al., 11 Jun 2026). The released dataset is hosted at https://hf.co/datasets/Krystalan/EvoBrowseComp under CC-BY-NC-SA 4.0. Prompts for all agents and judges are provided in the appendix, and the benchmark artifact includes QA pairs with evidence lists and source URLs. The paper describes tool definitions and prompts but does not provide a public code repository for the synthesis pipeline.

The broader significance of EvoBrowseComp becomes clearer when viewed together with BrowseComp-$E$3. That benchmark identifies three axes for tracking the evolution of browsing competence: vertical domain coverage, multimodal levels $E$4, and process metrics over subgoals. It spans 24 sub-domains organized into five major verticals—Science, Technology, Society, Culture, and Life—and structures visual reasoning at three levels: intra-region alignment, inter-region integration, and inter-image reasoning (Zhang et al., 13 Feb 2026). In that formulation, EvoBrowseComp is not only a benchmark name but also a measurement program: browsing competence is something to be tracked longitudinally across domains, modalities, hops, and intermediate reasoning quality.

The stated limitations of EvoBrowseComp are also structural rather than cosmetic. The three synthesis agents use DeepSeek-V3.2, so the synthesized data may reflect its biases or toxic behaviors. Evaluation judges only final answers, not full reasoning trajectories, making it difficult to separate correct reasoning from accidental hits. The paper does not report multilingual expansion beyond English and Chinese, and it does not detail broader fairness auditing (Wang et al., 11 Jun 2026). BrowseComp-$E$5 reports a complementary blind spot: its subgoals are expert-validated and unweighted, but no inter-annotator agreement statistics are reported, and the framework does not detail specialized OCR, layout parsing, or structured extraction, even though perception errors in noisy web graphics remain prominent (Zhang et al., 13 Feb 2026).

Taken together, these works define EvoBrowseComp as both a concrete evolving benchmark and a broader methodology for future-proof browsing evaluation. In its strict sense, it is a live-web, contamination-resistant benchmark of 800 complex questions. In its broader sense, it is a framework for measuring whether improvements in search agents reflect genuine browsing competence rather than memorized answers, static benchmark exposure, or shortcut exploitation.