EvoBrowseComp: Benchmarking Search Agents on Evolving Knowledge

Abstract: Search Agents -- LLMs augmented with search tools -- have intensified the need for future-proof evaluation benchmarks. Existing benchmarks such as BrowseComp rely on static knowledge, making them vulnerable to test-set contamination and parametric memorization. Consequently, models can achieve high scores through fact recall rather than genuine retrieval, obscuring true browsing competence via reasoning shortcuts. In this paper, we introduce EvoBrowseComp, an evolving benchmark of 400 English and 400 Chinese contamination-free complex questions synthesized via live-web traversal. To collect these questions, we design a three-agent collaborative framework: (1) a QA synthesis agent that retrieves fresh knowledge from the live web to synthesize QA pairs; (2) an information filtering agent that filters retrieved knowledge in terms of credibility and popularity to block parametric shortcuts; and (3) a high-level guidance agent that formalizes questions into reasoning graphs to reduce logical redundancy and shortcuts in synthesized QA pairs. Because the framework supports fully automated synthesis, EvoBrowseComp can be regularly updated to prevent data contamination and maintain temporal freshness. Extensive experiments confirm its great difficulty, requiring broad horizontal search. It establishes a scalable paradigm for auto-updatable, high-difficulty benchmarking that keeps pace with both evolving world knowledge and advancing agent capabilities.

• The paper introduces a three-agent synthesis framework that automates multi-hop QA generation and ensures contamination-resistant evaluations.
• It leverages live web traversal and cross-source validation to generate robust and fresh English and Chinese QA pairs.
• Evaluation results reveal significant reasoning and retrieval gaps in current SOTA LLMs, highlighting the need for further advancements.

EvoBrowseComp: A Formal Benchmark for Search Agents on Evolving Knowledge

Motivation and Context

EvoBrowseComp addresses critical gaps in the evaluation of search-augmented LLMs. Existing benchmarks—such as BrowseComp and BrowseComp-ZH—rely on static, manually curated QA datasets that are increasingly susceptible to test set contamination and parametric memorization. As pretraining corpora expand, benchmark questions leak into model parameters, enabling agents to solve purportedly complex queries via simple fact recall. This obscures actual browsing and multi-hop reasoning competence, impeding objective assessment of LLMs’ genuine information-seeking capabilities.

EvoBrowseComp introduces a contamination-resistant, auto-updatable evaluation paradigm. It synthesizes 400 English and 400 Chinese complex QA pairs via traversal of the live web, incorporating only emerging knowledge postdating typical model cutoffs (after Jan 1, 2026). The synthesis pipeline is fully automated, enabling continuous updates and retiring over-exposed questions. This ensures stringent requirements for answer freshness, complexity, and cross-source evidence aggregation.

Figure 1: An illustrative EvoBrowseComp question showcasing fresh knowledge (highlighted in orange) on reasoning paths and the final answer (in red).

Three-Agent Synthesis Framework

EvoBrowseComp’s benchmark construction leverages a three-agent iterative collaborative framework:

• QA Synthesis Agent: Executes live web search and evidence extraction from recently surfaced entities, with tool-assisted multi-turn interactions across search and targeted site visits. Synthesizes candidate QA pairs anchored in fresh knowledge, with complex multi-hop reasoning chains and explicit obfuscation to preclude parametric shortcuts.
• Information Filtering Agent: Conducts automated credibility assessment through cross-source validation for fresh evidence and popularity scoring for non-fresh evidence. Retains only credible, non-popular facts, blocking highly exposed shortcuts and unreliable rumors, critical given the volatility of fresh web content.
• High-Level Guidance Agent: Formalizes question structure as reasoning graphs—nodes representing entity/attribute sets, edges as intersection, complement, or projection operations. This agent detects logical redundancy and shortcuts, supplying explicit textual guidance for the QA synthesis agent to further enhance adversarial complexity and ensure non-redundancy.

  Figure 2: The three-agent collaborative framework for question synthesis, information filtering, and reasoning guidance.

  

  Figure 3: Example reasoning graph structuring multi-hop question semantics.

This pipeline supports iterative refinement. Each question requires several synthesis rounds, with termination conditioned on the removal of shortcuts/redundancy, minimum reasoning depth (≥5 hops), and guaranteed graph complexity (≥5 edges).

Data Quality and Diversity

Automated quality control is multi-tiered:

• Textual Quality: LLM-based QA pair filtering ensures fluency, clarity, and lack of ambiguity.
• Uniqueness/Difficulty: Cross-validation by six cutting-edge search agents discards questions with multiple plausible answers or insufficient difficulty (≥5 LLMs solve, question is removed). Human expert evaluation verifies evidence correctness, QA consistency, and answer inferability, with 87% QA pairs passing all checks.

Domain balancing assures broad coverage: nine domains (e.g., science, economy, geography) each contain ~44 relevant QA pairs.

Figure 4: Distribution across nine domains in EvoBrowseComp.

Source diversity is enforced by requiring aggregation across multiple root domains per question. On average, each QA pair involves evidence from 4.2 independent domains, and 90% require reasoning over at least three distinct sources.

Figure 5: Distinct root domain distribution per EvoBrowseComp question.

The dataset exhibits high average question length and graph complexity, particularly in Chinese, outstripping prior benchmarks (e.g., BrowseComp-ZH: 162.33 tokens and 8.07 nodes vs. 73.88 tokens and 6.63 nodes).

Evaluation Protocol and Experimental Results

LLMs are evaluated under tool-based and tool-free settings:

• Tool-Free: All LLMs, including state-of-the-art Claude-Opus-4.6, DeepSeek-V3.2, and Qwen3.5 variants, achieve ≤10% accuracy (e.g., Claude-Opus-4.6: 6.0% EN, 8.8% ZH); demonstrating effective contamination resistance and dependence on genuinely fresh retrieval.
• Tool-Based: Best-performing models achieve ≤45% (Claude-Opus-4.6: 44.8% EN, 36.8% ZH). Other SOTA models (GLM-5, Qwen3.5-397B, DeepSeek-V3.2) return similarly modest scores. Contrasted with BrowseComp (e.g., Qwen3.5-397B: 69.0%), EvoBrowseComp’s difficulty is substantially higher.

Reasoning efficiency remains problematic: DeepSeek-V4-Flash exhibits high exceed ratios (ER) on tool call constraints (max 40), often requiring excessive tool invocations per sample; reasoning efficiency optimization remains a challenge even as agentic LLMs improve in depth.

Implications and Future Directions

EvoBrowseComp's methodology decouples benchmark evolution from manual curation, enabling contamination-resistant, temporally fresh evaluation aligned with real-world information drift. It sets a new standard for automated synthesis of adversarial multi-hop QA—highly complex, obfuscated, and cross-source—driving robust assessment of browsing agents.

Practically, EvoBrowseComp will inform the design of deep research agents and search-augmented LLMs capable of navigating fragmented and evolving web environments. It exposes both the retrieval and reasoning gaps present in current SOTA models, emphasizing the need for further advancements in horizontal evidence discovery, reasoning efficiency (tool call optimization), and adversarial robustness.

Theoretically, its synthesis pipeline provides a graph-based formalization framework for multi-hop reasoning QA, which may underpin future developments in agentic data generation, adversarial benchmarking, and evaluation protocol design.

As benchmarks evolve alongside AI capabilities and web content, EvoBrowseComp is positioned for regular refreshes and expansion, supporting continuous, future-proof evaluation and rapid research iteration.

Conclusion

EvoBrowseComp establishes a scalable, automated, contamination-resistant paradigm for search agent benchmarking. It demonstrates stringent quality and adversarial complexity via a three-agent collaborative synthesis framework and reasoning graph formalization. Evaluation results underscore significant gaps in browsing competence and reasoning efficiency among current SOTA LLMs, compelling future progress in search-augmented agentic systems. Its methodology—auto-updatable, contamination-free, cross-lingual, and richly annotated—constitutes an essential substrate for the sustainable advancement of information-seeking AI models (2606.13120).