LiveBrowseComp: Are Search Agents Searching, or Just Verifying What They Already Know?

Abstract: Are LLM-based search agents genuinely searching, or using the web to verify what they already know? We study this question on BrowseComp with three diagnostics. Our analysis reveals Intrinsic Knowledge Dependence (IKD): even with tool access, agents often rely on intrinsic knowledge -- information encoded in the model before retrieval -- rather than on external evidence. Agents answer up to 44.5% of BrowseComp questions without tools, generate more than half of their search queries from internally produced hypotheses rather than retrieved leads, and perform worse than closed-book baselines when answer-supporting evidence is removed. These results suggest that static search benchmarks can reward memory-backed verification rather than evidence-driven discovery, conflating what agents already know with what they can find. We then introduce LiveBrowseComp, a deep-search benchmark designed to evaluate agents beyond intrinsic coverage. It contains 335 human-authored questions whose answers depend on facts published within the 90 days preceding benchmark construction, drawn from six updated sources and filtered to exclude globally salient events. On LiveBrowseComp, all evaluated agents fall below 2% closed-book accuracy, search-augmented scores drop by 25-40 points relative to BrowseComp, and prior model rankings no longer reliably predict performance. LiveBrowseComp is available at https://huggingface.co/datasets/Forival/LiveBrowseComp.

• The paper demonstrates that traditional benchmarks overestimate LLM search capabilities by relying on internal memory.
• It introduces LiveBrowseComp, a benchmark designed with recent, obscure facts and multi-step queries to reduce Intrinsic Knowledge Dependence.
• Experimental results reveal a significant drop in search performance when models rely solely on evidence retrieval, urging new evaluation strategies.

LiveBrowseComp: Disentangling Genuine Search from Parametric Knowledge in LLM Search Agents

Problem Formulation: Intrinsic Knowledge Dependence in Search Agent Evaluation

Conventional deep-search benchmarks such as BrowseComp have fueled rapid advances in LLM-based search agents, but a critical evaluation confound has become apparent: the majority of benchmark questions are solvable not through evidence-led retrieval, but via parametric knowledge internalized during model training. This phenomenon, termed Intrinsic Knowledge Dependence (IKD), leads to the conflation of memory-backed verification and genuine information-seeking. The authors systematically diagnose IKD using three evaluation protocols: closed-book answering (measuring intrinsic knowledge coverage), evidence-blocked search (detecting reliance on external evidence), and search trajectory analysis (quantifying whether queries are retrieval-led or memory-led).

Figure 1: Closed-book scores dominate static search benchmarks; the incremental gains from tool use are disproportionately distributed and not commensurate with closed-book coverage.

Empirical results show that even with tool access disabled, frontier agents solve up to 44.5% of BrowseComp questions using only internal memory. More than half of search queries are seeded by hypotheses originating from the model's own reasoning, rather than from prior retrieved evidence. When answer-supporting documents are withheld, search utility reverses: all models deteriorate beyond closed-book baselines, with major drops (e.g., MiniMax M2.5 deteriorates from 44.5% to 8.0%). These patterns confirm that traditional search benchmarks reward agents for what they already know, not for their evidence-discovery capabilities.

Figure 2: Analysis of search trajectories reveals majority of queries are model-originated; evidence after retrieval is underutilized.

Benchmark Development: Constructing LiveBrowseComp to Suppress IKD

To enable robust search agent evaluation, the paper introduces LiveBrowseComp—a benchmark designed to minimize overlap with parametric knowledge. The construction pipeline enforces stringent constraints: all questions are anchored to obscure facts published within 90 days prior to benchmark creation, with multi-step reasoning requirements and systematic exclusion of globally salient events. Seed sourcing spans six domains (news, film, games, cybersecurity, sports, earthquakes), each filtered for temporal recency and long-tail obscurity.

Figure 3: Schematic of the LiveBrowseComp concept; static benchmark questions lose difficulty as knowledge is assimilated into model parameters, but LiveBrowseComp maintains difficulty by leveraging dynamically updated facts.

Figure 4: Question bank construction pipeline integrates temporal/obscurity filtering, human annotation, and multi-annotator verification.

Quality control is rigorous: all questions undergo multi-stage human verification—peer review, difficulty calibration by independent annotators, and temporality checks—to ensure uniqueness, solvability, and genuine dependency on recent evidence. Final questions are retained only if unsolved by most humans within 30 minutes using web search, and all reference answers are traceable to post-cutoff evidence.

Figure 5: Human annotation time distributions are closely matched between BrowseComp and LiveBrowseComp, indicating comparability in search difficulty and excluding confounds from question hardness.

Experimental Results: Quantitative Suppression of Parametric Knowledge

LiveBrowseComp exposes the information boundary of LLM agents. Closed-book performance for all evaluated models drops below 2% on LiveBrowseComp, confirming effective suppression of parametric shortcuts. Search-enabled scores decline 25–40 points relative to static benchmarks, with avg@4 ranging from 28.0 to 43.2 versus 51–77 on BrowseComp. Rank order changes markedly—models with strong static scores (high IKD) do not retain advantage in the live benchmark, demonstrating that static leaderboards are inflated by knowledge coverage rather than search skill.

Figure 6: Closed-book accuracy is dramatically reduced on LiveBrowseComp (<2% for all models), confirming effective exclusion of parametric knowledge.

Correlation analysis affirms the decoupling: Spearman/Pearson coefficients between BrowseComp and LiveBrowseComp are significantly lower than between static benchmarks, conclusively demonstrating that static evaluations fail to predict live-search capability.

Figure 7: Score correlation between BrowseComp and LiveBrowseComp is much weaker than between two static benchmarks, confirming divergence in evaluation outcomes.

Search turn distribution analysis further suggests behavioral shifts: LiveBrowseComp eliminates the short-turn verification cluster seen in BrowseComp, forcing agents into longer, exploratory search trajectories without memory anchoring.

Figure 8: Distribution of search turns per question shifts from short-turn verification (BrowseComp) to longer exploratory paths (LiveBrowseComp).

Practical and Theoretical Implications

These findings necessitate re-thinking agentic evaluation and training. Static benchmarks reward guess-and-verify behaviors, underestimating the difficulty of genuine evidence discovery, and their leaderboards are increasingly invalid as model training windows expand. Live, temporally anchored benchmarks such as LiveBrowseComp provide a principled methodology for decoupling memory from search, ensuring that search agents are evaluated on their ability to find and synthesize new information. This also impels algorithmic development toward evidence-led search behaviors and context-aware retrieval strategies, rather than parametric-answer confirmation.

Likewise, periodic refresh cycles, stringent temporal cutoffs, and long-tail event selection should become standard in agentic benchmark construction. Theoretical work can leverage these paradigms to better characterize the generalization boundary between parametric knowledge and evidence-driven search, advancing open questions on the intersection of memory, reasoning, and information acquisition.

Future Directions

LiveBrowseComp lays groundwork for dynamic, contamination-immune evaluation systems. Future research can further disambiguate the boundary between parametric and retrieved knowledge via fine-grained trajectory tracing, adaptive benchmark generation, and targeted agent training that rewards evidence integration. There is also scope for cross-domain, cross-lingual, and multimodal instantiations of live benchmarks to stress-test search agents under real-world constraints.

Conclusion

LiveBrowseComp systematically neutralizes Intrinsic Knowledge Dependence, exposing the limitations of current LLM search agents whose strong static scores mask reliance on memory. By evaluating on recent, obscure facts beyond models' parametric coverage, LiveBrowseComp forces evidence-led discovery and reranks models based on genuine search ability. These results support dynamic, temporal evaluation as a cornerstone for search agent progress, with direct implications for benchmark design, deployment, and search-oriented training paradigms (2605.28721).