LiveBrowseComp: A Live Search Benchmark
- LiveBrowseComp is a live search benchmark that isolates evidence-driven discovery by enforcing a strict 90-day recency window and mitigating intrinsic knowledge dependence.
- It employs a multi-stage curation process with temporal filtering, long-tail obscurity, and answer-stability checks to generate human-authored, multi-hop questions.
- Empirical results reveal that LLM-based agents severely underperform on LiveBrowseComp, highlighting a significant drop compared to static benchmarks.
Searching arXiv for LiveBrowseComp and related benchmark papers to ground the article. LiveBrowseComp is a deep-search benchmark for evaluating whether LLM-based search agents are genuinely searching or primarily verifying hypotheses derived from intrinsic knowledge. It was introduced to address a specific failure mode—Intrinsic Knowledge Dependence (IKD)—observed on static browsing benchmarks such as BrowseComp, where agents can often answer questions without tools, generate many search queries from internally produced hypotheses, and underuse retrieved evidence even when supporting documents are found. LiveBrowseComp therefore constrains questions to depend on facts published within the 90 days preceding benchmark construction, filters out globally salient events, and requires multi-hop evidence chains whose crucial clues are temporally recent and not recoverable from older substitutes (Fan et al., 27 May 2026).
1. Concept and motivation
LiveBrowseComp arose from the claim that static search benchmarks can conflate evidence-driven discovery with memory-backed verification. The motivating paper studies this problem on BrowseComp and reports three diagnostics of IKD: substantial closed-book accuracy, degraded performance when answer-supporting evidence is removed, and search trajectories in which more than half of queries originate from the model’s own hypotheses rather than retrieved leads (Fan et al., 27 May 2026). BrowseComp itself was designed as a benchmark of 1,266 hard-to-find but easy-to-verify questions that measure persistence and creativity in web browsing, yet it does not control for the possibility that answers have already entered model parameters (Wei et al., 16 Apr 2025).
Within this framing, IKD denotes reliance on intrinsic, parametric knowledge to hypothesize answers, followed by web use mainly for confirmation rather than discovery. The LiveBrowseComp paper operationalizes the problem through closed-book accuracy, evidence-blocked search harm, query-origin analysis, and post-retrieval evidence-use analysis. A plausible implication is that a benchmark can register high search-agent scores while under-measuring actual browsing competence if intrinsic coverage remains high (Fan et al., 27 May 2026).
This concern also aligns with later work on evolving or freshness-constrained benchmarks. EvoBrowseComp argues that static benchmarks are vulnerable to test-set contamination and parametric memorization, and introduces a fully automated, regularly refreshable benchmark of post-cutoff knowledge for precisely that reason (Wang et al., 11 Jun 2026). LiveBrowseComp differs in construction, but the underlying diagnosis is closely related: both benchmarks attempt to separate retrieval competence from fact recall.
2. Dataset construction and benchmark design
LiveBrowseComp contains 335 human-authored questions. Each question depends on facts published within the 90 days preceding benchmark construction, and the benchmark draws seed events from six continuously updated sources: GDELT, TMDB, RAWG, CVE/NVD, SportsDB, and USGS (Fan et al., 27 May 2026).
The construction pipeline applies three filters before annotation. Stage 1 is temporal filtering, retaining only events whose core facts cannot be determined from information older than 90 days. Stage 2 is long-tail filtering, using source-specific obscurity criteria to exclude globally salient events. Stage 3 is answer-stability filtering, removing candidates whose answers may change during the window. Annotators then construct multi-step questions with a single short-string answer, ensure that at least one indispensable clue is temporally recent, and document a complete evidence chain. Independent verifiers perform correctness and uniqueness checks, difficulty calibration, and temporality verification; if older substitutes can replace all recent pages in the chain, the question is excluded (Fan et al., 27 May 2026).
This design sharply contrasts with BrowseComp, whose inverted problems are built from known facts and constraints but are not freshness-anchored (Wei et al., 16 Apr 2025). It also differs from BrowseComp-ZH, which targets the Chinese web through native reverse design and first-page exclusion across Baidu, Bing, and Google, but is still a static benchmark (Zhou et al., 27 Apr 2025). By construction, LiveBrowseComp treats recency and temporal necessity as core benchmark variables rather than incidental properties.
The benchmark is released as a single evaluation set with archived snapshots preserved for reproducibility. The paper does not specify a train/dev/test split (Fan et al., 27 May 2026).
3. Diagnostic framework and evaluation protocol
The paper defines several measures for diagnosing IKD. Closed-book accuracy is
The evidence-blocked harm measure is
where is accuracy when answer-supporting evidence has been removed in a controlled retrieval setting. Positive indicates that blocked search performs worse than closed-book answering, which the paper interprets as a sign of IKD (Fan et al., 27 May 2026).
Trajectory analysis uses the query-origin fraction
and the evidence-use rate
These measures aim to distinguish retrieval-led exploration from internally led verification (Fan et al., 27 May 2026).
The search-augmented setting uses a unified scaffold with identical tools and budgets across models: search(query) via serper.dev with up to 10 results, visit(url, goal) via Jina with goal-directed summarization, and a code_sandbox. The maximum context per sample is 256k tokens, and the iteration budget is 250 steps. System prompts are standardized, with temperature set to 0.7 and top_p to 0.9. Closed-book runs disable tools entirely, and the paper reports pass@4 and avg@4 under both closed-book and tool-use settings (Fan et al., 27 May 2026).
This emphasis on unified tooling parallels controlled-corpus work such as BrowseComp-Plus, which fixes a curated corpus and separates retriever quality from agent reasoning, although BrowseComp-Plus addresses fairness and reproducibility rather than freshness per se (Chen et al., 8 Aug 2025).
4. Empirical findings
The central empirical result is that LiveBrowseComp suppresses intrinsic coverage. All evaluated agents fall below 2% closed-book accuracy on the benchmark (Fan et al., 27 May 2026). This is a marked contrast with static benchmarks. On BrowseComp, closed-book pass@4 reaches up to 44.5%; on BrowseComp-ZH, it reaches up to 62.0%; and across 24 model–benchmark pairs the average closed-book score is 38.9 pass@4 (Fan et al., 27 May 2026).
Search-augmented performance also drops sharply relative to static settings. On BrowseComp, leading models score 51–77 avg@4, whereas on LiveBrowseComp the corresponding range is 28.0–43.2 avg@4. The paper characterizes this as a drop of roughly 25–40 points relative to BrowseComp. Rankings change as well: GLM 5.1 leads BrowseComp at 68.0 but scores 33.9 on LiveBrowseComp, while DeepSeek v3.2 is near-bottom on BrowseComp at 51.4 yet rises to 37.6 on LiveBrowseComp (Fan et al., 27 May 2026).
The paper also reports that human solve rates and time distributions are nearly identical across BrowseComp and LiveBrowseComp, at about 30–31%. This suggests that LiveBrowseComp is not simply harder in a generic sense for humans; rather, it is harder specifically for systems that benefited from memory-backed shortcuts on static benchmarks (Fan et al., 27 May 2026).
Trajectory diagnostics reinforce that interpretation. On BrowseComp-Plus under evidence-blocked conditions, all blocked scores fall below 10%, and average performance drops from 26.1% closed-book to 6.2% blocked. More than half of search queries are model-originated, exceeding 60% in later rounds, and evidence-use rates after retrieving supporting documents remain under one-third across models: 32.2% for DeepSeek v3.2, 24.7% for GLM-5.1, 30.8% for MiniMax M2.5, and 31.5% for Kimi-K2.5 (Fan et al., 27 May 2026).
5. Relation to adjacent BrowseComp benchmarks
LiveBrowseComp belongs to a broader family of benchmarks derived from or reacting to BrowseComp, but it occupies a distinct niche centered on recency and discovery rather than static, multimodal, or fixed-corpus evaluation.
| Benchmark | Core emphasis | Representative scale |
|---|---|---|
| BrowseComp | Static open-web hard fact seeking | 1,266 questions |
| BrowseComp-ZH | Chinese web browsing under native Chinese retrieval constraints | 289 questions |
| BrowseComp-Plus | Fixed-corpus, fair and transparent evaluation | 830 queries |
| MM-BrowseComp | Multimodal browsing with image/video dependencies and checklists | 224 questions |
| Video-BrowseComp | Open-web agentic video reasoning with mandatory temporal visual grounding | 210 questions |
| EvoBrowseComp | Evolving, contamination-resistant evaluation on fresh knowledge | 800 QA pairs |
| LiveBrowseComp | Fresh, long-tail, recent-information discovery beyond intrinsic coverage | 335 questions |
BrowseComp remains the canonical benchmark for hard, open-web retrieval with short, verifiable answers, but the LiveBrowseComp paper argues that its static nature can reward verification over discovery (Wei et al., 16 Apr 2025, Fan et al., 27 May 2026). BrowseComp-ZH extends the paradigm to the Chinese web and highlights multilingual retrieval complexity, yet it does not impose recency constraints (Zhou et al., 27 Apr 2025). BrowseComp-Plus replaces live retrieval with a curated corpus to improve fairness and disentangle retriever versus LLM effects (Chen et al., 8 Aug 2025).
Multimodal extensions make different interventions. MM-BrowseComp introduces mandatory dependence on images and videos encountered during browsing, along with irreducible reasoning checklists (Li et al., 14 Aug 2025). Video-BrowseComp further specializes this to open-web video, requiring temporal visual evidence and showing that even advanced search-augmented models achieve only 15.24% accuracy (Liang et al., 28 Dec 2025). BrowseComp- emphasizes visual, vertical, and verifiable multimodal search with subgoal-level process evaluation (Zhang et al., 13 Feb 2026). EvoBrowseComp, in turn, addresses contamination and temporal freshness through automated live-web synthesis anchored after January 1, 2026 (Wang et al., 11 Jun 2026).
Taken together, these benchmarks can be read as decomposing “browsing competence” into orthogonal stressors: recency and contamination resistance in LiveBrowseComp and EvoBrowseComp, multilingual retrieval in BrowseComp-ZH, multimodal perception in MM-BrowseComp and BrowseComp-, and temporal visual grounding in Video-BrowseComp. This suggests that LiveBrowseComp should be understood not as a replacement for BrowseComp, but as a benchmark that isolates one particular weakness of static search evaluation.
6. Significance, limitations, and research implications
LiveBrowseComp’s significance lies in its attempt to re-center evaluation on evidence-led discovery. By combining a strict 90-day recency window, long-tail filtering, answer-stability checks, and temporality verification, it reduces the chance that success can be driven by parametric recall or by first generating a likely answer and then seeking superficial confirmation (Fan et al., 27 May 2026).
The benchmark also has explicit limitations. The 90-day window is described as a heuristic, and different models may have different training cutoffs. The use of a single web-search backend, serper.dev, may influence retrieval quality and coverage. Human curation and verification are costly, which limits scalability (Fan et al., 27 May 2026). These concerns echo a broader tension in benchmark design: fixed benchmarks improve reproducibility but risk contamination, whereas live or fresh benchmarks improve epistemic validity but are harder to stabilize and reproduce (Chen et al., 8 Aug 2025, Wang et al., 11 Jun 2026).
Several research directions follow directly. One is improved trajectory grounding: the paper’s low evidence-use rates imply that retrieval does not automatically translate into reasoning use (Fan et al., 27 May 2026). Another is query reformulation based on retrieved content rather than internally generated hypotheses. A third is benchmark refresh and maintenance. EvoBrowseComp proposes a fully automated synthesis pipeline to keep questions temporally fresh and contamination-resistant, and this suggests one possible path for scaling the LiveBrowseComp philosophy beyond manual curation (Wang et al., 11 Jun 2026).
At a methodological level, LiveBrowseComp reframes the evaluation question. The relevant issue is not merely whether an agent can answer a web question with tools, but whether the answer was obtained by discovering and integrating external evidence that the model did not already possess. In that sense, LiveBrowseComp functions both as a benchmark and as a critique of static search-agent evaluation (Fan et al., 27 May 2026).