K-BrowseComp-Verified Benchmark

• K-BrowseComp-Verified is a benchmark split that rigorously evaluates web-browsing agents on multi-hop reasoning, compositional search, and evidence-grounded answer synthesis.
• It uses a fixed web corpus with human-verified evidence and hard negative mining to ensure fair, transparent, and reproducible evaluations across English and Korean contexts.
• The benchmark advances research by isolating retrieval quality, diagnostic error analysis, and multi-modal extensions, providing actionable insights for robust tool-aware architectures.

K-BrowseComp-Verified is a rigorously constructed benchmark split for evaluating web-browsing agents, situated within the BrowseComp-Plus framework. It is designed to assess agents’ capacity for compositional search, persistent multi-hop reasoning, and evidence-grounded answer synthesis in complex, often adversarial information spaces. The K-BrowseComp-Verified standard establishes a reproducible, high-fidelity reference for empirical measurement and isolation of agentic deep research capabilities, supporting both English and non-English (notably Korean) contexts and serving as an evaluative backbone for tool-use, retriever quality, and reasoning diagnostics in classical, cross-lingual, and multimodal agent workflows.

1. Benchmark Definition and Motivation

K-BrowseComp-Verified, as realized in BrowseComp-Plus, consists of a fixed, controlled web corpus ($\sim$100,000 documents), a curated set of complex multi-hop queries (830 in English; 300 in the Korean "Verified" split), and gold-standard evidence sets for each query (Chen et al., 8 Aug 2025, Lee et al., 1 Jun 2026). The core goals are:

• Fairness: All models are evaluated on the identical document set, eliminating API or time-based confounds.
• Transparency: Human-verified relevance annotations for each query demarcate exactly which documents and evidence chains support a given answer, enabling disentangled assessment of retrieval and reasoning quality.
• Reproducibility: Offline evaluation scripts and annotation files permit perpetual re-execution of experiments.

The Korean extension (K-BrowseComp-Verified) further emphasizes culturally specific, compositional search over Korean web sources, with careful grounding in domain entities and problem difficulty.

2. Dataset Construction and Annotation Protocol

Evidence and Hard Negative Workflow

• Evidence Gathering: Automated extraction of candidate snippets and supporting URLs (e.g., using commercial LLMs with web search) is followed by annotator-driven span selection to highlight minimal text spans justifying each required clue for the query. Each piece of evidence is exhaustively cross-validated by a secondary annotator, enforcing $>$80% inter-annotator agreement and strict coverage of every reasoning step.
• Hard Negative Mining: To prevent shortcut retrieval, $>$76 distractor documents per query are algorithmically selected via sub-query expansion and retrieval, ensuring keyword overlap without actual evidence content (Chen et al., 8 Aug 2025).
• Gold Document Tagging: “Gold” documents (mean 2.9/query) are explicitly tagged as containing the final answer (implicitly or explicitly).
• Korean Verified Split: Items are manually authored and validated (no LLM generation), with a requirement that all entities, data, and gold chains are rooted in the Korean web and pass a two-stage review for evidence accessibility and answer uniqueness (Lee et al., 1 Jun 2026).
• Quality Control: In total, annotation required $\sim$400 person-hours, demonstrating the scale and labor-intensiveness of human verification (Chen et al., 8 Aug 2025).

Example Structure

Field	Description
Query	Multi-hop, compositional fact-seeking question
E₍q₎	Set of gold-standard, evidence documents (human-verified)
N₍q₎	Set of challenging, content-overlapping hard negatives
Answer	Unique, concise target string

3. Evaluation Metrics and Experimental Results

Metrics (all queries indexed by $q$; $Q = \# \text{queries}$)

• Accuracy: $$\mathrm{Acc} = \frac{1}{Q}\sum_{q=1}^Q \mathbf{1}[a_q = a_q^*]$$.
• Precision@k: $$\mathrm{P@}k = \frac{1}{Q}\sum_{q=1}^Q \frac{\bigl|R_q[1:k] \cap E_q\bigr|}{k}$$.
• Recall (Citation Recall): $$\mathrm{Recall} = \frac{1}{Q}\sum_{q=1}^Q \frac{|\tilde{R}_q \cap E_q|}{|E_q|}$$
• Calibration Error (ECE): Standard binned discrepancy between predicted confidence and empirical accuracy.
• Korean Split: Pass@1 accuracy on 300 verified Korean examples.

Key Results

Model	Pass@1 (Korean Verified)	ECE (%)
GPT-5.5 (closed)	45.67	31.86
GPT-5.4-mini (closed)	30.67	37.88
DeepSeek-V4-Pro	30.00	17.72
GLM-5.1	30.67	27.07
Qwen3.6-35B-A3B	12.00	47.89

Evidence quality and retrieval accuracy directly control downstream answer accuracy—for example, shifting from BM25 to Qwen3-Embed-8B retrievers yields $+14$ points for GPT-5 (Chen et al., 8 Aug 2025). Oracle conditions (perfect retriever; gold evidence injected) push maximal accuracy $>$0, confirming that retrieval, not answer synthesis, is the primary bottleneck in the benchmark (Chen et al., 8 Aug 2025).

4. Diagnostic Error Analysis and State Maintenance

• Failure Modes: Dominant errors are F5 search-result selection failures, F7 candidate ledger/constraining failures, and F3 cross-source entity misbinding (Lee et al., 1 Jun 2026). Unmerged evidence threads and incomplete candidate set intersections are frequent causes of miss.
• Trajectory Analysis: State maintenance, entity-role binding, and constraint tracking over multi-hop, multi-domain Korean sources are primary unsolved subtasks, with Korean web structural peculiarities (semi-structured tables, local aliases) further increasing modeling difficulty.
• Calibration: ECEs vary widely, e.g., closed-source GPT-5.5 (31.86%) outperforms Korean-specialized LLMs even on Korean-centric domains.

5. Protocols and Reproducibility

• Evaluation Harness: Agents are run under a fixed interaction protocol (e.g., search_evals), enforcing hard budgets (e.g., 10 queries max), zero-shot processing (no in-context exemplars), and answer extraction via automated LLM parsing (Lee et al., 1 Jun 2026).
• Human Verification: For each query, at least two independently derived, evidence-grounded answer chains were confirmed. Gold spans cover all required logical steps.
• Offline Re-execution: The fixed corpus and qrels enable researchers to freely substitute tools or models and measure marginal gains under identical retrieval/evidence scenarios (Chen et al., 8 Aug 2025).

6. Cross-Lingual, Multimodal, and Extension Dimensions

• Cross-Lingual: K-BrowseComp-Verified forms the backbone of the XBCP (Cross-Lingual BrowseComp-Plus) benchmark, where queries are in English but evidence is distributed across twelve languages—including Korean, Chinese, Yoruba, Zulu—and performance drops by 15–23 percentage points in the cross-lingual setting, with oracle upper bounds always trailing monolingual by 5–13 points (Lu et al., 13 Jun 2026).
• Multimodal Extensions: Protocols for modality-tagged, irreducible reasoning sequences are realized in MM-BrowseComp, with each question’s “verified checklist” requiring explicit cross-modal step completion (Text / Image / Video) for credit (Li et al., 14 Aug 2025).
• Temporal and Live-Search: LiveBrowseComp extends the evaluation to temporally anchored, post-90-days-only facts, removing static answer coverage and forcing genuine evidence-driven discovery (Fan et al., 27 May 2026).

7. Implications and Benchmark Impact

K-BrowseComp-Verified provides a triage-grade, reproducible, fine-grained testbed for browsing agents and deep research models:

• Scientific Insight: By controlling search evidence, answer gold, and distractor selection, it allows rigorous ablation of retriever, agent, and orchestrator contributions.
• Progress Measurement: Both closed and open models show large performance gaps, especially in compositional Korean web search, cross-lingual, and multimodal settings.
• Methodological Recommendations: Extending evaluation to trajectory-level state tracking, parallel constraint intersection, and semi-structured parsing is critical. The benchmark exposes failures in model state management across long-horizon reasoning cycles and is especially valuable for research on memory-augmented or tool-aware agent architectures.
• Research Reproducibility: By decoupling search, retrieval, and reasoning, it allows the field to measure verifiable progress toward robust, language-aware browsing and retrieval agents.

References: (Chen et al., 8 Aug 2025, Lee et al., 1 Jun 2026, Lu et al., 13 Jun 2026, Li et al., 14 Aug 2025, Fan et al., 27 May 2026)