K-BrowseComp: Korean Web-Browsing Benchmark

• K-BrowseComp is a benchmark designed to assess Korean web-browsing agents, emphasizing multi-hop reasoning, evidence synthesis, and constraint tracking.
• It comprises 400 problems split into human-verified and synthetic diagnostic sets, ensuring rigorous, reproducible tests tailored to the Korean web context.
• Baseline results indicate significant performance drops in LLMs on Korean tasks compared to English benchmarks, highlighting challenges in trajectory-level state maintenance and parsing.

K-BrowseComp is a large-scale, Korean-grounded web-browsing agent benchmark specifically devised to evaluate the end-to-end capabilities of LLM based agents in information retrieval, evidence synthesis, constraint tracking, and multi-step reasoning over the open Korean web. Reflecting a paradigm shift in agent evaluation from foundational competence to compositional agentic capability, K-BrowseComp systematically measures how well modern LLM agents can follow multi-turn instructions, utilize search tools, and maintain trajectory-level state under authentic, linguistically complex, and culturally specific web conditions (Lee et al., 1 Jun 2026).

1. Design Motivations and Benchmark Structure

K-BrowseComp was introduced to fill a critical gap in the evaluation of agentic search systems for the Korean language. Existing benchmarks such as BrowseComp (English) and BrowseComp-ZH (Chinese) revealed substantial performance degradation in non-English contexts, due to local site structures, terminology, and information access conventions. No previous resource enabled systematic, controlled, and reproducible measurement of web-browsing agent performance on the public Korean web.

K-BrowseComp consists of 400 questions, divided into:

• K-BrowseComp-Verified (300 problems): Manually authored by native Korean speakers, human-verified for answer uniqueness, evidence stability, and multi-hop or parallel-branching logical structure.
• Synthetic Diagnostic Split (100 problems): Machine-generated, failure-mode-targeted, and adversarially filtered to specifically tax known agentic deficiencies.

Questions are balanced across two principal reasoning formats:

• Multi-hop: Agents must retrieve and maintain intermediate values or entities through at least four sequential evidence collection steps.
• Parallel-branching: Agents must identify solutions crossing at least four independent constraints, intersected over shared candidate sets.

The benchmark spans ten domains, with particular emphasis on Entertainment & Media, Places & Regions, Education/Exams, and Science/IT/Academia.

2. Dataset Construction and Validation Procedures

The foundation of K-BrowseComp-Verified is rigorous, human-centric data construction:

• Annotator Guidelines: Problems require searching Korean textual sources, enforce temporal stability, demand unique short answers, and forbid reliance on private or transient artifacts.
• Question Validation: Every question undergoes manual review for accessibility, uniqueness, temporally invariant answers, explicit constraint chains (an “expected_chain” field documents the intended web navigation route), intermediate checklists, and linguistic accuracy. Items lacking clear evidence trails or exhibiting ambiguity are removed or revised.
• Synthetic Diagnostic Generation: Eight agentic failure modes are systematically targeted, including ineffective initial search, semi-structured parsing, constraint/progress tracking, cross-source hopping, and intermediate reasoning errors. Prompt-based backward generation with hard few-shot exemplars, followed by three-stage filtering (non-surfaceable answer, well-formedness, adversarial solver failure), yields a pool where only ∼37.3% of candidate diagnostics survive.

This approach produces a high-fidelity dataset unavailable to naive translation or automatic extraction, ensuring that both core and adversarial splits are representative of realistic Korean web challenges, not English-centric artifacts (Lee et al., 1 Jun 2026).

3. Task Format, Tool Constraints, and Evaluation Protocol

Each K-BrowseComp entry contains:

• "problem": A Korean-language instruction or query.
• "answer": Unique, concise gold answer for automated grading.
• "type": Indicates multi-hop or parallel logical form.
• "expected_chain": List of stepwise proof obligations and supporting sources.
• "checklist": Explicit intermediate answer slots for stepwise verification.
• "korean_specific_keywords": Lexical cues needed for web navigation.

Agents are provided up to 10 search tool calls (via Perplexity Search API), alternating with internal reasoning turns. At each step, the agent may update constraints, tracking both candidates and role bindings, until a final answer is committed.

Primary Metric:

$$\text{Accuracy (Pass@1)} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}\bigl(\hat y_i = y_i\bigr)\times100\%$$

where $\hat y_i$ is the agent’s answer and $y_i$ is the gold answer, judged by a GPT-5.4-mini grader.

Calibration Analysis:

$$\mathrm{ECE} = \sum_{m=1}^M \frac{|B_m|}{N}\,\bigl|\mathrm{acc}(B_m) - \mathrm{conf}(B_m)\bigr|$$

Confidence calibration is quantified over five bins: lower ECE values correspond to more reliable confidence estimation.

No other statistical significance or variance measures are reported (Lee et al., 1 Jun 2026).

4. Baseline Results and Failure Mode Taxonomy

Performance on K-BrowseComp reveals a pronounced drop relative to English-centric analogs:

Model	Access	Verified Acc. (%)	Synthetic Acc. (%)	ECE (%)
GPT-5.5	Closed	45.67	26.00	31.86
DeepSeek-V4-Pro	Open	30.00	22.00	17.72
GLM-5.1	Open	30.67	19.00	27.07
Qwen3.6-35B-A3B	Open	12.00	15.00	47.89
Korean open-weight LLMs	Open	0.00–10.33	0.00–13.00	—

• English-centric APIs (e.g., GPT-5.5, DeepSeek-V4-Pro) degrade from ∼83–84% on standard BrowseComp to 30–45% on K-BrowseComp-Verified.
• Korean LLMs released from the national foundation model program score 0–10.33%, illustrating insufficient cross-lingual transfer.
• On the synthetic diagnostic split, no model exceeds 30% accuracy.

Diagnostics reveal that the synthetic split disproportionately stresses:

• Semi-structured parsing (59/100): Agents extract from wrong fields despite correct page access.
• Constraint tracking (21/100): Parallel conditions not maintained across trajectories.
• Access structure (14/100): Failure to navigate hierarchical or nested site organization.

Low accuracy is traced not to insufficient search budget (models routinely exhaust all 10 calls), but to fragility in tracking candidates, constraint state, and role assignment through multi-turn logical flows (Lee et al., 1 Jun 2026).

5. Insights on Agentic Bottlenecks and Surface Shifts

K-BrowseComp uniquely surfaces trajectory-level deficiencies not measurable in English benchmarks or single-hop QA:

• Parallel and multi-hop reasoning: Simple pattern or memory-based solutions are insufficient; agents must coordinate multiple constraints over disjoint sources and maintain intermediate values.
• Diagnostic surface shift: The synthetic split demonstrates a marked increase in question length and surface complexity, with a domain distribution shift (Entertainment & Media drops 36.3%→9.0%, Science/IT/Academia rises 6.7%→33.0%). Separability (ROC AUC ≈0.887) confirms a substantial challenge increment.
• Stress-test success indicators: High failure rates appear even when all relevant evidence is retrieved, indicating that error is largely due to workflow design—state maintenance, parsing, and constraint intersection.

This confirms that current agentic LLM architectures and search pipelines are poorly adapted for non-English, multi-hop web-browsing tasks, with a dominant bottleneck in trajectory-level memory and reasoning (Lee et al., 1 Jun 2026).

6. Recommendations, Reproducibility, and Broader Impact

K-BrowseComp is publicly released, including all source data, validation chains, expected trajectories, and evaluation code (MIT License, [GitHub: prometheus-eval/K-BrowseComp]). Evaluation is performed using Perplexity API, deep-research agent scaffold, and a fixed search call budget. Grading is automated with GPT-5.4-mini.

Useful recommendations for benchmark use and future extensions include:

• Incorporating adaptive budgeting, hybrid retrieval, and multi-agent state-sharing to alleviate current bottlenecks.
• Continuous refresh of question pools to cover emergent web phenomena and maintain surface challenge.
• Systematic adoption of adversarial and failure-mode-driven question generation for diagnostic coverage.
• Careful design of checklists and explicit reasoning chains to enable granular agent performance attribution.

While K-BrowseComp-Verified anchors the core of Korean web-agent evaluation, its synthetic split enables the field to measure progress on targeted trajectory-level weaknesses invisible in standard cross-lingual adaptation. The benchmark sets a rigorous standard for both model improvement and large-scale reproducibility (Lee et al., 1 Jun 2026).