K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contexts
Abstract: Frontier model evaluations are shifting from foundational capabilities (e.g., instruction following and reasoning) toward compositional, agentic ones, but Korean agentic benchmarks remain scarce. We introduce K-BrowseComp, a web-browsing agent benchmark grounded in Korean contexts, consisting of 400 problems. The 300-problem K-BrowseComp-Verified subset is manually constructed and validated by native Korean speakers. On this subset, frontier LLMs, including GPT-5.5, DeepSeek-V4-Pro, and GLM-5.1, reach only 30.00--45.67\%, a substantial drop from BrowseComp, while Korean LLMs released through Korea's Proprietary AI Foundation Model program obtain only 0.00--10.33\%. We further construct a 100-problem synthetic split using hard few-shot exemplars and failure-mode-targeted generation to exploit the asymmetry between solving and creating web browsing problems. On the adversarially filtered synthetic diagnostic split, the strongest model reaches only 26.00\%, and we report this split separately as a targeted stress test. We publicly release our data and code.
Summary
- The paper introduces K-BrowseComp, a benchmark using 300 human-validated and 100 synthetic problems to assess agentic reasoning in Korean web contexts.
- The evaluation shows leading LLMs achieve below 50% accuracy on tasks, with closed models outperforming domestic alternatives by a wide margin.
- Analysis uncovers trajectory-level failures in multi-hop and parallel reasoning, highlighting a need for improved state tracking and constraint management.
K-BrowseComp: Evaluating Web-Browsing Agents in Korean Language and Cultural Contexts
Motivation and Benchmark Design
K-BrowseComp addresses the explicit scarcity of agentic benchmarks for Korean, where measurement of compositional and web-based reasoning lags behind English- and Chinese-centric evaluation. The benchmark is motivated by both sovereignty concerns—given the unique local institutions, linguistic phenomena, and web-search conventions—and the necessity of providing adversarially difficult, generalization-stressing evaluations as frontier LLMs saturate static leaderboards.
The benchmark comprises a 300-problem manually validated subset and a 100-problem synthetic diagnostic split. Human-written items were constructed by native speakers under strict guidelines emphasizing answer uniqueness, temporal stability, requirement for nontrivial evidence synthesis across the Korean web, and reasoning formats that either require multi-hop search or satisfaction of multiple intersecting constraints.
Problems are anchored in Korean cultural, institutional, media, and socio-technical domains, with explicit requirements that all evidence must be textual and accessible on the public web. In parallel, the synthetic subset leverages LLM-driven question proposal conditioned on observed model failure modes, using an adversarial acceptance filter for challenging examples where verifying the correct answer is substantially easier than producing it.
Figure 1: Example problems illustrating both parallel-branching (multi-source constraint) and multi-hop (sequential dependency) reasoning required by the benchmark.
Dataset Characterization
The validated subset contains 300 questions with a near-equal split between multi-hop (53.3%) and parallel-branching (46.7%) formats. Category distribution is dominated by Entertainment/Media and Transportation/Places, but substantial coverage of sports, science, education, literature, and public policy is also achieved. Crucially, both the reasoning structure and category balance of the synthetic split are preserved; however, the synthetic split features a significant domain shift with higher representation of science/IT/academia and longer questions.
Figure 2: Distribution of the 300 validated K-BrowseComp questions by category and reasoning type (multi-hop vs. parallel-branching).
Design choices ensure answer uniqueness, answer temporal stability, publicly verifiable evidence, and compositionality necessities—going beyond simple keyword matching and instead requiring persistent trajectory-level state tracking unavailable in existing Korean benchmarks limited to static NLU or factual recall.
Evaluation Protocol and Baseline Results
A comprehensive suite of both closed- and open-weight LLMs is evaluated using a controlled harness (search_evals) with a fixed query budget and uniform web search backend. The closed models (GPT-5.5, DeepSeek-V4-Pro, Gemini-3.1-Flash-Lite) and major open-weight models (GLM-5.1, Qwen3.6-35B-A3B, Gemma-4-31B-it) are compared against Korean open-weight models from government initiatives and domestic efforts.
Results indicate that:
- GPT-5.5 achieves 45.67% accuracy on the verified set, with DeepSeek-V4-Pro and GLM-5.1 both remaining just above 30%. This represents a substantial drop from reported English/Chinese-centric performance (>80% on BrowseComp/BrowseComp-ZH).
- Korean open-weight models perform significantly worse: K-EXAONE-236B-A23B achieves 10.33%, A.X-4.0 5.33%, HyperCLOVAX-SEED-Think-32B 2.33%, and some models fail to complete valid evaluation runs.
- The synthetic split remains harder still; no evaluated model exceeds 26% accuracy. In adversarial problems constructed to push on specific failure modes (especially those surviving filtering by GPT-5.4-mini and Gemini-3-Flash-preview), even GPT-5.5 and DeepSeek-V4-Pro exhibit sub-30% pass@1 rates.
Figure 3: Model accuracy and calibration error on the human-verified subset; best-performing models remain well below 50% accuracy and often yield high calibration error.
Error Taxonomy and Trajectory-Level Analysis
Manual annotation surfaces a rich taxonomy of trajectory-level failure modes, ranging from ineffective initial search (F1), search-access structure failures (F2), cross-source hopping (F3), semi-structured page parsing (F4), premature candidate commitment (F5), entity normalization (F6), fine-grained constraint tracking (F7), to numeric or intermediate reasoning failures (F8).
A significant finding is that many model failures occur after relevant web evidence has already been retrieved: the evidence is not consolidated into a maintained candidate ledger, constraint network, or correct role-binding, leading to locally plausible but globally incorrect or incomplete solutions.
Figure 4: Example failures: (left) candidate capture (premature commitment leads to consistent but ultimately incorrect downstream evidence exploitation), (middle) unmerged branches (parallel constraints not intersected), (right) misbound chains (incorrect role assignment disrupts compositional entity retrieval and state maintenance).
Post-retrieval failures are thus the dominant impediment, with models failing to consistently propagate intermediate state, enforce global constraints, or finalize valid outputs. Budget increases for search calls do not alleviate these deficits; incorrect trials frequently consume more search calls than correct ones.
Synthetic Problem Generation and Validity
Synthetic question generation leverages a pipeline that (1) provides a browsing-agent LLM with hard human-written exemplars or explicit instruction to target particular failure modes, and (2) applies multi-stage filtering to ensure both difficulty (by requiring multiple established high-performing models to fail) and answer well-specifiedness (through page-level reference extraction and searchability checks).
The synthetic split demonstrates significant distributional separation from the hand-crafted set at the embedding, category, and length level (ROC AUC = 0.88 for domain classifier), but maintains similar reasoning format proportions. In practice, failures in the synthetic split further concentrate in semi-structured parsing and constraint tracking—mirroring observed trajectory bottlenecks.
Implications and Future Directions
K-BrowseComp reveals persistent and deep agentic limitations in both global frontier and domestic Korean models, particularly regarding stateful multi-step reasoning, cross-source evidence integration, and robust handling of Korea-specific web search operations. It also indicates that post-instructional Korean specialization and continued language pretraining have negligible impact on browsing-agent ability unless trajectory-level mechanism improvements are introduced.
Practically, the results highlight that progress on static language understanding or synthetic benchmarks (non-agentic) does not translate to surfacing, synthesizing, and reasoning with open-domain web evidence, especially in culturally specialized and non-English environments. The persistent gap with closed proprietary APIs suggests substantial infrastructure, training, and evaluation deficiencies remain for Korean agentic LLMs.
The central challenge for future AI system design is building mechanisms that ensure robust, persistent candidate, constraint, and role tracking over multi-turn, open-domain browsing tasks. Without concrete advances in agentic memory and compositional reasoning—especially as exposed in settings requiring fine, Korea-centric web knowledge—the deployment of competent, safe, and practical Korean web agents will remain limited.
Conclusion
K-BrowseComp provides a high-precision, challenging benchmark for compositional web-browsing in Korean, quantifying a large performance gap for current agentic models and highlighting repeated failure modes beyond retrieval. Its adversarial synthetic split further stresses blind spots in model development, making it an essential tool for diagnosing, benchmarking, and driving future progress in non-English browsing agents and agentic AI more broadly (2606.02404).
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
What this paper is about (simple overview)
Imagine a scavenger hunt on the Korean web: you’re given a tricky question, you have to search multiple Korean websites, follow clues across pages, and combine facts to find one exact answer. This paper introduces K-BrowseComp, a new “scavenger hunt” test for AI web-browsing agents focused on Korean contexts. The goal is to see how well AI can search, read, and reason across Korean websites—not just answer simple questions.
What the researchers wanted to know
They set out to answer a few clear questions:
- Can today’s strongest AI models act like reliable Korean web helpers?
- Where do these models struggle the most when browsing Korean sites (for example, starting with the wrong search, misreading a table, or losing track of rules)?
- Can AI itself help create more tough, fair, and well-defined web-browsing questions that expose these weaknesses?
How they did it (methods in everyday language)
Think of two kinds of puzzles they built:
1) Human-verified puzzles (300 questions)
- Native Korean speakers wrote 300 realistic, hard web questions that:
- Have one clear answer that doesn’t change over time.
- Need multiple steps (like “find A, then use A to find B, then cross-check C”).
- Require visiting several Korean websites.
- The authors checked each question by hand to make sure the answer can be found online and is unique and stable.
2) AI-generated “stress test” puzzles (100 questions)
- The team noticed common ways AI fails (like mixing up names, misreading tables, or forgetting a rule). They then asked an AI browsing agent to write new questions that deliberately target these weak spots.
- To keep quality high, every AI-made question had to pass three filters: 1) Searchability test: If the answer pops up directly from a simple search result, it’s too easy and gets rejected. 2) Clarity test: Given the source page, the answer must be findable and unique. 3) Difficulty test: Strong models must actually fail it for the right reason (e.g., misreading a table), otherwise it gets rejected.
- Only 100 of 268 candidate questions passed all filters. These form a separate “diagnostic” split.
How models were tested
- All models used the same web-search tool and the same small budget of 10 searches per question (a fair “same gear” rule).
- The system checked the model’s final answer against the gold answer.
- They also measured “calibration” (how well the model’s confidence matches how often it’s correct).
What they found (main results and why they matter)
- Even top models struggled on Korean browsing:
- On the 300 human-verified questions, strong frontier models scored about 30%–46% correct. This is far lower than their performance on a similar English-focused benchmark.
- Korean open models scored much lower (around 0%–10%).
- The AI-made “stress test” was even tougher:
- The best model scored 26% correct on the 100-question split, and some models got 0%.
- Doing “more searching” didn’t fix it:
- On questions they got wrong, models typically used more searches, not fewer. That means the main problem wasn’t laziness—it was losing track of the information once they found it.
- The most common failure patterns were easy to understand:
- Stopping early or producing malformed answers (the model never cleanly finished).
- Starting with the wrong search direction (like using the wrong keywords).
- Getting stuck behind tricky pages (for example, content hidden behind tabs or expandable sections).
- Misreading semi-structured pages (tables, lists, rankings).
- Picking the wrong candidate from a list of similar items.
- Mixing up names or aliases (e.g., different spellings or old names).
- Forgetting to check all rules at once (e.g., picking something that fits clue A but violates clue B).
- Slipping on simple logic (dates, counts, comparisons).
- Key insight: Many failures happened after useful pages were already found. Models often:
- Commit too early to a “pretty good” answer and stop double-checking.
- Keep clues in separate branches and never merge them into a single consistent checklist.
- Mix up roles across steps (e.g., confusing which person, place, or institution each fact is about).
- Fail to produce a final, exact answer even when they’re close.
Why this matters:
- For Korean users, web tasks often depend on local language, sites, and search habits—this is where current models are weakest.
- For the global AI community, a benchmark in a different linguistic and cultural setting tests whether models can truly generalize beyond English-centric content.
What this could change (implications)
- Better AI web helpers for Korean users: The benchmark shows concrete weak spots that teams can fix—like handling Korean site layouts, tracking rules across steps, and stabilizing the final answer.
- Stronger generalization research: Since models already ace many older tests, challenging, culturally grounded tasks like these help push real progress.
- Smarter data creation: The paper shows a practical way to use AI to generate new, tough questions by targeting known failure patterns, then carefully filtering for clarity and fairness.
- Community resource: The authors are releasing the data and code so researchers can build and test better browsing agents.
A quick note on limits
- The verified set is 300 questions and leans toward certain topics (like entertainment and places).
- All tests used one browsing setup and one search budget; different tools or budgets might change scores.
- The web changes over time—some links or rankings may shift, so ongoing maintenance is needed.
Bottom line
K-BrowseComp is a new, carefully checked challenge that tests whether AI can be a reliable Korean web-browsing assistant. Today’s strongest models still struggle—often not because they can’t find pages, but because they lose track of details across steps. By pinpointing where and how they fail, this benchmark gives a clear roadmap for building better, more trustworthy web agents for Korean users and beyond.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
The paper leaves several concrete gaps and open questions that future work can address:
- Dataset scale and balance
- How does model performance vary across domains underrepresented in the verified set (e.g., science/IT, public policy, government services), and does rebalancing the category distribution change rankings?
- What is the impact of increasing the verified set size (e.g., to 1k–5k items) on score stability, per-category reliability, and error taxonomy coverage?
- Human baselines and difficulty calibration
- What are human success rates, time-to-solution, and failure patterns for native Korean speakers under the same harness and time budgets, and how do they compare to model performance?
- Can the benchmark establish a calibrated difficulty scale (e.g., “2-hop/4-constraint/hidden-table”) that correlates with human and model error rates?
- Search backend and harness sensitivity
- How sensitive are results to the search backend (e.g., Naver, Google, Bing, Kakao), location, or IP/locale settings commonly used by Korean users?
- Do conclusions change when using different browsing agents/harnesses (e.g., headless browser with click/scroll/form controls vs. the current Perplexity-backed toolset)?
- Budget, decoding, and pass@k effects
- How do accuracy and failure modes change with larger search budgets (e.g., 20–50 calls), different stopping criteria, or multiple independent runs (pass@k)?
- What is the trade-off between abstention policies and expected calibration error (ECE) under varying decoding temperatures and planning strategies?
- Evidence access and dynamic content
- To what extent do tasks require dynamic page interactions (JavaScript-rendered tables, pagination, client-side search) that the current harness cannot perform, and how much does enabling such actions reduce F2 failures?
- How often do answers depend on non-textual artifacts (e.g., PDFs, images), and should the benchmark admit OCR/PDF parsing tools explicitly?
- Measurement validity and grader reliability
- What is the error rate of the LLM-based answer extractor and matcher (GPT-5.4-mini) relative to human judges or rule-based normalization, and how much does grader bias affect reported accuracy/ECE?
- Can the benchmark include evidence-justification scoring (e.g., verifying cited URLs/snippets) to penalize unsupported correct strings and detect hallucinated provenance?
- Temporal robustness and maintenance
- How stable are answers and evidence over time (e.g., 3–12 months), and can an automated monitor flag items whose evidence or rankings changed enough to break uniqueness or accessibility?
- What protocols (e.g., versioning, scheduled revalidation) best preserve reproducibility for a web-grounded benchmark?
- Synthetic split validity and biases
- How much does adversarial filtering against specific target models (gpt-5.4-mini, gemini-3-flash-preview) bias item selection and difficulty for other models, and what changes when targets are rotated or expanded?
- What is the human-verified quality of the synthetic items (uniqueness, clarity, stability), and can inter-annotator agreement be reported for a sampled subset?
- How transferable are gains from training on the synthetic split to the verified split, and vice versa (i.e., does targeted stress training overfit to F4/F7)?
- Failure-mode coverage and granularity
- Are there Korean-specific sub-modes (e.g., Hangul–romanization mismatch, Hanja disambiguation, spacing/particle variation, honorific forms) that deserve separate categories beyond the current F6?
- Can the benchmark provide per-model, per-category, and per-failure-mode distributions to identify where each model struggles most, not just aggregate accuracy/ECE?
- Cross-lingual and bilingual behavior
- How do agents perform when allowed bilingual searches (Korean↔English) or cross-lingual hops (e.g., Korean query → English source → Korean confirmation), and does this alleviate F3/F6 errors?
- Does prompting in Korean vs. Korean+English mixed scripts change retrieval quality, query formulation, or success rates?
- Contamination and prior exposure
- Are any gold answers or question phrasings memorized or near-duplicates in pretraining corpora, and how much would contamination (if present) inflate apparent ability on a web-browsing task?
- Fine-grained, stepwise evaluation
- Can the benchmark include gold intermediate entities/constraints and graded partial credit (e.g., % constraints satisfied) to disentangle retrieval, parsing, and integration errors more precisely?
- Would publishing canonical gold trajectories (or multiple acceptable paths) enable consistent stepwise scoring and agent training, and how to protect against overfitting?
- Tool-use and protocol reliability
- How much do failures derive from protocol mismatches (malformed tool calls, output format) versus reasoning/content issues, and can standardized conformance tests be added?
- Do explicit memory tools (candidate tables, constraint checklists, role-binding maps) reduce F5/F7/F3 errors, and which memory representations are most effective?
- Category- and source-level analyses
- Which Korean site families (e.g., government portals, entertainment databases, education sites, Naver encyclopedia, forums) systematically trigger F2/F4 errors, and what tooling (schema-aware parsers, site-specific adapters) mitigates them?
- Are there systematic performance gaps across categories (e.g., Entertainment vs. Science/IT) that persist after rebalancing and harness changes?
- Generalization beyond Korean
- Do methods that help on K-BrowseComp transfer to other regional-language browsing benchmarks (e.g., Japanese, Vietnamese), and what failures are truly language-specific vs. agent-general?
- Safety, ethics, and licensing
- Are there risks that queries or evidence encourage scraping of pages that restrict automated access or include sensitive personal data, and should the benchmark codify exclusion/usage policies?
- Can the benchmark include checks for unsafe browsing behavior (e.g., following harmful links) and measure adherence to safe exploration guidelines?
- Releasing stronger baselines and interventions
- Which concrete agentic interventions (planning modules, explicit constraint trackers, semi-structured parsers, entity normalizers, bilingual query planners) measurably reduce the dominant post-retrieval failures identified, and by how much under controlled ablations?
Practical Applications
Immediate Applications
The following applications can be deployed now by leveraging the K-BrowseComp benchmark, its failure taxonomy, and the released code/data.
- Benchmark-driven QA and model selection for Korean web agents — sectors: software, search, consumer AI, enterprise AI
- What: Use K-BrowseComp-Verified (300 items) to compare browsing agents (closed/open LLMs) on pass@1 accuracy, expected calibration error (ECE), and search-call efficiency before product rollouts.
- Tools/workflows: Integrate the dataset with search_evals-based harnesses; spin up CI dashboards that track accuracy/ECE over time; add alerts for regressions on categories (e.g., entertainment/media vs. public policy).
- Assumptions/dependencies: Stable access to web search APIs (e.g., Perplexity Search) and model inference; adherence to evaluation harness protocols; periodic revalidation to handle web drift.
- Failure-mode diagnostics to guide agent improvements — sectors: software, robotics (information-seeking agents), academia
- What: Use the F0–F8 taxonomy (e.g., constraint-tracking, cross-source hopping, semi-structured parsing) to localize error hotspots and prioritize fixes like candidate-table tracking, role-binding persistence, or improved table parsing.
- Tools/workflows: Instrument trajectories to auto-label failures; add unit tests targeting F4/F7 (dominant errors); implement “candidate capture” checks; implement answer-finalization validators.
- Assumptions/dependencies: Agent must expose step-wise logs and tool calls; dev teams need bandwidth to patch agent planning/memory and parsing modules.
- Red-teaming and QA for Korean-market assistants — sectors: consumer AI, customer support, e-commerce, media
- What: Use verified and synthetic splits to stress-test assistants on Korean websites, emphasizing multi-hop retrieval, semi-structured data, and culturally grounded queries.
- Tools/workflows: Integrate as a red-team pack in pre-release testing; flag overconfidence via ECE and force abstention/fallback when confidence is low.
- Assumptions/dependencies: Product teams must support abstention policies and human escalation routes; compliance with site ToS and rate limits.
- Procurement and vendor evaluation in the public sector — sectors: policy, e-government, education
- What: Government agencies and universities can benchmark contractor systems on K-BrowseComp to set minimum service-level thresholds for Korean browsing tasks.
- Tools/workflows: RFPs referencing pass@1/ECE thresholds; auditing of tool-call conformance and trajectory completeness.
- Assumptions/dependencies: Agreement on standardized harness and budget (search calls, tools); public disclosure of evaluation conditions.
- Curriculum, assignments, and reproducible research — sectors: academia, education
- What: Use K-BrowseComp as a lab or course module on agent evaluation, calibration, and multilingual web retrieval.
- Tools/workflows: Jupyter labs with search_evals harness; student projects on fixing F3/F7 or designing constraint-aware memory modules.
- Assumptions/dependencies: Access to at least one capable LLM API or open-weight model; stable API keys and cost budgeting.
- Targeted data curation for fine-tuning and RLAIF — sectors: software, ML platforms
- What: Convert failure clusters into supervised traces and feedback signals (e.g., reward models that penalize premature commitment or misbound chains).
- Tools/workflows: Mine trajectories for negative/positive exemplars; construct “verify-before-finalize” training curricula; integrate with tool-use instruction tuning.
- Assumptions/dependencies: Legal/ethical use of web content; guardrails against contamination of the benchmark test set.
- Agent prompt/policy tuning for better finalization — sectors: software, search
- What: Improve answer stabilization, termination criteria, and abstention policies using observed F0 (malformed or incomplete trajectories).
- Tools/workflows: Add explicit “evidence sufficiency checks” and “unique-answer checks”; require a final compact exact answer backed by URL citations.
- Assumptions/dependencies: Agent must support multi-stage reasoning and self-verification; product acceptance of slightly higher latency for better reliability.
- Website-side accessibility audits for Korean content — sectors: media, public institutions, publishers
- What: Use failure analyses (F2/F4) to advise site owners on making semi-structured content (tables, registries) more machine-accessible.
- Tools/workflows: Run pilot agents against institutional pages; recommend schema.org markup, consistent headers, and stable anchors.
- Assumptions/dependencies: Site owners’ willingness to modify templates; balance between SEO and agent-accessible structure.
Long-Term Applications
These opportunities need further research, model improvements, or scaled infrastructure.
- Production-grade Korean browsing agents for public services — sectors: policy, e-government, healthcare, education
- What: Build agents that can reliably retrieve regulations, procedures, health advisories, and admissions information from Korean websites, with calibrated abstention.
- Tools/products: State-tracking modules (candidate sets, constraints); verifiable tool-use; human-in-the-loop fallback; audit trails for procurement.
- Assumptions/dependencies: Stronger LLMs on Korean text; robust tool protocols; legal compliance for scraping/citation; ongoing benchmark maintenance.
- Automated, failure-mode-targeted benchmark generation at scale — sectors: software, evaluation, MLOps
- What: Generalize the synthetic pipeline to continuously produce adversarial, verifiable items for regression testing across verticals and locales.
- Tools/products: A “Benchmark Forge” service that ingests seed pages + failure taxonomy and emits vetted tasks; CI integration for weekly stress suites.
- Assumptions/dependencies: Access to capable proposer/tester models; compute budget; human spot checks; drift and contamination monitoring.
- Cross-locale agentic evaluation suite (beyond Korean) — sectors: global AI evaluation
- What: Port methodology to other languages/regions to test out-of-distribution generalization and mitigate “soft contamination.”
- Tools/products: Multi-regional K-BrowseComp variants; shared leaderboards with ECE and trajectory analytics; standardized harnesses.
- Assumptions/dependencies: Native annotators for validation; localized failure taxonomies; multilingual retrieval backends.
- Semi-structured parsing and extraction libraries tuned for Korean web — sectors: software, search, data platforms
- What: Create robust parsers for tables, rankings, and institutional pages common in Korean sites; wrap high-value domains (e.g., public records, sports, education) with adapters.
- Tools/products: Domain adapters and schema mappers; hybrid DOM + vision parsing; lightweight site-specific plugins for agents.
- Assumptions/dependencies: Site stability; maintenance burden as layouts evolve; respect for site ToS.
- Memory and state-maintenance architectures for browsing — sectors: software, robotics
- What: Develop explicit representations for candidate sets, constraints, and role bindings to reduce F3/F5/F7 failures after retrieval.
- Tools/products: Constraint-checking modules; entity-role binding trackers; planners that interleave verify-and-prune steps; differentiable “candidate tables.”
- Assumptions/dependencies: Research into agent cognition and planning; acceptable compute/latency overhead; compatibility with current tool APIs.
- Safety, calibration, and abstention standards for agent deployment — sectors: policy, finance, healthcare
- What: Tie ECE and abstention rates on K-BrowseComp to risk controls; certify agents for high-stakes domains only if calibration thresholds are met.
- Tools/products: Policy frameworks mapping ECE to allowed autonomy levels; standardized “evidence sufficiency” tests and trace audits.
- Assumptions/dependencies: Agreement on metrics and thresholds; third-party auditors; logging and reproducibility requirements.
- Verticalized browsing agents for Korean content — sectors: finance, media/entertainment, travel, local commerce, sports
- What: Specialized agents that navigate domain-specific Korean sources (e.g., KBO stats, regional travel advisories, media catalogs) with high reliability.
- Tools/products: Domain-specific seed-page repositories; constraint libraries; KPI dashboards (e.g., recall@domain, constraint-satisfaction rate).
- Assumptions/dependencies: Domain expertise to encode constraints; partnerships for API access where available; maintenance as sources change.
- Research on retrieval backend sensitivity and budget policies — sectors: academia, software
- What: Systematically study how different search backends and call budgets affect success and failure composition on Korean web tasks.
- Tools/workflows: A/B testing harness swapping search providers; dynamic budget policies; meta-learning to allocate search calls adaptively.
- Assumptions/dependencies: Multiple backend contracts; replicable conditions to attribute gains to models vs. retrieval.
- Continuous benchmark maintenance and governance — sectors: evaluation, policy
- What: Establish a community process for revalidating items, ensuring temporal stability, and handling takedowns/changes.
- Tools/workflows: Versioned releases; item-level metadata (access dates, mirrors); automated link rot checks; dual gold-standard verification.
- Assumptions/dependencies: Funding and community stewardship; legal review for dataset distribution; infrastructure for archival snapshots.
Glossary
- Adversarial difficulty: A construction filter that requires candidate questions to defeat target models, ensuring they are challenging. "The third tests adversarial difficulty"
- Adversarial question: A deliberately challenging item crafted to induce model errors or target specific weaknesses. "is tasked with drafting a single adversarial question."
- Adversarially filtered: Selected by screening out items that models can answer, yielding a harder evaluation set. "On the adversarially filtered synthetic diagnostic split"
- Agentic: Pertaining to autonomous agents that plan, act, and use tools over multiple steps. "compositional, agentic ones"
- Constraint-tracking failure: A failure mode where the model does not consistently enforce all constraints across steps. "F7: Constraint-tracking failure (unmerged evidence branches)."
- Cross-source hopping failure: A failure mode where the model fails to connect evidence across different sources or contexts. "F3: Cross-source hopping failure (misbound evidence chains)."
- Data contamination: Leakage of benchmark content into training data, leading to overestimated performance. "Recent work on data contamination and benchmark generalization shows that evaluation results can overstate model ability"
- Expected calibration error: A metric summarizing the gap between model confidence and accuracy across bins. "we report expected calibration error over five equal-width confidence bins"
- Explicit abstention: When a model deliberately declines to answer, counted as failure in adversarial filtering. "where failure includes both an incorrect answer and an explicit abstention."
- Failure-mode-targeted generation: Creating new items by explicitly aiming at known error categories to increase difficulty. "using hard few-shot exemplars and failure-mode-targeted generation"
- Frontier models: The most capable, cutting-edge LLMs at the time of evaluation. "Frontier models increasingly saturate existing benchmarks"
- Hard few-shot exemplars: Especially difficult examples provided as demonstrations to guide model generation of new items. "hard few-shot exemplars"
- In-distribution and out-of-distribution evaluation: Testing within the training-like domain vs. on novel domains, respectively. "blurring the boundary between in-distribution and out-of-distribution evaluation"
- Information asymmetry: A task property where generating or verifying an answer is easier than solving from scratch. "A defining property of browsing tasks is their information asymmetry"
- Intermediate reasoning failure: Errors in basic operations like counting, ordering, or comparison during multi-step reasoning. "Intermediate reasoning failure"
- Multi-hop reasoning: Solving questions by chaining several dependent steps across sources. "requires multi-hop reasoning"
- Multi-turn interaction: Dialog or iterative tool use over multiple exchanges to accomplish a task. "and multi-turn interaction"
- Multilingual sentence-transformer model: A neural encoder that embeds text from multiple languages into a shared vector space. "We further embed all questions with a multilingual sentence-transformer model"
- Open-domain QA: Question answering that relies on retrieving information from broad, unrestricted sources. "the core components of open-domain QA"
- Open-weight models: Models whose parameters are publicly available for use and fine-tuning. "The open-weight baselines include DeepSeek-V4-Pro, GLM-5.1, Qwen3.6-35B-A3B, and Gemma-4-31B-it"
- Parallel-branching: Reasoning that aggregates multiple independent constraints gathered in parallel. "parallel-branching (i.e., gathering information from multiple websites)"
- Pareto frontier: The set of models not dominated on multiple metrics (e.g., accuracy vs. calibration). "The dashed line marks the Pareto frontier."
- Pass@1: Probability that the first attempt produces a correct answer; here, a single-run accuracy measure. "We report the single-run accuracy, which corresponds to pass@1 in this setting."
- Perplexity Search API: An external web search service used as the retrieval backend for agents. "All runs use the same external retrieval pipeline with the Perplexity Search API."
- ROC AUC: Area under the Receiver Operating Characteristic curve, summarizing separability of two classes. "with an ROC AUC of 0.8873."
- Search-access structure failure: Errors due to inability to access content hidden by complex page structures. "Search-access structure failure"
- Search-call budget: A cap on how many search queries an agent can issue per question. "a budget of 10 search calls per question"
- Search-result selection failure: Retrieving relevant evidence but choosing the wrong source or candidate. "Search-result selection failure"
- Semi-structured parsing failure: Misinterpreting tables, lists, rankings, or databases on web pages. "Semi-structured parsing failure"
- Sparse entity normalization failure: Inability to align rare or variant entity names to the correct canonical form. "Sparse entity normalization failure"
- Synthetic split: A set of benchmark items generated by models (not humans) for diagnostic evaluation. "We further construct a 100-problem synthetic split"
- Targeted stress test: A deliberately hard subset designed to probe specific weaknesses under controlled conditions. "we report this split separately as a targeted stress test."
- Tool calling: Invoking external tools (e.g., web search) from within a model’s reasoning process. "instruction following, reasoning, and tool calling"
- Trajectory-level failure-mode taxonomy: A structured categorization of where and how multi-step agent runs fail. "we define a trajectory-level failure-mode taxonomy, summarized in Table~\ref{tab:failure_modes}."
- Verifiability structure: Designing items so that once the evidence path is known, the answer can be uniquely and publicly verified. "This pipeline uses the verifiability structure of browsing questions"
Collections
Sign up for free to add this paper to one or more collections.
