WebRetriever Benchmark

• WebRetriever is a large-scale benchmark designed to evaluate web agents on live websites using 800 sites and 1,550 tasks.
• It distinguishes evaluation protocols by measuring navigation, knowledge-assisted interaction, and end-to-end task completion via the NavEval framework.
• Empirical findings highlight low success rates for present agents, underlining the gap between page arrival and real task completion.

WebRetriever is a large-scale benchmark for realistic online evaluation of web agents on live websites. It was introduced to measure capabilities that matter in deployment rather than only benchmark-style navigation success, and combines three elements that are usually separated: broad website coverage, a judge that uses richer interaction evidence than screenshots alone, and evaluation protocols that distinguish navigation, knowledge-assisted interaction, and end-to-end task completion with information extraction. The benchmark contains 800 websites and 1,550 tasks, and its core claim is that navigation success alone is an insufficient predictor of real-world application effectiveness (Dong et al., 7 Jul 2026).

1. Definition and conceptual scope

WebRetriever is designed for the evaluation of web agents that must operate on live websites under real interactive conditions. The benchmark is motivated by four shortcomings in prior work: limited scale and domain diversity, screenshot-only or shallow LLM-as-Judge evaluation, overreliance on navigation success metrics, and weak coverage of deployment requirements such as knowledge-assisted interaction and information extraction (Dong et al., 7 Jul 2026).

The benchmark therefore treats web-agent evaluation as a three-part problem. First, an agent must navigate unfamiliar sites. Second, it may need to use external operational knowledge or documentation. Third, it may need to complete an end-to-end task that requires extracting the correct information rather than merely reaching a plausible page. This framing is central to WebRetriever’s design: the benchmark is not only about clicking through interfaces, but about whether the agent can finish the user’s job.

A recurrent misconception addressed by the benchmark is that final-page arrival or visual plausibility is enough. WebRetriever explicitly rejects that assumption. The paper argues that subtle errors in search queries, filters, sort orders, or object selection can invalidate a result even when the final page looks correct. This is why the benchmark emphasizes fine-grained interaction semantics and not only screenshot similarity.

2. Benchmark construction and governance

Website selection combines breadth with domain realism. The benchmark uses SimilarWeb traffic data as a baseline, selecting the top 30 sites by traffic within each sector, and then adds specialized vertical sites from authoritative internet research institutions such as 199it, especially for business intelligence coverage (Dong et al., 7 Jul 2026). The paper states that WebRetriever spans eight core sectors and lists Technology / Internet, Business / Finance, Education / Research, Culture, Entertainment / Travel, Life Services, Healthcare, Industrial Manufacturing, and Public Services. The abstract further characterizes the scope as covering consumer, professional, and enterprise sectors. Supplementary material adds multilingual and geographic breadth, including websites in English, Chinese, Spanish, and French, and websites from the US, China, Europe, Asia, Africa, and Latin America.

Task creation was carried out by a 14-member annotation team including domain experts, mid-level managers, senior data analysts, and university students. Tasks were refined under three principles: uniqueness, stability, and logical authenticity. Difficulty is categorized by the number of human reference action steps, denoted ${Step}_{n}$: easy if $n < 6$, medium if $6 \leq n \leq 15$, and hard if $n > 15$.

The benchmark uses a five-stage governance pipeline: task design, annotation, cross-validation, quality review, and final verification. It also enforces strict annotator–reviewer separation, ran 12 calibration sessions, and applies a sequential multi-stage quality-control process rather than standard parallel inter-annotator agreement labeling. Every task underwent three independent review stages, and revision was mandatory whenever disagreements arose. Reported correction rates were 30%, 26%, and 27% across the three review stages, and 54% of tasks received at least one correction. This unusually heavy review load is part of the benchmark’s attempt to make live-web evaluation auditable despite website volatility.

Operationally, each task starts from a predefined website entry URL, and search engine access is disabled. That restriction is important: it prevents agents from bypassing within-site interaction by delegating the problem to generic web search.

3. NavEval and fine-grained automatic judgment

WebRetriever’s automatic evaluator is NavEval (“Navigation Evaluation”), an LLM-as-Judge framework that uses richer interaction context than screenshots alone. Formally, given task description $T$, website URL $U$, a sequence of web requests $R = (r_1, r_2, \ldots, r_n)$, a sequence of executed actions $A = (a_1, a_2, \ldots, a_n)$, and final screenshot $I$, NavEval outputs a binary success judgment

$$P = \text{NavEval}(T, U, R, A, I), \qquad P \in \{\text{True}, \text{False}\}.$$

It also applies a rule-based request filtering step,

$n < 6$0

where the filtering matches requests against the task URL at the subdomain level, removes task-irrelevant or random fields, normalizes structured payloads, and removes invalid entries (Dong et al., 7 Jul 2026).

The evaluator’s methodological distinction is that screenshots are used for state verification, while network requests and URL trajectories are used for operation verification. This enables explicit checking of whether the agent actually executed the right search query, applied the right filters, selected the right item, or submitted the right operation. The supplementary prompt instructs the judge to decompose a task into atomic requirements and verify each requirement using network requests, URL trajectory, and final screenshot jointly.

The benchmark defines Success Rate (SR) and Human Agreement Rate (AR). AR compares automated predictions $n < 6$1 to human labels $n < 6$2 over a task set, and is defined as

$n < 6$3

The use of AR is central because WebRetriever is not only proposing a benchmark; it is also proposing a new evaluator that is meant to align with human judgment more closely than prior screenshot-dominated approaches.

4. The three evaluation protocols

A major contribution of WebRetriever is the separation of evaluation into three complementary protocols rather than a single navigation score.

Protocol	Capability assessed	Tasks
Protocol I	Navigation proficiency	1,000
Protocol II	Knowledge-assisted interaction	1,000
Protocol III	End-to-end task completion with information extraction	100

There are 550 overlapping tasks between Protocols I and II, differing only in whether operational documentation is provided. Protocol I measures basic navigation proficiency from the entry URL using only the task description. Protocol II evaluates whether the agent can use operational documentation/manuals to navigate more effectively on unfamiliar sites. Protocol III is the most deployment-oriented setting: the agent must not only reach the relevant page, but also retrieve and extract the correct answer.

The benchmark’s documentation for Protocol II is itself constructed through a multi-step pipeline: agents generate trajectories, NavEval filters successful ones, annotators refine the trajectories, and an LLM generates concise operational documentation from the refined trajectories. This protocol is meant to simulate realistic settings in which agents can consult internal knowledge bases, SOPs, or workflow manuals.

Protocol III is deliberately designed around “Deep Research–style” tasks and is governed by three principles: Authoritativeness, Interaction Necessity, and Determinism. The supplementary material describes representative scenario families such as Document Extraction, Form Interaction, Multi-source Comparison, Complete Data Retrieval, and Multi-dimensional Chart. In the main results table, Protocol III is reported with human evaluation.

Illustrative tasks show the protocol differences clearly. Protocol I examples include searching for a Kaggle dataset with specified usability constraints or performing advanced search in CiNii Books with publication-year and full-text filters. Protocol II examples include using va.gov to find guide-dog information for blind veterans or searching komoot.com with multiple hiking filters. Protocol III examples include identifying French universities in the top 0.2% on CWUR, finding the city in Europe with the lowest office rent and expected increase on CBRE, or extracting prerequisite logic from the MIT course catalog.

5. Empirical findings

WebRetriever’s main empirical result is that current web agents perform poorly once the evaluation is broadened from navigation to deployment-oriented task completion. Human-assessed average success rates are 21.1% on Protocol I, 29.2% on Protocol II, and 11.8% on Protocol III (Dong et al., 7 Jul 2026).

At the agent level, Gemini-2.5-Pro (Computer-Use) achieves the highest reported human success rates: 37.1 on Protocol I, 45.2 on Protocol II, and 21.0 on Protocol III. Claude-4.5 (Computer-Use) reports 28.1, 36.3, and 16.0 respectively. Browser-Use reports 24.0, 31.6, and 11.0. UI-TARS-1.5 reaches 16.5, 24.8, and 8.0. Agent-E reaches 11.6, 20.4, and 9.0, while SeeAct reaches 9.2, 17.1, and 6.0. These numbers are low even before considering production reliability thresholds.

The difference between Protocol II and Protocol I quantifies the value of operational knowledge. For example, Gemini-2.5-Pro improves from 40.9 to 49.2 when documentation is added in the comparison reported by Table 5, a gain of +8.3, while Claude-4.5 improves from 31.3 to 39.7, a gain of +8.4. Conversely, removing documentation from Protocol II reduces Gemini-2.5-Pro from 50.1 to 41.4 and Claude-4.5 from 40.1 to 31.9. The benchmark therefore supports a precise conclusion: documentation helps substantially, but does not close the gap to reliable performance.

The strongest argument against navigation-only evaluation comes from Protocol III ablation. For Gemini-2.5-Pro, full end-to-end performance is 21.0, but rises to 43.0 when extraction is removed, a 22.0-point drop attributable to extraction. For Claude-4.5, the corresponding figures are 16.0 and 34.0, a drop of 18.0 points. This shows that page arrival and answer correctness are materially different capabilities.

NavEval itself performs strongly. On WebRetriever, NavEval with Claude-4.5-Sonnet achieves average AR = 91.2, with reported values of 92.2 for SeeAct, 91.3 for Agent-E, 90.9 for Browser-Use, 91.4 for Gemini-2.5-Pro, and 90.1 for Claude-4.5. Competing evaluators are substantially lower: Autonomous Eval reports average AR in the 66.9–75.9 range depending on the judge, AgentTrek Eval in the 55.5–62.7 range, WebVoyager in the 65.2–77.1 range, and WebJudge in the 71.2–81.0 range. On Online-Mind2Web, NavEval reports 96.5 for SeeAct, 97.4 for Agent-E, 97.1 for Browser-Use, and 97.0 average AR, while the baselines remain below 88% Avg AR. The paper therefore presents NavEval not merely as a benchmark-specific tool, but as a more general evaluator.

6. Interpretation, misconceptions, and relation to adjacent work

WebRetriever is organized around two explicit corrections to common assumptions. The first is that navigation success is a sufficient proxy for real usefulness. The benchmark’s protocol split and Protocol III ablations show that this is false. The second is that screenshots alone are enough for evaluation. NavEval’s use of network requests, URL trajectories, and rule-based request filtering is a direct response to the failure of screenshot-only methods to capture exact query formulation, filtering operations, and backend semantics (Dong et al., 7 Jul 2026).

The benchmark also exposes recurring failure modes in current agents. The supplementary analysis highlights pop-up windows, collapsible or hidden content, alphabetical index navigation, and hallucinated interactions, including clicking non-interactive regions or performing unnecessary actions when the needed information is already visible. These are not marginal artifacts; they are characteristic failure modes of live web interaction.

WebRetriever’s live-web design creates maintenance obligations. The authors explicitly state that the benchmark will be continuously maintained, and that tasks made obsolete by website changes will be replaced by new tasks of matching difficulty. At the same time, two limitations remain salient. First, Protocol III contains only 100 tasks, which makes the hardest and most deployment-relevant setting relatively small. Second, NavEval, despite its strong AR numbers, remains an LLM-as-Judge pipeline whose robustness under future site changes and edge-case domains remains an open question.

In the broader literature, WebRetriever occupies a distinct position. It differs from LineRetriever, which is a planning-aware observation reduction method for AxTree-based web agents and operates on the current page observation rather than large-scale benchmark construction or live-website evaluation (Kerboua et al., 30 Jun 2025). It also differs from LiveWeb-IE, which evaluates online web information extraction on 15 permission-granted websites, 46 page groups, and 342 natural-language queries; that benchmark addresses the extraction layer on live rendered pages, whereas WebRetriever targets large-scale agent evaluation across 800 websites with explicit separation between navigation, knowledge-assisted interaction, and end-to-end extraction (Yang et al., 14 Mar 2026). A plausible implication is that WebRetriever functions as a benchmark for agentic web operation at scale, while LineRetriever and LiveWeb-IE illuminate narrower subproblems inside that broader stack.

Taken together, these design choices make WebRetriever a deployment-oriented benchmark rather than a conventional navigation suite. Its central significance is methodological: it makes visible the gap between “reaching a page” and “completing a web task correctly,” and it provides both a large live-web task set and an evaluation framework aimed at measuring that gap rigorously.