---
title: WebRetriever Benchmark
url: https://www.emergentmind.com/topics/webretriever
type: topic
---

# WebRetriever Benchmark

WebRetriever is a large-scale benchmark for realistic online evaluation of web agents on live websites. It was introduced to measure capabilities that matter in deployment rather than only benchmark-style navigation success, and combines three elements that are usually separated: broad website coverage, a judge that uses richer interaction evidence than screenshots alone, and evaluation protocols that distinguish navigation, knowledge-assisted interaction, and end-to-end task completion with information extraction. The benchmark contains **800 websites** and **1,550 tasks**, and its core claim is that navigation success alone is an insufficient predictor of real-world application effectiveness [2607.06118].

## 1. Definition and conceptual scope

WebRetriever is designed for the evaluation of web agents that must operate on **live websites** under real interactive conditions. The benchmark is motivated by four shortcomings in prior work: limited scale and domain diversity, screenshot-only or shallow LLM-as-Judge evaluation, overreliance on navigation success metrics, and weak coverage of deployment requirements such as knowledge-assisted interaction and information extraction [2607.06118].

The benchmark therefore treats web-agent evaluation as a three-part problem. First, an agent must navigate unfamiliar sites. Second, it may need to use external operational knowledge or documentation. Third, it may need to complete an end-to-end task that requires extracting the correct information rather than merely reaching a plausible page. This framing is central to WebRetriever’s design: the benchmark is not only about clicking through interfaces, but about whether the agent can finish the user’s job.

A recurrent misconception addressed by the benchmark is that final-page arrival or visual plausibility is enough. WebRetriever explicitly rejects that assumption. The paper argues that subtle errors in search queries, filters, sort orders, or object selection can invalidate a result even when the final page looks correct. This is why the benchmark emphasizes fine-grained interaction semantics and not only screenshot similarity.

## 2. Benchmark construction and governance

Website selection combines breadth with domain realism. The benchmark uses **SimilarWeb traffic data** as a baseline, selecting the **top 30 sites by traffic within each sector**, and then adds specialized vertical sites from authoritative internet research institutions such as **199it**, especially for **business intelligence** coverage [2607.06118]. The paper states that WebRetriever spans **eight core sectors** and lists **Technology / Internet**, **Business / Finance**, **Education / Research**, **Culture**, **Entertainment / Travel**, **Life Services**, **Healthcare**, **Industrial Manufacturing**, and **Public Services**. The abstract further characterizes the scope as covering **consumer**, **professional**, and **enterprise** sectors. Supplementary material adds multilingual and geographic breadth, including websites in **English**, **Chinese**, **Spanish**, and **French**, and websites from the US, China, Europe, Asia, Africa, and Latin America.

Task creation was carried out by a **14-member annotation team** including domain experts, mid-level managers, senior data analysts, and university students. Tasks were refined under three principles: **uniqueness**, **stability**, and **logical authenticity**. Difficulty is categorized by the number of human reference action steps, denoted ${Step}_{n}$: **easy** if \(n < 6\), **medium** if \(6 \leq n \leq 15\), and **hard** if \(n > 15\).

The benchmark uses a five-stage governance pipeline: **task design**, **annotation**, **cross-validation**, **quality review**, and **final verification**. It also enforces **strict annotator–reviewer separation**, ran **12 calibration sessions**, and applies a **sequential multi-stage quality-control process** rather than standard parallel inter-annotator agreement labeling. Every task underwent **three independent review stages**, and revision was mandatory whenever disagreements arose. Reported correction rates were **30%**, **26%**, and **27%** across the three review stages, and **54% of tasks** received at least one correction. This unusually heavy review load is part of the benchmark’s attempt to make live-web evaluation auditable despite website volatility.

Operationally, each task starts from a **predefined website entry URL**, and **search engine access is disabled**. That restriction is important: it prevents agents from bypassing within-site interaction by delegating the problem to generic web search.

## 3. NavEval and fine-grained automatic judgment

WebRetriever’s automatic evaluator is **NavEval** (“Navigation Evaluation”), an LLM-as-Judge framework that uses richer interaction context than screenshots alone. Formally, given task description \(T\), website URL \(U\), a sequence of web requests \(R = (r_1, r_2, \ldots, r_n)\), a sequence of executed actions \(A = (a_1, a_2, \ldots, a_n)\), and final screenshot \(I\), NavEval outputs a binary success judgment
\[
P = \text{NavEval}(T, U, R, A, I),
\qquad
P \in \{\text{True}, \text{False}\}.
\]
It also applies a rule-based request filtering step,
\[
\mathcal{R}_{l}^{\prime} = \mathcal{F}_{rule}(R, U),
\]
where the filtering matches requests against the task URL at the subdomain level, removes task-irrelevant or random fields, normalizes structured payloads, and removes invalid entries [2607.06118].

The evaluator’s methodological distinction is that **screenshots are used for state verification**, while **network requests and URL trajectories are used for operation verification**. This enables explicit checking of whether the agent actually executed the right search query, applied the right filters, selected the right item, or submitted the right operation. The supplementary prompt instructs the judge to decompose a task into atomic requirements and verify each requirement using network requests, URL trajectory, and final screenshot jointly.

The benchmark defines **Success Rate (SR)** and **Human Agreement Rate (AR)**. AR compares automated predictions \(\hat{y}_t\) to human labels \(y_t\) over a task set, and is defined as
\[
\text{AR} = \frac{1}{|\mathcal{T}|} \sum_{t \in \mathcal{T}}
\begin{cases}
1, & \text{if } \hat{y}_t = y_t,\\
0, & \text{otherwise}.
\end{cases}
\]
The use of AR is central because WebRetriever is not only proposing a benchmark; it is also proposing a new evaluator that is meant to align with human judgment more closely than prior screenshot-dominated approaches.

## 4. The three evaluation protocols

A major contribution of WebRetriever is the separation of evaluation into three complementary protocols rather than a single navigation score.

| Protocol | Capability assessed | Tasks |
|---|---|---:|
| Protocol I | Navigation proficiency | 1,000 |
| Protocol II | Knowledge-assisted interaction | 1,000 |
| Protocol III | End-to-end task completion with information extraction | 100 |

There are **550 overlapping tasks between Protocols I and II**, differing only in whether operational documentation is provided. Protocol I measures basic navigation proficiency from the entry URL using only the task description. Protocol II evaluates whether the agent can use **operational documentation/manuals** to navigate more effectively on unfamiliar sites. Protocol III is the most deployment-oriented setting: the agent must not only reach the relevant page, but also retrieve and extract the correct answer.

The benchmark’s documentation for Protocol II is itself constructed through a multi-step pipeline: agents generate trajectories, NavEval filters successful ones, annotators refine the trajectories, and an LLM generates concise operational documentation from the refined trajectories. This protocol is meant to simulate realistic settings in which agents can consult internal knowledge bases, SOPs, or workflow manuals.

Protocol III is deliberately designed around “Deep Research–style” tasks and is governed by three principles: **Authoritativeness**, **Interaction Necessity**, and **Determinism**. The supplementary material describes representative scenario families such as **Document Extraction**, **Form Interaction**, **Multi-source Comparison**, **Complete Data Retrieval**, and **Multi-dimensional Chart**. In the main results table, Protocol III is reported with human evaluation.

Illustrative tasks show the protocol differences clearly. Protocol I examples include searching for a Kaggle dataset with specified usability constraints or performing advanced search in **CiNii Books** with publication-year and full-text filters. Protocol II examples include using **va.gov** to find guide-dog information for blind veterans or searching **komoot.com** with multiple hiking filters. Protocol III examples include identifying French universities in the top **0.2%** on **CWUR**, finding the city in Europe with the lowest office rent and expected increase on **CBRE**, or extracting prerequisite logic from the **MIT** course catalog.

## 5. Empirical findings

WebRetriever’s main empirical result is that current web agents perform poorly once the evaluation is broadened from navigation to deployment-oriented task completion. Human-assessed average success rates are **21.1%** on Protocol I, **29.2%** on Protocol II, and **11.8%** on Protocol III [2607.06118].

At the agent level, **Gemini-2.5-Pro (Computer-Use)** achieves the highest reported human success rates: **37.1** on Protocol I, **45.2** on Protocol II, and **21.0** on Protocol III. **Claude-4.5 (Computer-Use)** reports **28.1**, **36.3**, and **16.0** respectively. **Browser-Use** reports **24.0**, **31.6**, and **11.0**. **UI-TARS-1.5** reaches **16.5**, **24.8**, and **8.0**. **Agent-E** reaches **11.6**, **20.4**, and **9.0**, while **SeeAct** reaches **9.2**, **17.1**, and **6.0**. These numbers are low even before considering production reliability thresholds.

The difference between Protocol II and Protocol I quantifies the value of operational knowledge. For example, **Gemini-2.5-Pro** improves from **40.9** to **49.2** when documentation is added in the comparison reported by Table 5, a gain of **+8.3**, while **Claude-4.5** improves from **31.3** to **39.7**, a gain of **+8.4**. Conversely, removing documentation from Protocol II reduces **Gemini-2.5-Pro** from **50.1** to **41.4** and **Claude-4.5** from **40.1** to **31.9**. The benchmark therefore supports a precise conclusion: documentation helps substantially, but does not close the gap to reliable performance.

The strongest argument against navigation-only evaluation comes from Protocol III ablation. For **Gemini-2.5-Pro**, full end-to-end performance is **21.0**, but rises to **43.0** when extraction is removed, a **22.0-point** drop attributable to extraction. For **Claude-4.5**, the corresponding figures are **16.0** and **34.0**, a drop of **18.0 points**. This shows that page arrival and answer correctness are materially different capabilities.

NavEval itself performs strongly. On WebRetriever, NavEval with **Claude-4.5-Sonnet** achieves average **AR = 91.2**, with reported values of **92.2** for SeeAct, **91.3** for Agent-E, **90.9** for Browser-Use, **91.4** for Gemini-2.5-Pro, and **90.1** for Claude-4.5. Competing evaluators are substantially lower: **Autonomous Eval** reports average AR in the **66.9–75.9** range depending on the judge, **AgentTrek Eval** in the **55.5–62.7** range, **WebVoyager** in the **65.2–77.1** range, and **WebJudge** in the **71.2–81.0** range. On **Online-Mind2Web**, NavEval reports **96.5** for SeeAct, **97.4** for Agent-E, **97.1** for Browser-Use, and **97.0** average AR, while the baselines remain below **88% Avg AR**. The paper therefore presents NavEval not merely as a benchmark-specific tool, but as a more general evaluator.

## 6. Interpretation, misconceptions, and relation to adjacent work

WebRetriever is organized around two explicit corrections to common assumptions. The first is that **navigation success is a sufficient proxy for real usefulness**. The benchmark’s protocol split and Protocol III ablations show that this is false. The second is that **screenshots alone are enough for evaluation**. NavEval’s use of network requests, URL trajectories, and rule-based request filtering is a direct response to the failure of screenshot-only methods to capture exact query formulation, filtering operations, and backend semantics [2607.06118].

The benchmark also exposes recurring failure modes in current agents. The supplementary analysis highlights **pop-up windows**, **collapsible or hidden content**, **alphabetical index navigation**, and **hallucinated interactions**, including clicking non-interactive regions or performing unnecessary actions when the needed information is already visible. These are not marginal artifacts; they are characteristic failure modes of live web interaction.

WebRetriever’s live-web design creates maintenance obligations. The authors explicitly state that the benchmark will be **continuously maintained**, and that tasks made obsolete by website changes will be replaced by new tasks of **matching difficulty**. At the same time, two limitations remain salient. First, **Protocol III contains only 100 tasks**, which makes the hardest and most deployment-relevant setting relatively small. Second, NavEval, despite its strong AR numbers, remains an **LLM-as-Judge** pipeline whose robustness under future site changes and edge-case domains remains an open question.

In the broader literature, WebRetriever occupies a distinct position. It differs from **LineRetriever**, which is a planning-aware observation reduction method for AxTree-based web agents and operates on the current page observation rather than large-scale benchmark construction or live-website evaluation [2507.00210]. It also differs from **LiveWeb-IE**, which evaluates online web information extraction on **15 permission-granted websites**, **46 page groups**, and **342 natural-language queries**; that benchmark addresses the extraction layer on live rendered pages, whereas WebRetriever targets large-scale agent evaluation across **800 websites** with explicit separation between navigation, knowledge-assisted interaction, and end-to-end extraction [2603.13773]. A plausible implication is that WebRetriever functions as a benchmark for agentic web operation at scale, while LineRetriever and LiveWeb-IE illuminate narrower subproblems inside that broader stack.

Taken together, these design choices make WebRetriever a deployment-oriented benchmark rather than a conventional navigation suite. Its central significance is methodological: it makes visible the gap between “reaching a page” and “completing a web task correctly,” and it provides both a large live-web task set and an evaluation framework aimed at measuring that gap rigorously.

Source: https://www.emergentmind.com/topics/webretriever