WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation

Abstract: As web agents increasingly demonstrate capabilities in automated task execution, the development of robust evaluation frameworks for assessing their navigation and task completion performance has emerged as a critical research priority. However, existing benchmarks exhibit fundamental limitations. First, they suffer from insufficient scale and limited domain diversity, constraining comprehensive evaluation of cross-domain generalization. Second, prevailing LLM-as-Judge evaluation methodologies inadequately capture fine-grained interaction semantics, particularly regarding precise query formulation and filtering operations. Third, current benchmarks predominantly emphasize navigation success metrics while neglecting critical requirements for real-world deployment scenarios. To address these limitations, we introduce WebRetriever, a large-scale benchmark encompassing 800 websites and 1,550 tasks across diverse domains, including consumer, professional, and enterprise sectors, with comprehensive coverage of user intent patterns. We propose NavEval (Navigation Evaluation), a novel LLM-as-Judge framework that leverages rich interaction context beyond visual screenshots, achieving state-of-the-art alignment with human judgment across multiple evaluation datasets. Furthermore, we establish three complementary evaluation protocols that collectively provide holistic assessment of web agent capabilities: navigation proficiency, knowledge-assisted interaction, and end-to-end task completion with information extraction. Extensive experimental analysis reveals substantial performance disparities across evaluation protocols, demonstrating that navigation success alone is an insufficient predictor of real-world application effectiveness. WebRetriever delivers fine-grained diagnostic insights into agent capabilities and establishes a rigorous foundation for advancing web agent research and development.

• The paper introduces WebRetriever, a scalable benchmark with 1,550 tasks across diverse domains to assess real-world web agent performance.
• The paper presents NavEval, a novel multi-modal evaluation method that integrates structured data and visuals to achieve over 90% agreement with human judgments.
• The paper reveals critical performance gaps in end-to-end task execution, underscoring the need for improved multi-step reasoning and knowledge integration.

WebRetriever: A Comprehensive Benchmark for Robust Web Agent Evaluation

Motivation and Context

The emergence of web agents powered by advanced LLMs and multimodal architectures has enabled increasing levels of automation on real-world websites, extending from simple path-planning tasks to complex business logic and knowledge integration across a diverse landscape of domains. However, effective evaluation of these agents remains highly constrained due to fundamental benchmark limitations: prior efforts fail to achieve adequate scale, suffer from domain and intent diversity shortfalls, and rely on brittle or imprecise metrics largely focused on navigation without ensuring true task completion or real-world readiness. Furthermore, the lack of scalable, high-fidelity automated evaluation severely impedes progress, as existing LLM-as-a-judge and rule-based protocols fail to capture critical fine-grained interaction semantics.

To address these challenges, "WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation" (2607.06118) introduces a new comprehensive benchmark (WebRetriever) and evaluation methodology (NavEval), together establishing rigorous protocols for practical and diagnostic assessment of web agents.

Figure 1: WebRetriever addresses the shortcomings of previous benchmarks by enhancing dataset scale/diversity, automating evaluation with higher accuracy, and proposing realistic deployment protocols.

Dataset Construction and Diversity

WebRetriever targets critical gaps in prior work by offering 1,550 unique tasks distributed over 800 high-quality, actively-maintained websites. Site curation is guided by internet traffic statistics, ensuring ecological validity and broad real-world deployment relevance. The dataset systematically covers eight industrial verticals, including Technology, Finance, Education, Entertainment, Life Services, Healthcare, Manufacturing, and Public Services. Sites range across geographies and languages (including non-English content), capturing the global and heterogeneous nature of the web.

Figure 2: Distribution of WebRetriever's website coverage by domain, confirming broad industrial and application scope.

Task design leverages input from diverse annotator pools (domain experts to end-users), building both general-intent and professional-intent scenarios. This yields tasks with highly varied interaction patterns, from standard consumer information retrieval to professional multi-step workflows. These are organized by interaction count into easy, medium, and hard categories, with explicit validation for uniqueness, stability, and logical alignments to real business practices.

Figure 3: Distribution of general user intents, illustrating broad behavioral and goal diversity in the dataset.

WebRetriever also incorporates dynamic maintenance: tasks rendered obsolete by website evolution are replaced iteratively, supporting reproducibility in both current evaluation and longitudinal studies.

NavEval: Fine-Grained Automated Evaluation

Most automated evaluation pipelines—either rule-based or screenshot-centric LLM judgment—fail to accurately capture the nuanced, longitudinal interaction logic necessary for robust agent assessment. NavEval, the novel LLM-as-a-judge approach introduced in this work, overcomes these deficiencies with a multi-modal input strategy: it combines structured interaction data (action traces, URL and network request logs) with terminal screenshot evidence, processed through rule-based filtering to prune noise and non-semantic signal.

Figure 4: NavEval pipeline integrates intermediate signals, semantic traces, and visual context for robust, high-agreement evaluation.

NavEval structures all interaction evidence into a condensed input for the LLM, which then executes fine-grained, requirement-level reasoning to determine binary and confidence-weighted outcome labels. By leveraging diverse context—particularly network requests filtered to retain only semantically relevant actions—NavEval achieves over 90% agreement with human annotation, outperforming competing automated approaches (WebJudge, AgentTrek Eval, WebVoyager, Autonomous Eval) across a range of agents and tasks.

Protocols for Holistic Assessment

WebRetriever defines three evaluation protocols, each explicitly capturing different characteristics of real-world agent deployment:

• Protocol I: Navigation Proficiency – Agents must reach the target page by interpreting natural language task descriptions.
• Protocol II: Knowledge-Assisted Navigation – Agents are additionally provided with operational documentation generated by a semi-automated pipeline, simulating access to domain- or workflow-specific support resources.

  Figure 5: Semi-automated operational documentation pipeline that produces high-quality procedural knowledge for Protocol II.

• Protocol III: End-to-End Task Completion – Beyond page arrival, agents must extract and submit target information, requiring cross-modal and cross-step reasoning.

These protocols collectively diagnose navigation, knowledge integration, and full-cycle task execution abilities—dimensions that existing navigation-centric metrics neglect. Task allocation across protocols is balanced: 1,000 each for Protocols I and II, and 100 (high-complexity) for Protocol III, with shared tasks supporting direct ablation studies.

Experimental Results and Analyses

Comprehensive experiments reveal strong and actionable deficiencies in current web agents’ real-world capabilities:

• Navigation Alone is Insufficient: Average human-judged success rates under Protocol I are only 21.1%, despite much higher navigation scores on previous benchmarks.
• External Knowledge Aids—but Only Moderately: Operational documentation in Protocol II yields a modest improvement, with the best agents (Gemini-2.5-Pro, Claude-4.5) reaching 45.2% and 36.3% success, respectively. However, a significant gap between optimal and practical performance persists.
• End-to-End Execution Remains Unsatisfactory: Protocol III exposes a dramatic drop to 11.8% average success, evidencing that nearly all state-of-the-art web agents fail to reliably complete realistic task workflows, even after successful navigation. Extraction, cross-modality integration, and multi-step reasoning are major failure points.

Ablation studies confirm the above trends: the availability of operational documentation yields approximately 8% success rate improvement but does not compensate for fundamental limitations in agent reasoning and extraction. Removing extraction constraints (i.e., evaluating navigation but not downstream actions) artificially doubles success rates, falsifying direct comparisons with benchmarks that stop at navigation metrics.

Figures 9–13 (case examples) illustrate qualitative shortfalls: agents frequently hallucinate actions, misunderstanding interface affordances, or fail at multi-step interaction sequences such as collapsible content expansion, index browsing, and chart analysis. Detailed error analyses indicate that limitations stem from brittle or incomplete internal representations, inability to robustly parse hidden interface states, and shallow reasoning over document or tabular data.

NavEval: Evaluation Efficacy

In cross-benchmark human agreement tests, NavEval achieves 91–97% agreement, setting a new standard for automated agent evaluation. This is largely attributed to fine-grained request analysis and improved contextual disambiguation, as opposed to screenshot-only or unsupervised LLM judgment approaches. NavEval is robust to LLM backbone variations, with Claude-4.5-Sonnet slightly outperforming GPT-4o and O4-mini, but all models yielding strong performance gains over prior art via NavEval’s signal integration design.

Implications and Future Directions

WebRetriever and NavEval provide a new standard for realistic, high-coverage, deployment-oriented web agent evaluation. Empirical findings indicate:

• Prevailing benchmark scores significantly overestimate deployed agent utility. Successful navigation does not guarantee actionable, robust task completion.
• Integrating external, operational knowledge (manuals, best practices, process documentation) is necessary but insufficient; major architectural improvements in LLM-driven perception, memory, and reasoning are required.
• The field must directly target cross-modality integration, dialog-based and retrieval-augmented reasoning, and robust long-horizon planning to close the gap from demonstration to real use.
• Automated evaluation can and should move beyond annotation bottlenecks, with NavEval-style multi-modal protocols rapidly accelerating research iteration and enabling reliable comparison.

Conclusion

WebRetriever establishes a rigorous foundation for the next generation of web agent research, combining scale, diversity, robustness, and diagnostic breadth unattainable in prior benchmarks. The accompanying NavEval framework introduces a reliable, high-fidelity automated evaluation pipeline suitable for both academic and industrial settings. Experiments reveal substantial gaps in agent performance on realistic tasks, emphasizing the field’s need to progress beyond navigation-centric metrics toward true end-to-end web task automation. Future research must explicitly target knowledge integration, robust extraction, and dynamic environment handling to ensure practical, deployable web agents.