Web-CogBench: Cognitive Evaluation Suite

• Web-CogBench is a cognitive benchmark that defines and measures web agents' abilities to memorize, understand, and explore using real-world website scenarios.
• It employs diverse modalities and robust metrics such as Accuracy, ROUGE-L, and LVM judging to diagnose and compare agent performance.
• The framework provides actionable insights into agent strengths and weaknesses, guiding future research on adaptive, knowledge-driven web intelligence.

Web-CogBench is a comprehensive evaluation suite specifically designed to assess and dissect the cognitive and knowledge-driven capabilities of web agents operating in visually rich and interactive online environments. Rooted in a cognitive science framework and assembled from real-world website scenarios, Web-CogBench enables the precise measurement of agent proficiency across factual, conceptual, and procedural domains, providing a diagnostic lens from fine-grained perception to complex, goal-oriented reasoning (Guo et al., 3 Aug 2025).

1. Motivation and Design Principles

Web-CogBench addresses limitations in prior web agent benchmarks, which typically emphasized either low-level perception (such as optical character recognition or UI element detection) or high-level, end-to-end task success (such as automated form filling), while offering limited insight into the internal cognitive stratification of agent reasoning. Inspired by Bloom’s Taxonomy, it models web-agent competency along three cognitive dimensions:

• Memorizing: the agent's capacity to recall factual web knowledge.
• Understanding: the extraction and synthesis of conceptual relationships.
• Exploring: procedural reasoning, including planning, execution, and goal fulfillment.

Web-CogBench isolates these axes, allowing for targeted measurement, demarcation of skill boundaries, and comparison across agents with different architectures or training curricula. The framework further supports correlation analysis between low-level perceptual skills and emergent, high-level intelligence (Guo et al., 3 Aug 2025).

2. Task Taxonomy and Cognitive Mapping

Each benchmark task in Web-CogBench is derived from the Web-CogDataset—a structured corpus curated from 14 real-world websites—and mapped to fundamental knowledge types and cognitive processes as formalized in the Web-CogKnowledge Framework.

• Factual Knowledge → Memorizing
  • Element Attribute Recognition: Predict the role and name of a UI element from a screenshot.
  • Next Page Prediction: Given a highlighted UI element, select or describe the resulting navigated page.
  • Source Element Prediction: Identify which of several marked elements leads to a given target page.
• Conceptual Knowledge → Understanding
  • Element Understanding: Provide an open-ended description of an element’s visual attributes, layout position, and likely function.
  • WebPage Understanding: Summarize a page’s organizational structure, key components, and overall layout.
• Procedural Knowledge → Exploring
  • User’s Intention Prediction: Infer the high-level user instruction underlying a series of UI screenshots.
  • Popup Close: Identify valid strategies to dismiss a modal or popup interruption.
  • Single-Step Exploration: Select which element best fulfills a simple user-issued command.

This mapping ensures both coverage (by spanning the spectrum of web interaction reasoning) and diagnostic granularity (by assigning performance to discrete cognitive mechanisms).

3. Benchmark Formats and Modalities

Web-CogBench tasks employ multiple modalities and formats to reflect real interaction diversity:

• Inputs: Full-page and trajectory-based screenshots (with or without bounding box highlighting), snippets from the Accessibility Tree for structural and semantic context, and temporally sequenced transitions for action tasks.
• Outputs:
  • Classification/multiple choice: As in Next Page Prediction.
  • Open-ended text generation: For roles and descriptions in Element Attribute Recognition or Understanding tasks.
  • Element ID selection: E.g., determining the causal element in navigation or action selection.
• Implied Difficulty: The level of abstraction, perceptual load, and reasoning horizon is encoded in each task type. Memorization typically involves single-step perception; conceptual understanding requires integrating multiple facts; procedural exploration demands planning, error-tolerance, and stateful execution (e.g., correctly handling popups mid-trajectory).

4. Evaluation Metrics and Formalism

Web-CogBench employs a domain-appropriate set of metrics tailored to the range of output formats:

• Accuracy: Used for discrete-decisions and classification tasks, such as Next Page and Source Element Prediction, Popup Close, and Single-Step Exploration.
• ROUGE-L: Applied to short, open-ended generative tasks (e.g., Element Attribute Recognition) where token-level exact match is too strict.
• Large Vision-LLM (LVM) Judge: For long-form open responses, such as Element or WebPage Understanding, an LVM is prompted to provide a 5-point scale score.
• Markov Decision Process (MDP) Formalism: The agent’s interaction is modeled as a POMDP: $P = (S, A, O, K, T, R)$, with policy

$$(h_t, a_t) = \pi_\theta(\cdot | K, I, Q, o_1, h_1, a_1, ..., o_t)$$

where $R$ is a binary reward function.

No custom scoring functions beyond the above and standard Accuracy/ROUGE-L are introduced; the focus is on comparability and statistical rigor (Guo et al., 3 Aug 2025).

5. Experimental Results and Benchmarks

Empirical evaluation using Web-CogBench (Table 4 in (Guo et al., 3 Aug 2025)) demonstrates the following agent proficiency distribution (all scores in %):

Model	Memorizing	Understanding	Exploring	Overall
Claude Sonnet 4	80.6	58.5	87.4	76.8
Gemini 2.5 Pro	85.6	68.0	93.7	80.2
Qwen2.5-VL-7B	67.6	61.0	77.9	69.8
UI-TARS-7B-SFT	76.3	54.0	63.5	46.4
Web-CogReasoner (proposed)	91.9	74.1	95.3	84.4

Web-CogReasoner, a knowledge-driven Chain-of-Thought agent developed using the Web-CogKnowledge Framework, leads across all axes—especially in Memorizing (91.9%) and Exploring (95.3%)—demonstrating the training protocol's efficacy in instilling factual and procedural reasoning. Notably, even for the more semantically complex Understanding domain, it exceeds the next-best model (Gemini 2.5 Pro) by approximately 6 percentage points (Guo et al., 3 Aug 2025).

6. Insights, Limitations, and Future Directions

Key insights from benchmarking on Web-CogBench include:

• Structured two-stage training (progressing from Factual through Conceptual to Procedural knowledge) consistently improves agent capabilities across all evaluated cognitive strata.
• Performance on Web-CogBench correlates well with real-world web task scores (e.g., WebVoyager), supporting its validity as a diagnostic proxy.

Identified weaknesses:

• Open-ended descriptions in Understanding tasks routinely fall below 80%, indicating persistent limits in semantic and visual-language integration.
• Robustness to UI variability (e.g., handling unexpected popups or mid-flow interruptions) remains incomplete.

Directions for future research:

• Incorporating metacognitive feedback loops for hallucination detection and self-correction during open-domain QA.
• Expanding the evaluation suite to include adaptive difficulty (multi-page workflows, asynchronous dynamics).
• Developing granular LVM or human-in-the-loop judging for nuanced, open-ended responses, mitigating over-reliance on proxy metrics such as ROUGE-L and static LVM summarization (Guo et al., 3 Aug 2025).

7. Significance and Relation to Broader Research

Web-CogBench provides the first fine-grained, Bloom’s-inspired, cognitive-stage-oriented benchmark for web agents, enabling researchers to analyze and enhance models along a progression from perceptual memorization to high-level procedural action. Its task and metric structure serves as an investigative complement to other web agent and code-generation benchmarks (see also WebCoderBench (Liu et al., 5 Jan 2026) for end-to-end web-app generation evaluation with interpretable, LLM-as-judge metrics).

A plausible implication is that the adoption of such staged cognitive benchmarks will accelerate systematic advances in web agency, emphasizing knowledge structuring and reasoning rather than mere perceptual or success-rate metrics, and will shape future research directions in multimodal, interactive web AI.