Web-CogDataset: Cognitive Web Agent Dataset

• Web-CogDataset is a hierarchically structured, multimodal resource that organizes factual, conceptual, and procedural web knowledge for training cognitive web agents.
• It employs a graded taxonomy from perceptual to procedural tasks, capturing UI details and dynamic interactions from real-world web interfaces.
• The dataset underpins the Web-CogReasoner framework, enabling targeted evaluations through Chain-of-Thought reasoning and progressive task difficulty.

Web-CogDataset is a hierarchically structured, multimodal resource developed specifically for instilling and evaluating foundational and advanced knowledge required by web agents to exhibit cognitive reasoning capabilities. Designed as the core of the Web-CogReasoner framework, it captures factual, conceptual, and procedural knowledge, systematically curated from real-world web interfaces and interactions. The dataset operationalizes cognitive models for digital environments, enabling both knowledge acquisition—memorizing and understanding—and the practical execution of agentic actions via knowledge-driven Chain-of-Thought reasoning (Guo et al., 3 Aug 2025).

1. Taxonomy and Hierarchical Structure

Web-CogDataset is formulated according to the Web-CogKnowledge Framework, which decomposes agent capabilities into three explicit knowledge types:

• Factual Web Knowledge: This layer covers the discrete, perceptual facts about web elements and immediate environments. Sample tasks include Element Attribute Recognition, Sub-elements Prediction, Page Change Prediction, Next Page Prediction, and Source Element Prediction. These evaluate an agent’s capacity to identify properties such as semantic role, accessible name, or predict the visual effect of specific UI interactions.
• Conceptual Web Knowledge: At this level, the agent is taught to synthesize multiple factual elements into higher-level abstractions, such as understanding the composition and functional semantics of composite UI entities. Tasks include Element Understanding, Webpage Understanding, and Caption QA, targeting the semantic integration necessary for interpreting varied web interfaces.
• Procedural Web Knowledge: This top layer deals with action-oriented, sequential reasoning required for real-world operation. Example tasks are User’s Intention Prediction, Popup Close, Single-Step Web Task, and Noisy Multi-Steps Web Task. Here the agent is assessed on its ability to execute multi-step plans under real-world ambiguities such as UI interruptions or noisy state transitions.

A representative tabular overview of knowledge types and exemplar tasks:

Knowledge Type	Example Tasks	Purpose
Factual	Attribute/Next Page Prediction	Perceptual identification
Conceptual	Element/Webpage Understanding	Semantic integration
Procedural	Intention Prediction, Web Tasks	Sequenced action planning

2. Data Curation and Annotation Methodology

The dataset is curated from 14 well-known real-world websites, with a systematic pipeline tailored to mimic agents’ real interaction contexts:

• Metadata Extraction: For each web page, multimodal metadata is gathered, prominently from the accessible tree (AX Tree). Structured details such as the CSS selector, element role, accessible name, outerHTML, bounding box, and screenshot are stored, capturing both static and dynamic aspects across browsing events.
• Interaction Layering: Dynamic data is captured using tools such as Playwright to iteratively traverse UI states (e.g. Layer 1: baseline, Layer 2–3: hover/click states). This ensures that the dataset encodes the evolution of UI layouts corresponding to real user actions.
• Granular Task Definition: The final resource includes 12 granular tasks, each tightly aligned with one of the knowledge layers. For example, in “Element Attribute Recognition,” agents must infer the role and accessible name from a highlighted screenshot, while “Popup Close” isolates procedural capacity to handle UI disruptions.

3. Integration with Cognitive Agent Training

Web-CogDataset is engineered as the “textbook” for agent-centric cognitive training within the Web-CogReasoner architecture:

• Stage-wise Knowledge Induction: The agent first undertakes supervised imitation learning for factual and conceptual tasks, instilling the “what” of web environments as a static knowledge repository.
• Procedural Competence: In subsequent stages, procedural tasks are incorporated to induce “how” knowledge—dynamic planning and manipulation in environments with real contingencies.
• Chain-of-Thought Reasoning Pipeline: The integration of static and dynamic knowledge is formalized as:

$$\text{Task Prompt} \rightarrow \text{Knowledge-driven CoT Reasoning} \rightarrow \text{Planning} \rightarrow \text{Action}$$

where explicit segments of factual, conceptual, and procedural reasoning are instantiated, with the CoT mechanism realized in the agent’s interactive planning.

• Model Architecture and Optimization: Experiments utilize a Qwen2.5-VL-7B backbone, with staged curriculum learning—first on recognition/understanding, then rollout to complex procedural tasks. Ablation studies demonstrate incremental benefits for both memorizing and cognitive exploration capabilities.

4. Web-CogBench: Evaluation Protocols and Metrics

Complementing the training dataset, Web-CogBench provides a comprehensive evaluation suite to assess agent performance along the same knowledge axes:

• Evaluation Dimensions: Memorizing (factual recall), Understanding (semantic abstraction), and Exploring (procedural execution) are directly measured through task-specific benchmarks.
• Metrics: For example, Element Attribute Recognition leverages ROUGE-L for generative recall; other tasks, such as Next Page Prediction, use accuracy-based scoring.
• Granularity: The evaluation protocol measures not just aggregate scores but also alignment to the hierarchical task structure, quantifying how well an agent’s static knowledge translates into actionable, goal-directed behavior.

5. Significance for Web-Centric Cognitive Research

Web-CogDataset enables systematic inquiry into several core questions:

• How does knowledge granularity—ranging from atomic facts to high-level procedures—impact web agent generalization?
• What is the interplay between static information (semantic roles, page structure) and dynamic behavior (user interaction, multi-step navigation)?
• Can structured knowledge curation enable stronger generalization to unseen websites or novel tasks, as shown by superior performance on transfer and ablation scenarios?

A plausible implication is that systematically organized knowledge resources facilitate agentic reasoning and transfer, as documented through experimental superiority over other models in the generalization to unseen cognitive tasks.

6. Limitations and Comparative Context

Web-CogDataset is distinguished from other web corpora such as WebVision (Li et al., 2017), ClueWeb22 (Overwijk et al., 2022), and IW-Bench (Guo et al., 14 Sep 2024) by its explicit cognitive and procedural orientation. While general web corpora are often designed for retrieval, classification, or pretraining, Web-CogDataset’s task-driven construction directly targets the “knowledge-to-action” gap for web agents. In contrast, datasets such as ClueWeb22 focus on scale and multi-modal signals supporting pretraining, and IW-Bench provides fine-grained layout and element accuracy metrics for image-to-web synthesis tasks. Web-CogDataset’s granularity in representing agent-relevant knowledge, combined with a cognitively motivated taxonomy, supports principled evaluation and systematic enhancement of cognitive reasoning in digital environments.

7. Illustrative Figures and LaTeX Representations

Key visualizations, such as Figure \ref{fig:webcogdataset}, delineate the knowledge/task matrix; Figure \ref{fig:overview} provides the overall pipeline of the Web-CogKnowledge Framework. Math formalism undergirds the transformation pipeline:

$$\text{Task Prompt} \rightarrow \text{Knowledge-driven CoT Reasoning} \rightarrow \text{Planning} \rightarrow \text{Action}$$

This mathematical structure encapsulates the operational translation of static web knowledge into dynamic web agency.

---

In summary, Web-CogDataset is a purpose-built, hierarchically organized, and task-centric dataset tailored for instilling and evaluating factual, conceptual, and procedural web knowledge. Its systematic curation and integration with comprehensive cognitive benchmarks enable precise measurement and development of cognitively proficient web agents, bridging the gap between static knowledge and interactive digital reasoning (Guo et al., 3 Aug 2025).