Ego2Web: A Web Agent Benchmark Grounded in Egocentric Videos

Abstract: Multimodal AI agents are increasingly automating complex real-world workflows that involve online web execution. However, current web-agent benchmarks suffer from a critical limitation: they focus entirely on web-based interaction and perception, lacking grounding in the user's real-world physical surroundings. This limitation prevents evaluation in crucial scenarios, such as when an agent must use egocentric visual perception (e.g., via AR glasses) to recognize an object in the user's surroundings and then complete a related task online. To address this gap, we introduce Ego2Web, the first benchmark designed to bridge egocentric video perception and web agent execution. Ego2Web pairs real-world first-person video recordings with web tasks that require visual understanding, web task planning, and interaction in an online environment for successful completion. We utilize an automatic data-generation pipeline combined with human verification and refinement to curate well-constructed, high-quality video-task pairs across diverse web task types, including e-commerce, media retrieval, knowledge lookup, etc. To facilitate accurate and scalable evaluation for our benchmark, we also develop a novel LLM-as-a-Judge automatic evaluation method, Ego2WebJudge, which achieves approximately 84% agreement with human judgment, substantially higher than existing evaluation methods. Experiments with diverse SoTA agents on our Ego2Web show that their performance is weak, with substantial headroom across all task categories. We also conduct a comprehensive ablation study on task design, highlighting the necessity of accurate video understanding in the proposed task and the limitations of current agents. We hope Ego2Web can be a critical new resource for developing truly capable AI assistants that can seamlessly see, understand, and act across the physical and digital worlds.

• The paper introduces Ego2Web, a benchmark that integrates egocentric video input with digital web agents to enable multimodal task execution.
• It employs a hybrid model–human curation process and a novel evaluation protocol (Ego2WebJudge) to ensure high-fidelity performance measurement.
• Experimental results reveal significant gaps in multimodal reasoning, highlighting challenges in temporal grounding and cross-modal retrieval.

Ego2Web: Bridging Egocentric Visual Perception and Web-Agent Task Execution

Motivation and Benchmark Design

Ego2Web targets a critical gap at the intersection of multimodal web agents and real-world human-centric perception. Previous web-agent evaluation settings confine agents to purely digital domains—mainly acting on screenshots or textual instructions—without incorporating the user’s real-time physical context. Ego2Web is introduced to bridge this gap by grounding web-agent tasks in egocentric videos that capture real-world environments. Agents are tasked to interpret first-person video input, perform spatio-temporal grounding of relevant visual cues, and execute correct actions within web environments (e.g., e-commerce, media retrieval, navigation).

Figure 1: Ego2Web benchmark tasks combine egocentric visual grounding and diverse web task domains; agents must perceive visual cues and operate on the web accordingly.

This benchmark is structured around 500 verified video-instruction pairs spanning diverse domains. Each task requires multimodal grounding: extracting semantic and visual information (e.g., brand, category, spatial position) from egocentric video, and then planning web actions that achieve the instructed goal. The pipeline ensures tasks are not trivially solvable by textual abstraction alone, necessitating visual evidence from the video for task success.

Dataset Curation Pipeline

The Ego2Web dataset is constructed using a hybrid model–human collaboration process. Initially, egocentric videos sourced from large-scale datasets (e.g., Ego4D) are parsed by frozen MLLMs (e.g., Qwen3-VL) to generate structured, timestamped captions—dense descriptions of objects, actions, and scene context. Next, an LLM is prompted to create web task instructions anchored to the video’s visual context and constrained to a curated set of high-frequency domains (Amazon, YouTube, Wikipedia, Google Maps, etc.).

Figure 2: Semi-automatic pipeline leverages MLLM-generated captions for task synthesis and integrates human verification for quality assurance.

Human annotators perform rigorous refinement and validation, ensuring instructions are visually grounded, feasible within the website context, and formulated with sufficient clarity. The result is a diverse, realistic dataset covering the full spectrum of real-world web agent use cases, challenging models’ abilities to understand both physical and digital environments.

Task Distribution

The Ego2Web benchmark exhibits substantial domain diversity, with majority coverage across e-commerce, media retrieval, knowledge lookup, and map/local services, plus smaller tail domains for long-range generalization.

Figure 3: Task type distribution reveals dataset diversity across e-commerce, media, knowledge lookups, local/maps and tail environments.

This domain spread necessitates strong cross-modal grounding and robust web action planning, as models must generalize from relatively structured platforms to highly variable, complex web interfaces.

Evaluation Protocol: Ego2WebJudge

Evaluation of agent success is inherently non-trivial due to live, unconstrained web environments and the necessity of strict visual grounding. Ego2Web introduces Ego2WebJudge, a multimodal LLM-based automatic evaluator extending previous LLM-as-a-Judge frameworks. The evaluator receives the task instruction, action history, selected screenshots, and annotated egocentric video evidence; it performs key-point extraction from the instruction, selects relevant screenshots (to remain within context limits), and integrates all modalities to determine binary task success.

Figure 4: Overview of agent execution and evaluation, integrating egocentric video input, web actions, screenshots, and a final response, assessed via Ego2WebJudge.

Ego2WebJudge achieves high fidelity to human judgment (>84% agreement), outperforming previous automatic methods. Strict evaluation principles are applied: task completion is marked successful only with direct visual evidence confirming objective and constraints, reducing false positives and partial matches.

Experimental Results

Evaluation across six state-of-the-art web agents, including SeeAct, Browser-Use variants (with Gemini and GPT-4.1), Claude Computer-Use, and GPT-5.4, demonstrates pronounced headroom in agent performance. The best agent, Browser-Use Gemini-3-Flash, achieves 58.6% success rate (human evaluation), indicating substantial task difficulty and the necessity of robust multimodal grounding.

Key Findings

• Agent performance is tightly linked to input modality: Agents with direct access to raw egocentric video outperform those relying on caption-based abstractions or sparse keyframes, confirming the necessity of true visual perception and temporally dense modeling for reliable task completion.
• Domain-specific variation: Knowledge lookup yields highest success rates, driven by structured web content and straightforward objectives, while e-commerce and local/maps tasks remain challenging due to interface complexity and compositional reasoning requirements.
• Ablation reveals clear hierarchy: No visual input renders agents virtually ineffective (4.4% SR); detailed captions improve performance (23.6% SR), but only raw video input enables robust multimodal reasoning (48.2% SR).
• Failure analysis: Object misidentification, temporal reasoning errors, and cross-modal retrieval misalignment are prevalent, especially in tasks requiring accurate tracking and compositional mapping (see Figure 5).

  Figure 5: Visualization of agent failure due to temporal misgrounding and incorrect cross-modal retrieval.

Practical and Theoretical Implications

Ego2Web sets a new standard for multimodal agent evaluation, bridging real-world perception and actionable web tasks. The benchmark reveals non-trivial gaps in current agent designs, particularly in temporal reasoning, spatial grounding, and perception–action alignment. The strict evaluation protocol enables principled measurement of an agent's ability to leverage real-world context for web task execution, offering a scalable alternative to expensive human evaluation.

Practically, Ego2Web provides a testbed for the development of agents with AR glasses, wearables, and robots—where seamless integration of physical perception and digital interaction is mandatory. Theoretically, the benchmark exposes limitations in multimodal representation learning, model alignment, and evaluation methodology, motivating research into integrated architectures and improved grounding protocols for embodied assistant agents.

Figure 6: Examples illustrating contextual grounding: agents must recognize visually anchored cues and execute domain-specific web actions.

Future Directions

Future developments could include scaling dataset size and diversity, extending to active perception scenarios (e.g., interactive AR), and implementing more granular reward signals for trajectory-level evaluation. Improved multimodal agent architectures capable of dense video processing, robust temporal modeling, and compositional cross-modal retrieval are necessary to approach human-level performance. Ego2Web's evaluation methodology may further catalyze research into evaluation metrics incorporating intermediate reasoning steps and fine-grained visual verification.

Conclusion

Ego2Web establishes a rigorous benchmark for web agents requiring egocentric visual grounding, integrating automatic, scalable evaluation with high agreement to human judgment. The dataset, evaluation framework, and analysis collectively highlight the persistent gaps in multimodal agent capabilities, especially in perception–action integration. The benchmark will be instrumental in advancing multimodal agent systems able to operate seamlessly across both physical and digital environments, serving as a foundation for real-world, assistive AI deployment.

Figure 7: Example egocentric video with timestamped structured captions, enabling multimodal grounding for web agents.