Ego2Web: Egocentric Vision & Web Task Integration

• Ego2Web is a benchmark and system framework that pairs egocentric visual perception with automated web task execution.
• It employs a semi-automatic pipeline combining first-person videos, spatio-temporal grounding, and human-verified task synthesis across diverse domains.
• Ego2WebJudge uses a multimodal LLM protocol to evaluate agent performance, highlighting advances in context-aware, cross-modality AI.

Ego2Web is a benchmark and system framework designed to integrate egocentric visual perception—captured through first-person video streams—with sophisticated web-based task execution. It enables comprehensive assessment and development of AI agents that must link the recognition of real-world, visually grounded information (e.g., from wearable cameras or AR glasses) to the automated completion of relevant tasks in web environments. Ego2Web advances the field by offering a challenging, multimodal benchmark and a practical engineering architecture, thus pushing agents toward genuinely context-aware, cross-modality capabilities (Yu et al., 23 Mar 2026, Yang et al., 1 Mar 2026).

1. Motivation and Conceptual Scope

Existing web-agent benchmarks—such as WebArena, VisualWebArena, and OSWorld—are limited to digital contexts, evaluating perception and interaction purely within browser sessions or DOM structures. Such platforms cannot assess scenarios requiring a grounded pipeline where the agent first perceives physical objects or events, for example using smart glasses, and then acts effectively within digital interfaces (e.g., online shopping based on observed objects in the user's environment). Ego2Web addresses this critical limitation by providing the first benchmark to pair egocentric video streams with web tasks that specifically require: (1) unstructured visual perception, (2) precise spatio-temporal grounding (“Which item did the user pick up third in the video?”), and (3) web-based planning and execution (search, filter, purchase, etc.). This workflow models emerging use cases such as AR assistants and wearable AI for accessibility and daily-life navigation (Yu et al., 23 Mar 2026, Yang et al., 1 Mar 2026).

2. Benchmark Construction and Data Pipeline

Ego2Web employs a semi-automatic pipeline to curate high-quality, visually grounded video-task pairs:

• Video Sourcing: 3-minute, first-person clips are sampled from egocentric datasets including Ego4D and EgoSchema, each comprising at least 30 spoken narrations and dense user interactions.
• Task Synthesis: For each video profile $V_{meta}$, syntheses are generated by passing dense, 5 s clip-level JSON captions (derived by a frozen MLLM such as Qwen3-VL) and a set of allowed domains to an LLM planner (e.g., GPT-5). Instructions are produced that demand the integration of at least two visual cues and stipulate explicit “Must Match” entities or features.
• Human Annotation: Annotators rigorously verify each candidate for (a) visual grounding, (b) practical web feasibility, and (c) unambiguous, clear task phrasing. Improperly grounded or infeasible examples are discarded or edited.
• Task Domains: The benchmark comprises 500 total pairs, partitioned across the following task categories:

Category	Example Domains
E-Commerce	Amazon, Walmart, Etsy
Media Retrieval	YouTube, IMDb, Bilibili
Knowledge Lookup	Wikipedia, StackExchange
Local/Maps	Google Maps, Yelp, TripAdvisor
Others	Booking engines, social/news sites

Representative tasks might include: identifying a precisely specified physical object from video (“mint-green toaster”), then searching and purchasing the same model online (Yu et al., 23 Mar 2026).

3. Evaluation Protocol: Ego2WebJudge

Ego2Web introduces Ego2WebJudge, a multimodal LLM-as-a-Judge protocol, for scalable, high-fidelity evaluation:

• Inputs: Task instruction $I$, annotated keyframe(s) $v$ (with ground-truth visual cues), agent action history $A = \{a_1, \ldots, a_n\}$, screenshots $S = \{s_1, \ldots, s_k\}$, and LLM-extracted key-points defining atomic success criteria.
• Workflow:

1. Key-point Extraction: The LLM decomposes $I$ into precise, testable requirements.
2. Key-screenshot Selection: Each screenshot $s_i$ is rated for task relevance; only those above a fixed threshold $\delta$ are included for judgment.
3. Multimodal Integration: The system inputs all context to a multimodal LLM, which issues: $v$6

• Agreement: Quantitative evaluation yields approximately $84\%$ agreement between Ego2WebJudge and human raters, outperforming previous evaluation strategies by $6$–$I$0 percentage points (Yu et al., 23 Mar 2026).
• Significance: This robust agreement supports the protocol’s scalability for large-batch benchmarking with minimized manual overhead.

4. Empirical Results and Error Analysis

Experimental evaluation on Ego2Web reveals substantial headroom across all categories for current state-of-the-art multimodal agents:

• Overall Success Rates (SR; evaluated with both human annotators and Ego2WebJudge):

Agent + Methodology	Human SR (%)	Ego2WebJudge SR (%)
Browser-Use (Gemini-3-Flash)	58.6	57.2
SeeAct, Claude, GPT-5.4 (others)	<45	<45

• Oracle gap: Up to $I$1 headroom to reach the theoretical upper bound.
• Domain-Specific Performance (average SR, Ego2WebJudge w/ Gemini-2.5 Pro):

Task Category	Avg. SR (%)
E-Commerce	21.7
Media Retrieval	30.1
Knowledge Lookup	50.0
Local/Maps	23.1
Others	14.4

Easiest cases involve structured lookup (e.g., Wikipedia), whereas highly interactive or visually precise domains (e.g., maps, booking) exhibit markedly lower success.

• Error Taxonomy (by proportion of failures):

1. Object Misidentification: $I$2—incorrect grounding of video entities.
2. Temporal Misunderstanding: $I$3—incorrect sequence or timing recognition.
3. Cross-modal Retrieval Failure: $I$4—object is grounded but wrong data extracted.
4. Coarse Matching: $I$5—accepting only semantically similar entities.
5. Other: Planning timeouts, CAPTCHA barriers, banking flows.

This suggests that stronger video perception and event localization are critical for future progress (Yu et al., 23 Mar 2026).

5. Ablation Studies and Role of Visual Input

Ablation studies quantify the reliance of agent performance on the modality and quality of visual input (Browser-Use with Gemini-3-Flash, evaluated via Ego2WebJudge Gemini-2.5 Pro):

• No visual input (text-only): SR = 4.4%
• Detailed captions only (Gemini-3.1-Pro summaries): SR = 23.6%
• Raw video input: SR = 48.2%

Direct use of video more than doubles the success rate over text-only proxies, indicating that fine-grained spatio-temporal understanding cannot be substituted by caption generation or other lossy intermediate representations. Captions capture some semantics, but lack critical details for precise task grounding. A plausible implication is that multimodal web agents must tightly integrate continuous visual streams directly into their planning and action pipelines (Yu et al., 23 Mar 2026).

Further, error propagation analysis shows that early failures in egocentric video interpretation (object or event misalignment) systematically degrade subsequent web search queries, result filtration, and the verification of candidate solutions. This underscores the necessity of robust, end-to-end multimodal reasoning architectures.

6. System Architecture for Egocentric-Web Agents

Operational Ego2Web systems, as described in Egocentric Co-Pilot (Yang et al., 1 Mar 2026), deploy an event-driven, neuro-symbolic framework distributed across smart glasses, optional edge/cloud infrastructure, and browser-based service endpoints:

• Pipeline:
  • Smart Glasses: Hardware-level preprocessing (VAD for audio, cropping for video, IMU-based gaze tracking), upstreamed via WebRTC (H.264 video, Opus audio, JSON data).
  • On-Device Multiplexing: WebRTC/LiveKit clients organize streams.
  • Cloud Gateway: Receives multimodal data, orchestrates further processing.
  • Core Orchestration:
  • Egocentric Reasoning Core: Unified event log $I$6, enabling Temporal Chain-of-Thought (T-CoT) for fine-grained local reasoning and Hierarchical Context Compression (HCC) for scalable, long-horizon context utilization.
  • Multimodal Intent Layer: Fuses noisy ASR, gaze, and visual cues into structured, pluggable intent schemas.
  • Web Tools Toolbox: Encapsulates perception, symbolic reasoning, and off-the-shelf web APIs, callable via plan–then–execute protocols.
  • Backend Services: Support for CalendarAPI, MapAPI, NutritionAPI, ChessEngine, etc., with event-driven invocation.

This architectural scaffold shows how egocentric sensing can be leveraged for comprehensive, context-aware web agency (Yang et al., 1 Mar 2026).

7. Applications, Evaluation, and Future Impact

Ego2Web benchmarks and system blueprints are evaluated both in silico and in human-in-the-loop paradigms:

• Egocentric QA benchmarks (Egolife, HD-EPIC): Ego2Web outperforms baseline SOTA MLLMs in QA accuracy by $I$7–$I$8 points, with mean latencies below $I$9 seconds (Yang et al., 1 Mar 2026).
• Assistive scenarios: In human studies, Ego2Web achieves a $v$0 task completion rate and a user satisfaction rating of $v$1 vs. $v$2 and $v$3 for commercial smart-glass agents. Completion time is reduced from $v$4s (commercial) to $v$5s (Ego2Web), approaching human assistant performance.
• End-to-end city navigation example: Agents can resolve ambiguous, context-dependent queries (e.g., “Where is the café I saw on 5th?”) by integrating egocentric observation, temporal reasoning, and live map search, returning natural-language step-by-step directions with sub-second latency.

This suggests that comprehensive egocentric-to-web pipelines offer a practical route to accessible, always-on digital assistants, especially for populations with visual or cognitive impairments.

Major research directions motivated by Ego2Web include tighter integration of video perception with symbolic web planning, robust error recovery in cross-modal pipelines, and scalable, auditable evaluation protocols for agentic multimodal AI.

---

References:

• "Ego2Web: A Web Agent Benchmark Grounded in Egocentric Videos" (Yu et al., 23 Mar 2026)
• "Egocentric Co-Pilot: Web-Native Smart-Glasses Agents for Assistive Egocentric AI" (Yang et al., 1 Mar 2026)