Ego2WebJudge: Multimodal Web Agent Evaluator

• Ego2WebJudge is a multimodal evaluation framework that enforces real-world visual grounding by integrating egocentric video keyframes with web interaction data.
• It employs systematic key-point extraction and screenshot relevance filtering with LLM-based judgment to accurately mirror human annotation.
• The framework outperforms predecessors by increasing agreement rates by 4–6 percentage points, thus enhancing reliability and scalability.

Ego2WebJudge is an automatic evaluation framework developed for the Ego2Web benchmark, which assesses web agents' ability to execute real-world tasks that require integrating egocentric visual perception and web interaction. Unlike prior evaluators that focus exclusively on web-based interactions, Ego2WebJudge enforces strict multimodal grounding linked to the user's physical environment by incorporating annotated keyframes from egocentric videos, filtered task-relevant screenshots, and explicit task specifications. As a LLM-as-a-Judge system, Ego2WebJudge achieves substantially higher agreement with human annotation than predecessor methods, setting a new standard for robust, scalable agent evaluation in tasks that span the physical and digital domains (Yu et al., 23 Mar 2026).

1. Motivation and Design Goals

Ego2WebJudge addresses a major limitation in existing web agent benchmarks, such as WebVoyager and WebJudge, which lack grounding in real-world visual context and are susceptible to agent outputs that superficially match instructions without corroborating physical evidence. The design is motivated by three primary goals:

• Practical reliability: Achieve high annotator–LLM agreement for robust benchmarking.
• True multimodal grounding: Success must reflect satisfaction of both textual instructions and visual constraints present in egocentric video evidence.
• Scalability: Automate evaluation over large volumes of live web-agent interactions without constant human adjudication (Yu et al., 23 Mar 2026).

2. Architecture and Evaluation Workflow

Ego2WebJudge operates in three sequential phases, using a multimodal LLM backbone throughout:

A. Key-Point Extraction:

For every episode, the judge LLM distills the agent’s natural language task instruction $I$ into a set of explicit key points $K = \{k_1, ..., k_m\}$ critical for task completion. Each key point corresponds to a required constraint (e.g., "item color = mint", "brand = KitchenAid"), ensuring that evaluation is fine-grained and directly tied to instructive intent.

B. Screenshot Relevance Filtering:

The agent's full interaction trajectory typically yields 5–20 screenshots, $S = \{s_1, ..., s_n\}$. Each screenshot is rated for relevance to these key points through an LLM-based 1–5 scale. Screenshots above a set threshold ($\delta = 3$) are retained as $S_{\text{key}}$, discarding ancillary or irrelevant pages (e.g., login screens).

C. Strict Success/Failure Judgment:

The judge LLM receives as input:

• The original instruction $I$ and extracted key points $K$,
• Egocentric video evidence (provided as representative keyframes),
• A textual trace of the agent’s action history,
• The filtered, task-relevant screenshots $S_{\text{key}}$.

A system prompt enforces rigorous verification, directing the judge to return "Success" only if every key point is visibly and textually satisfied, with any ambiguity resulting in "Failure." The process can be summarized as: $K = \{k_1, ..., k_m\}$3 A simplified excerpt of the judging template:

“You have: (1) Egocentric video keyframes; (2) Task instruction; (3) Key points; (4) Agent’s action history; (5) Key screenshots. You must strictly verify that every key point is satisfied and that the final webpage is visually consistent with the video evidence. If any doubt or mismatch, mark Failure.”

3. Evaluation Metrics

The framework quantifies performance primarily through agreement with human annotation. For $N$ total episodes, with human judgment $h_i \in \{\texttt{S}, \texttt{F}\}$ and automatic output $K = \{k_1, ..., k_m\}$0, the following metrics are defined:

• Agreement Rate (AR):

$K = \{k_1, ..., k_m\}$1

• Precision, Recall, and F1 (for the Success class):

$K = \{k_1, ..., k_m\}$2

Here, TP is true positives, FP is false positives, and FN is false negatives, following standard usage. This provides a rigorous, interpretable evaluation aligned with annotation standards.

4. Experimental Results and Comparative Analysis

Ego2WebJudge was benchmarked using a suite of SOTA agent baselines (SeeAct, Browser-Use with GPT-4.1 and Gemini-3-Flash, Claude 3.7 & 4.5, GPT-5.4) and a panel of LLM-based judge models (Qwen3-VL-Flash, Gemini-2.5-Pro, GPT-4o). Human ground truth was obtained by majority among three annotators.

Comparison Table: Agreement Rate (AR) with human judgments (average across six agents):

Judge Model	WebVoyager	WebJudge	Ego2WebJudge
Gemini-2.5-Pro	70.7%	76.1%	80.8%
GPT-4o	74.7%	78.4%	84.0%

Ego2WebJudge consistently outperforms prior methods, increasing agreement by 4–6 percentage points. Notably, the fully multimodal evaluation prevents false positives common in text- or screenshot-only approaches, such as undetected mismatches of visual attributes. The agent with highest human success (BU-Gemini-3-Flash, 58.6%) receives an Ego2WebJudge score within two percentage points, indicating tight calibration (Yu et al., 23 Mar 2026).

5. Comparative Architecture: Distinguishing Features

Earlier evaluation protocols (WebVoyager, WebJudge) rely on agent action text or unfiltered screenshots, lacking real-world visual grounding and prone to success claims based on superficial string or layout matching. These methods admit misclassifications if, for instance, a page’s title matches the instruction but the agent fails on attribute compliance (e.g., correct color or brand only verifiable via video evidence).

Ego2WebJudge addresses this by:

• Incorporating egocentric video keyframes into the judging prompt, enabling attribute cross-validation with the real world.
• Filtering screenshots for relevance, suppressing context overflow.
• Imposing a strict checklist via key-point extraction, minimizing LLM hallucinations. Ablations show that omission of key-point extraction or screenshot filtering lowers judgment coherence, and removing video evidence drops AR by approximately five points, confirming the necessity of full multimodal context (Yu et al., 23 Mar 2026).

6. Limitations and Prospective Improvements

Several limitations are intrinsic to the current design:

• Context Window Constraints: Very long agent trajectories can exceed the LLM’s prompt capacity, even post-filtering.
• Video Coverage: Sampling only a few keyframes risks omitting temporally distributed cues; denser frame selection or learned visual embeddings suggest a likely path forward.
• Judge Bias: Single-LLM evaluation is exposed to systematic calibration error; ensembling and integration of explicit image-text comparators may further boost reliability.
• Task Edge Cases: Scenarios involving CAPTCHAs, multi-user authentication, or highly dynamic web content are not robustly handled under the current rubric. This suggests future work may focus on dynamic prompt compression, vision module augmentation (e.g., color histogram comparisons), and multi-judge consensus.

7. Broader Relevance and System Integration

Ego2WebJudge is directly influencing emerging multimodal agent assessment pipelines. Its principles have informed the development of ancillary frameworks such as CHECK-MAT, where a web-service based, human-in-the-loop feedback architecture is proposed for large-scale, rubric-driven evaluation of multimodal tasks, including handwritten mathematical solution assessment (Khrulev, 29 Jul 2025). The method’s scalable, API-oriented design and explicit rubric alignment form a foundation for extending automatic judging into new domains, including feedback-driven continual fine-tuning and cross-task rubric harmonization.

In summary, Ego2WebJudge advances the state of automatic agent evaluation by enforcing strict alignment to both real-world and digital cues using multimodal LLM prompts, demonstrably improving reliability and grounding in web agent benchmarks and suggesting a roadmap for scalable, rubric-driven evaluation in multimodal AI systems (Yu et al., 23 Mar 2026, Khrulev, 29 Jul 2025).