WebForge-Bench: Browser-Agent Evaluation

• WebForge-Bench is a browser-agent benchmark that generates self-contained static websites embedding real-world noise to address the realism, reproducibility, and scalability trilemma.
• Its automated four-agent pipeline—Plan, Generate, Refine, and Validate—ensures consistent task generation and interactive challenges across 7 domains and 3 difficulty levels.
• The benchmark uses a seven-dimensional difficulty vector to profile agent capabilities, offering detailed insights beyond aggregate accuracy scores.

Searching arXiv for the specified paper and related benchmark context. WebForge-Bench is a browser-agent benchmark constructed by the WebForge framework to address what its authors describe as the realism–reproducibility–scalability trilemma in web-agent evaluation. It consists of 934 validated tasks spanning 7 domains and 3 difficulty levels, and is generated end-to-end without human annotation through a four-agent pipeline that produces self-contained static websites with embedded real-world data and real-web noise. The benchmark is explicitly designed for capability profiling rather than single-score ranking, using a seven-dimensional difficulty vector to control navigation depth, interaction burden, visual complexity, information density, reasoning demands, and risk (Yuan et al., 13 Apr 2026).

1. Benchmarking problem and design objective

Prior browser-agent benchmarks are described as falling into three broad categories. Real-website benchmarks such as WebVoyager and Mind2Web offer high realism, but their dependence on live sites introduces content drift and deprecation, which undermines reproducibility. Controlled sandbox benchmarks such as WebArena and EntWorld preserve reproducibility, but omit “real-web noise” such as pop-ups, cookie banners, and network delays, thereby simplifying the interaction regime. Manual-curation or schema-based automated benchmarks such as BenchAgents and AutoBencher improve some aspects of scale, but trade off interactivity and scalability.

WebForge-Bench is introduced as a direct response to this trilemma. Its core design premise is that realism, reproducibility, and scalability can be jointly pursued if tasks are instantiated as self-contained static websites with zero external dependencies, while still embedding authentic real-world data and noise. In this formulation, reproducibility derives from isolation from the live web; realism derives from noise injection and interaction structure; scalability derives from a fully automated generation and validation workflow.

The benchmark therefore occupies a distinct methodological position within browser-agent evaluation. It is neither a replay of live websites nor a purely synthetic toy environment. Instead, it is an automated benchmark-generation system whose outputs are intended to remain interactive, realistic, and exactly reproducible across repeated evaluation runs.

2. Four-agent generation and validation pipeline

WebForge transforms a domain–difficulty pair into a validated task through four specialized stages: Plan, Generate, Refine, and Validate (Yuan et al., 13 Apr 2026). Formally, for a domain $d \in \mathcal{D}$, an overall difficulty tier $l \in \{1,2,3\}$, and a per-dimension difficulty vector $\boldsymbol{\delta} \in \{1,2,3\}^7$, the pipeline is defined as

$$(d,l)\xrightarrow{f_{\rm plan}} \mathcal{P}\xrightarrow{f_{\rm gen}} \mathcal{W}\xrightarrow{f_{\rm refine}} \mathcal{W}^*\xrightarrow{f_{\rm val}} \{0,1\}.$$

Here $\mathcal{P}$ denotes a structured plan blueprint comprising the task objective, the difficulty vector $\boldsymbol{\delta}$, the page schema, and the solution path. The object $\mathcal{W}$ denotes a preliminary web environment consisting of HTML/CSS/JS files $\mathcal{S}$, answer configuration $\mathcal{A}$, and metadata $\mathcal{M}$. The refined environment $l \in \{1,2,3\}$0 is obtained after rule-based quality fixes and noise injection. The validator returns $l \in \{1,2,3\}$1 only if the task is solvable in at most 50 browser actions.

The Plan Agent uses a dual-stage LLM procedure:

$l \in \{1,2,3\}$2

where the first stage is a high-temperature draft and the second is a low-temperature refinement pass. The first stage produces a creative blueprint; the second verifies logical consistency and adjusts $l \in \{1,2,3\}$3 so that the overall difficulty satisfies the compositional constraint

$l \in \{1,2,3\}$4

The Generation Agent maps the plan into a fully static site,

$l \in \{1,2,3\}$5

and includes real-data lookups, localStorage state management for carts and forms, and encrypted ground-truth intended as anti-cheating protection. The Refinement Agent applies a quality ruleset $l \in \{1,2,3\}$6,

$l \in \{1,2,3\}$7

covering functional completeness, broken-link fixes, CSS/JS errors, and real-web noise including cookie banners, stochastic pop-ups, and network latency. Its operation is described as an Assess→Plan→Execute→Verify cycle.

The Validation Agent replays the official solution path in a Chromium browser under an Observe–Reason–Act loop with up to 50 actions. Its decision rule is

$l \in \{1,2,3\}$8

This final replay-based validation is intended to catch rendering defects and unsolvable cases that may survive earlier stages.

3. Seven-dimensional difficulty control

A central feature of WebForge-Bench is its explicit seven-dimensional difficulty framework. Each task receives a vector

$l \in \{1,2,3\}$9

The seven dimensions are Jump Depth, Jump Breadth, Page Interaction, Visual Complexity, Information Complexity, Reasoning/Calculation, and Risk Factor. They are defined as follows.

Dimension	Abbrev.	Level definitions
Jump Depth (page transitions)	depth	1: 1–2 pages; 2: 3–5; 3: $\boldsymbol{\delta} \in \{1,2,3\}^7$0
Jump Breadth (max options per page)	breadth	1: $\boldsymbol{\delta} \in \{1,2,3\}^7$1 links; 2: 3–5; 3: $\boldsymbol{\delta} \in \{1,2,3\}^7$2
Page Interaction (actions/page)	interact	1: $\boldsymbol{\delta} \in \{1,2,3\}^7$3 clicks/fields; 2: 3–5; 3: $\boldsymbol{\delta} \in \{1,2,3\}^7$4
Visual Complexity	vis	1: DOM-text only; 2: single chart/image; 3: multi-chart reasoning
Information Complexity	info	1: prominent key info; 2: moderate scan; 3: buried in noise/long docs
Reasoning/Calculation	reason	1: direct lookup; 2: simple arithmetic/filter; 3: multi-step logic, optimization
Risk Factor (irreversible ops)	risk	1: read-only; 2: irreversible w/ explicit confirm; 3: subtle irreversible

Overall task difficulty is not assigned independently of these dimensions; rather, it is enforced through combinatorial constraints on $\boldsymbol{\delta} \in \{1,2,3\}^7$5:

$\boldsymbol{\delta} \in \{1,2,3\}^7$6

This construction is explicitly intended to prevent a nominally hard task from being hard along only one axis. Level 3 therefore requires simultaneous pressure on multiple dimensions rather than a single isolated bottleneck. In methodological terms, the benchmark treats difficulty as a structured vector rather than a scalar, enabling capability diagnosis across heterogeneous cognitive and interactional demands.

4. Benchmark composition and construction statistics

WebForge-Bench is built from 1,260 candidates generated by crossing 7 domains, 3 difficulty levels, and 60 seeds per domain–level combination (Yuan et al., 13 Apr 2026). After validation, 934 tasks remain, yielding a pipeline pass rate of 74.1%.

The seven domains are:

1. Consumer Transaction/Service
2. Content Moderation/Compliance
3. Enterprise Process/Collaboration
4. Information Retrieval/Analysis
5. Platform Management/Ops
6. Tool Usage
7. Content Creation/Publishing

The domain- and level-specific validated counts show substantial variation. Information Retrieval/Analysis has the highest total pass rate at 160 tasks and 88.9%, while Consumer Transaction/Service has 115 tasks and 63.9%. By difficulty tier, Level 2 has the highest aggregate pass rate at 340 validated tasks and 81.0%, compared with 301 tasks and 71.7% for Level 1, and 293 tasks and 69.8% for Level 3.

A concise summary of the validated totals is given below.

Domain	Total validated	Pass rate
D1 (Consumer Tx)	115	63.9%
D2 (Moderation)	125	69.4%
D3 (Enterprise)	131	72.8%
D4 (Info Retrieval)	160	88.9%
D5 (Platform Mgmt)	131	72.8%
D6 (Tool Usage)	141	78.3%
D7 (Content Create)	131	72.8%
Total	934	74.1%

These statistics are informative in two senses. First, they indicate that automated benchmark generation is not assumed to be uniformly reliable across domains. Second, they show that the final benchmark is not balanced by forcing identical post-validation counts; instead, it preserves the empirical outputs of the generation-and-validation process.

5. Evaluation protocol and scoring

The evaluation protocol uses final-state accuracy under a bounded-action regime. Agents are allowed up to 50 browser actions, and only the final output is scored. The benchmark supports three answer types: Direct Answer, Operation Code, and Mixed. Operation Code uses an encrypted judge, and the comparison between agent output and ground truth is handled by an automatic LLM judge, thereby avoiding human annotation.

The model suite comprises 14 configurations across several categories. The closed-source multimodal group includes Gemini-3-Pro/Flash, Gemini-2.5-Flash-Lite, Claude-4.5-Sonnet, and GPT-5.2/Mini/Nano. The open-source multimodal group includes Kimi-K2.5, Qwen3-VL-235B, and Qwen3-Omni-30B. The text-only group includes DeepSeek-V3.2 and GLM-4.7. Some models were also evaluated in an ablation setting with DOM only and no screenshots.

This protocol places the emphasis on end-to-end task completion rather than stepwise supervision. Because the benchmark uses self-contained environments and automated judging, the evaluation stack is aligned with the benchmark’s broader claim of zero human annotation from construction through scoring.

6. Empirical results, capability profiles, and interpretive significance

The reported results indicate that difficulty stratification is strongly discriminative (Yuan et al., 13 Apr 2026). Averaged across models, Level 1 accuracy is 73.9%, Level 2 accuracy is 54.8%, and Level 3 accuracy is 28.1%, with an overall average of 52.6%. The paper summarizes this pattern by noting that Level 1 tasks are easy, Level 2 tasks produce a clear drop, and Level 3 tasks become strongly discriminative, with model performance ranging from 2% to 58%.

Among the explicitly listed models, Gemini-3-Pro attains 86.4% on Level 1, 82.1% on Level 2, 58.0% on Level 3, and 75.9% overall. Claude-4.5 achieves 85.7%, 74.7%, 48.1%, and 69.9%, respectively. Gemini-3-Flash records 82.4%, 73.5%, 44.0%, and 67.1%. These trajectories underscore that aggregate accuracy alone does not adequately characterize behavior under increasing task complexity.

Cross-domain analysis reveals further heterogeneity. The easiest domains on average are D4, Information Retrieval/Analysis, at 56.9%, and D7, Content Creation/Publishing, at 57.2%. The hardest are D1, Consumer Transaction/Service, and D2, Content Moderation/Compliance, both at 48.3%. The benchmark gives a concrete example of hidden capability bias: GPT-5-Mini scores 73.8% on D4 but only 50.4% on D3, a 23 percentage point swing obscured by aggregate metrics.

The visual modality is also shown to be materially important. Removing screenshots reduces overall accuracy by 16 to 17 percentage points; for Gemini-3-Pro, the reported drop is from 75.9% to 59.2%. This establishes that at least part of the benchmark’s difficulty is genuinely multimodal rather than reducible to DOM-only parsing.

Per-dimension results reinforce the utility of the seven-axis design. For Gemini-3-Pro, Jump Depth declines from 86.5% at Level 1 to 60.2% at Level 3; Jump Breadth from 84.8% to 51.2%; Page Interaction from 84.0% to 65.0%; Visual Complexity from 90.8% to 55.8%; Information Complexity from 84.7% to 53.2%; Reasoning/Calculation from 91.4% to 58.3%; and Risk Factor from 80.6% to 23.1%. The reported interpretation is that the breakdown confirms monotonic scaling and that Visual Complexity and Reasoning/Calc are particularly discriminative.

Two broader clarifications follow from these results. First, a common assumption in browser-agent evaluation is that a single aggregate score is sufficient for model comparison; WebForge-Bench is explicitly presented as evidence that cross-domain and per-dimension capability biases can be invisible under such aggregation. Second, realism and reproducibility are often treated as mutually exclusive in browser benchmarks; WebForge-Bench is framed as a counterexample by combining self-contained HTML/CSS/JS environments with noise injection and automated validation. A plausible implication is that future browser-agent evaluation may increasingly shift from monolithic leaderboards toward structured capability profiling across task dimensions, domains, and modality conditions.