WebForge: Browser-Agent Benchmark

• WebForge is a browser-agent benchmark framework that creates self-contained static web environments with realistic disturbances.
• It employs a four-agent pipeline and a seven-dimensional difficulty control to generate and validate tasks across 7 domains and 3 difficulty levels.
• Empirical results show varied model performance, with accuracy dropping significantly at higher difficulty levels, highlighting the impact of visual and reasoning challenges.

Searching arXiv for the cited works and nearby benchmark context to ground the article. arXiv search query: WebForge browser agent benchmark (Yuan et al., 13 Apr 2026) WebForge is the name of a 2026 arXiv framework for browser-agent benchmarking that targets the realism–reproducibility–scalability trilemma by automatically generating interactive, self-contained web environments through a four-agent pipeline—Plan, Generate, Refine, and Validate—and by organizing tasks with a seven-dimensional difficulty control framework. Using this pipeline, the authors construct WebForge-Bench, a benchmark of 934 tasks spanning 7 domains and 3 difficulty levels, and argue that multi-dimensional evaluation reveals capability profiles that single aggregate scores obscure (Yuan et al., 13 Apr 2026). The term is not used in the 2017 arXiv paper on LHC Electrical Quality Assurance; that work presents an unnamed “conceptual framework” for rapid web-application design and explicitly does not name it “WebForge” (Mertik et al., 2017).

1. Nomenclature and scope

In current arXiv usage, WebForge denotes the browser-agent benchmark framework introduced in “WebForge: Breaking the Realism-Reproducibility-Scalability Trilemma in Browser Agent Benchmark” (Yuan et al., 13 Apr 2026). Its central object is not a generic website-development toolkit, but a system for constructing benchmark tasks as fully packaged static web environments with deterministic evaluation, browser-executed validation, and controlled difficulty.

A necessary disambiguation arises because a separate paper, “A Conceptual Framework for Supporting a Rapid Design of Web Applications for Data Analysis of Electrical Quality Assurance Data for the LHC,” also concerns rapid web-application design, but the term “WebForge” does not appear in that paper. The authors there refer only to “the conceptual framework,” “the framework,” or “the framework for rapid design,” in the context of ELQA data analysis for CERN’s LHC (Mertik et al., 2017). Accordingly, “WebForge” properly refers to the 2026 browser-agent benchmark framework rather than the 2017 ELQA dashboard architecture.

2. The trilemma WebForge is designed to address

WebForge is motivated by a benchmark-design problem formulated as a three-way trade-off. Real-website benchmarks maximize realism but lose reproducibility because of content drift and evaluation noisiness. Controlled environments guarantee reproducibility but sacrifice realism by omitting real-web disturbances such as pop-ups, cookie dialogs, and latency, while also depending on manual curation. Automated benchmark construction promises scalability, yet prior work is described as largely targeting non-interactive or non-browser tasks and as not automatically generating complete, interactive web environments with multi-dimensional difficulty control (Yuan et al., 13 Apr 2026).

The framework therefore defines its design target through three simultaneous properties. It is intended to be realistic by injecting real-web disturbances and using real-world data and design priors; reproducible by packaging each task as a self-contained static website with deterministic final-state evaluation and browser-executed validation; and scalable by using a fully automated four-agent pipeline that generates tasks and environments end-to-end without human annotation.

This formulation also structures WebForge’s comparative position relative to earlier benchmarks. Real-website suites such as Mind2Web, WebVoyager, and MMInA are presented as realistic but vulnerable to drift and nondeterminism. Controlled suites such as WebArena, VisualWebArena, WorkArena/WorkArena++, and EntWorld are presented as reproducible but comparatively “clean,” manually intensive, and weak in a priori difficulty control. WebForge’s claim is not that it reproduces the live web, but that it produces benchmark environments that preserve selected forms of realism while retaining determinism and automation (Yuan et al., 13 Apr 2026).

3. Four-agent pipeline and environment generation

Given a domain $d$ from a set of seven domains and a difficulty level $l \in \{1,2,3\}$, WebForge maps $(d,l)$ through four specialized agents. The Plan Agent outputs a task blueprint $\mathcal{P}$, including an objective, a seven-dimensional difficulty vector $\delta \in \{1,2,3\}^7$, a solution path, and a ground-truth answer type. The Generation Agent produces a web environment $$\mathcal{W} = (\mathcal{S}, \mathcal{A}, \mathcal{M})$$, where $\mathcal{S}$ denotes site files, $\mathcal{A}$ the answer configuration, and $\mathcal{M}$ metadata. The Refinement Agent transforms this into a repaired environment $$\mathcal{W}^* = f_{\text{refine}}(\mathcal{W}, \mathcal{R})$$, and the Validation Agent verifies solvability inside a Chromium-based browser (Yuan et al., 13 Apr 2026).

Agent	Input	Output
Plan	$l \in \{1,2,3\}$0	Task blueprint $l \in \{1,2,3\}$1
Generate	$l \in \{1,2,3\}$2	$l \in \{1,2,3\}$3
Refine	$l \in \{1,2,3\}$4	$l \in \{1,2,3\}$5
Validate	$l \in \{1,2,3\}$6, solution path	Binary solvability verdict

The Plan Agent is dual-stage. Stage 1 uses a creativity-oriented LLM at high temperature $l \in \{1,2,3\}$7 to produce diverse drafts; Stage 2 uses a precision-oriented LLM at low temperature $l \in \{1,2,3\}$8 to perform logic checks, enforce difficulty rules, and revise 30–50%+ of the draft. The blueprint must be self-contained, solvable, deterministic, and answer-verifiable.

The Generation Agent interprets the plan, fills placeholders with real data via search while preserving domain intent and difficulty vector $l \in \{1,2,3\}$9, collects visual and design priors, and constructs a fully navigable static site in HTML/CSS/JS. Every navigation element must point to a working page, and stateful interactions such as shopping carts and form workflows are implemented via localStorage. The framework also specifies final-state-only evaluation and three answer types: Direct Answer, Operation Code, and Mixed. To deter trivial extraction, it uses encrypted data storage such as Base64-encoded answers, deceptive error or confirmation codes, and code obfuscation.

The Refinement Agent performs rule-driven QA with an Assess → Plan → Execute → Verify loop. It checks functional completeness, visual correctness, state determinism, task security, realism, and interaction feedback; repairs dead links and blocking dialogs; and injects noise such as pop-up advertisements, cookie consent banners, and simulated network latency. These disturbances are dismissible and persisted via localStorage, which is intended to mimic real browsing while preserving solvability.

The Validation Agent replays the provided solution path in the same class of browser used for evaluation, using an Observe–Reason–Act loop with up to 50 steps and 3 retries for transient failures. Its verdict is defined as

$(d,l)$0

The failure modes it flags include ground-truth mismatch, reasoning logic flaws in the provided solution path, and repeated action failures (Yuan et al., 13 Apr 2026).

4. Difficulty control and benchmark composition

A defining feature of WebForge is its seven-dimensional difficulty control framework. The dimensions are Jump Depth, Jump Breadth, Page Interaction, Visual Complexity, Info Complexity, Reasoning/Calc, and Risk Factor. Each dimension is graded from L1 to L3 and is operationalized by the Plan Agent and enforced during refinement (Yuan et al., 13 Apr 2026).

For Jump Depth, L1 corresponds to 1–2 page transitions, L2 to 3–5, and L3 to 6+. For Jump Breadth, L1 involves 1–2 links or options to evaluate on a page, L2 3–5, and L3 6+. Page Interaction ranges from 1–2 simple interactions at L1, to 3–5 multi-field forms or dropdowns at L2, to 6+ interactions such as complex wizards at L3. Visual Complexity ranges from all information being present in text or DOM at L1, through simple chart or image reading at L2, to correlation across multiple visuals at L3. Info Complexity ranges from prominent key information at L1 to high-density, noisy, or buried information at L3. Reasoning/Calc ranges from direct lookup at L1 to multi-step reasoning, optimization, and complex math at L3. Risk Factor ranges from read-only tasks at L1 to subtle irreversible actions with no obvious warnings at L3.

Overall level assignment is constrained by explicit aggregation rules. Level 1 permits at most 2 dimensions at L2, requires all others at L1, and allows no L3. Level 2 requires at least 2 dimensions at L2 and permits at most 1 dimension at L3. Level 3 requires at least 2 dimensions at L3 and at least 2 at L2 (Yuan et al., 13 Apr 2026).

WebForge-Bench is built by generating 1,260 candidate tasks, corresponding to 7 domains × 3 levels × 60 tasks per cell. Of these, 934 pass validation, giving an overall pipeline pass rate of 74.1%. The 7 domains are Consumer Transaction/Service, Content Moderation/Compliance, Enterprise Process/Collaboration, Info Retrieval/Analysis, Platform Management/Ops, Tool Usage, and Content Creation/Publishing. Per-domain totals range from 115 tasks in D1 to 160 in D4, with per-domain pass rates from 63.9% to 88.9% (Yuan et al., 13 Apr 2026).

The benchmark’s statistical profile is also reported. Across the 934 tasks, the per-dimension distributions show substantial use of L2 settings, with dimension-specific variation. For example, Jump Depth is distributed as L1 288 (30.8%), L2 380 (40.7%), and L3 266 (28.5%), while Risk Factor is much more concentrated at L1, with only 13 L3 tasks (1.4%). Inter-dimension correlations, measured by Spearman coefficients, have average off-diagonal absolute value $(d,l)$1 with range 0.262–0.754. The strongest reported correlation is Jump Depth versus Reasoning/Calc at $(d,l)$2, whereas Page Interaction and Risk Factor have the weakest average correlations. Task solvability by difficulty, defined as “solved by at least one of 14 evaluated models,” is 95.0% for L1, 93.2% for L2, and 76.5% for L3, for an overall 88.5% (Yuan et al., 13 Apr 2026).

5. Evaluation protocol and empirical results

WebForge evaluates 14 model configurations spanning closed-source multimodal systems, open-source multimodal systems, and text-only systems. Multimodal models are also tested in DOM-only “T” variants to isolate the effect of visual input. Evaluation is executed in a Chromium-based browser in GUI mode, with up to 50 actions per task. Agents receive either DOM only or DOM plus screenshots. The success criterion is final-state correctness, compared against ground truth by a simple evaluator LLM string comparison; no semantic judging is required (Yuan et al., 13 Apr 2026).

The primary metric is accuracy. Secondary run-time diagnostics include turns per task, actions per task, and prompt/output tokens aggregated by difficulty level for logging-capable models. Results are reported by overall difficulty level, by domain, and by individual difficulty dimension, so that model behavior can be profiled beyond a single aggregate score.

On the full 934-task benchmark, the highest overall accuracy reported is for Gemini-3-Pro at 75.9%, with difficulty-stratified accuracies of 86.4% on L1, 82.1% on L2, and 58.0% on L3. Claude-4.5-Sonnet reports 69.9% overall; Gemini-3-Flash 67.1%; Kimi-K2.5 66.4%; GPT-5.2 59.5%; DeepSeek-V3.2 48.8%; and GLM-4.7 50.2%. Difficulty stratification is pronounced: Level-3 performance ranges from 2.4% for Qwen3-Omni-30B to 58.0% for Gemini-3-Pro, a spread of 56 points. Averaged across models, the easiest domains are D7 Content Creation/Publishing at 57.2% and D4 Info Retrieval/Analysis at 56.9%, while the hardest are D1 Consumer Transaction/Service and D2 Content Moderation/Compliance, both at 48.3% (Yuan et al., 13 Apr 2026).

Ablation studies are used to isolate the effect of major design decisions. Removing visual input from multimodal systems produces substantial drops: Gemini-3-Pro declines from 75.9% to 59.2%, and Gemini-3-Flash from 67.1% to 51.2%, with the Level-3 drop for Gemini-3-Pro exceeding 20 percentage points. Pipeline ablation on the 1,260 candidates reduces validation pass rate from 74.1% for the full pipeline to 59.5% when Plan refinement is removed and to 51.4% when the Refinement Agent is also removed. Runtime scaling is also reported: prompt tokens increase by roughly 5–8× from Level 1 to Level 3, and actions and turns increase similarly; at Level 3, Claude-4.5-Sonnet is reported as most expensive at 1,608K prompt tokens, while Gemini-3-Pro uses 1,036K (Yuan et al., 13 Apr 2026).

Per-dimension analysis is central to the benchmark’s argument. Accuracy decreases monotonically from L1 to L3 across all seven dimensions. Visual Complexity and Jump Breadth are described as particularly discriminative; for example, Gemini-3-Pro drops from 90.8% to 55.8% across Visual Complexity levels. Reasoning/Calc sharply separates stronger from weaker models at L3, with Gemini-3-Pro at 58.3% and GPT-5-Nano at 6.4%. Because Risk Factor has only 1.4% L3 tasks, the paper notes that large drops there may partly reflect small-sample variability (Yuan et al., 13 Apr 2026).

The paper also includes an illustrative Level-3 Consumer Transaction/Service task in which an agent must cross-reference a color-coded pricing heatmap with a seasonal bloom chart, choose a Saturday satisfying multiple constraints, book for 80 guests with Premium catering, and return both confirmation code and total. The ground truth is the date 2026-05-16, total $11,440.00, and confirmation code GEG-2026-05841. The site includes Base64-encoded answers, deceptive codes for specific error types, and dismissible noise elements such as a cookie banner and a promotional popup (Yuan et al., 13 Apr 2026).

6. Reproducibility mechanisms, limitations, and projected extensions

WebForge’s reproducibility model rests on self-contained static packaging. Each task is a folder of HTML/CSS/JS, images, encrypted data, and metadata; no external APIs, servers, or services are required, and opening index.html in a Chromium-based browser suffices to load the environment. Determinism is further supported by localStorage-based state management, bundled assets, and final-state evaluation based on exact output matching rather than step-level semantic interpretation (Yuan et al., 13 Apr 2026).

The framework’s realism is intentionally selective rather than exhaustive. It injects disturbances associated with ordinary browsing—cookie banners, pop-ups, and latency—but keeps them dismissible and persistent so that they do not compromise solvability. This suggests a design choice in which “realism” is operationalized as the presence of common nuisance factors rather than as full fidelity to live, networked services. The paper’s own limitations support that interpretation: it states that there are no true backends, and therefore the system cannot fully represent tasks requiring live APIs, multi-user concurrency, persistent server-side state, or genuinely real-time data such as stock prices or flight availability (Yuan et al., 13 Apr 2026).

Additional constraints are reported. Some widgets, including date inputs, can be brittle across agent frameworks. Risk Factor L3 is naturally rare because subtle irreversible actions without warnings are uncommon; the release did not enforce targeted generation of such tasks. More broadly, the paper does not position WebForge as a complete substitute for live-web evaluation, but as an attempt to combine realism, reproducibility, and automation more effectively than prior benchmark classes.

The stated future directions are to extend the pipeline to generate large-scale training data for web agents rather than only evaluation tasks, to incorporate lightweight backends such as serverless sandboxes for richer state and multi-user realism while preserving reproducibility, and to expand domains and dimensions while allowing configurable distributions—for example, enforcing Risk Factor L3 for safety studies (Yuan et al., 13 Apr 2026).