WebWorld-Bench: Web Simulation Evaluation

• WebWorld-Bench is an intrinsic benchmark for web world models that assesses simulated next states using dual metrics, Factuality Score and Web Turing Score.
• It employs a dual-metric design combining pointwise functional assessments and pairwise realism judgments across nine dimensions including long-horizon consistency and multi-format robustness.
• The benchmark isolates simulator quality from downstream performance, enabling researchers to analyze causal effects, schema alignment, and simulation fidelity comprehensively.

Searching arXiv for the cited paper and closely related benchmarks mentioned in the provided data. arXiv search query: "WebWorld: A Large-Scale World Model for Web Agent Training" WebWorld-Bench is an intrinsic evaluation suite for web world models introduced in "WebWorld: A Large-Scale World Model for Web Agent Training" (Xiao et al., 16 Feb 2026). It measures two properties of action-conditioned web simulation: how accurately a model captures the causal effect of an action on the open web, and how realistic the resulting simulated state appears relative to the real browser next state. The benchmark is built around dual metrics—Factuality Score and Web Turing Score—applied across nine evaluation dimensions spanning long-horizon consistency, semantic transitions, fine-grained sensitivity, multi-tab context, multi-format robustness, and natural-language verbalization of state changes. Within the paper’s framework, WebWorld-Bench serves as an intrinsic benchmark that isolates simulator quality, in contrast to extrinsic evaluation, which measures downstream agent performance when trained with or using the world model (Xiao et al., 16 Feb 2026).

1. Definition and evaluative role

WebWorld-Bench evaluates simulated next states directly against recorded browser trajectories. Its central object of measurement is the predicted state $\hat{s}_{t+1}$ conditioned on an instruction $I$ (if any) and an interaction history $h_t = (s_0, a_0, \ldots, s_t, a_t)$. The benchmark then compares $\hat{s}_{t+1}$ with the logged ground-truth next state $s_{t+1}$ using two judge-based procedures: a pointwise functional assessment and a pairwise realism assessment (Xiao et al., 16 Feb 2026).

This design distinguishes intrinsic evaluation from extrinsic evaluation. In the paper’s terminology, intrinsic evaluation directly scores simulated next states given history, action, and ground truth, thereby isolating simulator quality independent of downstream agent loops. Extrinsic evaluation instead measures downstream task outcomes when the world model is used for trajectory synthesis or inference-time search in real environments such as WebArena and MiniWob++, and therefore conflates simulator quality with planner, value model, and tool-use components (Xiao et al., 16 Feb 2026).

The benchmark is explicitly motivated by the difficulty of evaluating open-web simulation using standard structural similarity metrics. The paper states that structural DOM or HTML similarity often yields uniformly low scores on the open web because of high structural variance, while semantic overlap metrics such as ROUGE or BERTScore can be insensitive to complex changes. WebWorld-Bench addresses that problem with a dual-metric design combining functional correctness and perceptual realism (Xiao et al., 16 Feb 2026).

2. Benchmark composition and coverage

WebWorld-Bench is constructed from open-web tasks collected using the same hierarchical pipeline as training, but the evaluation data are strictly held out to avoid contamination. The benchmark spans diverse domains, with the paper citing lifestyle, tech, and education as examples. Exact counts per dimension are not specified, but all reported scores are normalized to $[0,1]$ or presented as percentages in tables (Xiao et al., 16 Feb 2026).

The benchmark supports long-horizon simulations and multiple state representations. Trajectories of up to $30+$ turns are supported by the simulator, while the long-horizon evaluation subset uses trajectories exceeding 10 steps. States are primarily represented as accessibility trees, with converted representations including XML, HTML, Markdown, and Playwright extraction. A dedicated Web2NAL setting evaluates verbal descriptions of changes rather than structured state output (Xiao et al., 16 Feb 2026).

The action space is unified and includes element-based actions such as click, fill, select_option, and hover, as well as coordinate mouse actions, keyboard actions, browser navigation and tab operations, and meta-control actions. This breadth is relevant because WebWorld-Bench evaluates not only local DOM modifications but also inter-page continuity and format conversion robustness (Xiao et al., 16 Feb 2026).

The nine evaluation dimensions are as follows:

Dimension	Coverage in the benchmark	Construction cue
Long-Horizon Consistency	Trajectories with $>10$ steps	Full history provided
Base Semantics	Macroscopic page transitions	Semantic transitions
Fine-Grain Sensitivity	Local, minimal UI changes	LLM-filtered actions
Multi-Tab	Inter-page transitions in one session	Multi-page continuity
XML	Multi-format robustness	Post-hoc state transformation
HTML	Multi-format robustness	Post-hoc state transformation
Markdown	Multi-format robustness	Post-hoc state transformation
Playwright	Multi-format robustness	Post-hoc state transformation
Web2NAL	Natural-language verbalization	Description of state changes

The benchmark’s structure makes it possible to probe different failure modes separately. Long-Horizon Consistency stresses accumulation of state over extended histories; Fine-Grain Sensitivity targets minimal but causally important UI changes such as toggles and dropdown expansions; Multi-Tab targets continuity across pages in the same session; and the four format dimensions test whether learned dynamics transfer across XML, HTML, Markdown, and Playwright renderings of the same underlying state (Xiao et al., 16 Feb 2026).

3. Metrics and score aggregation

WebWorld-Bench uses two metrics: Factuality Score and Web Turing Score. The former is a pointwise metric for functional correctness; the latter is a pairwise metric for perceptual realism (Xiao et al., 16 Feb 2026).

For Factuality Score, the judge assigns a per-step rubric value

$s_{i,t} \in \{0.0, 0.4, 0.7, 1.0\}.$

The rubric is defined as follows: $1.0$ if the main causal effect is present and unambiguous, $I$0 if it is mostly present with incompleteness, $I$1 if it is applied to the right kind of element but has a largely wrong effect, and $I$2 if the main effect is missing. Per-trajectory, per-dimension, and overall factuality are then aggregated by arithmetic averaging:

$I$3

$I$4

$I$5

For Web Turing Score, the judge is shown two anonymized candidates at each evaluated step: the model’s simulated next state and the real browser next state. The judge must choose which state is “more believable and realistic”; the prompt permits no ties. The per-step outcome is

$I$6

if the simulated state is chosen as more realistic, and $I$7 otherwise. Aggregation proceeds in the same way:

$I$8

$I$9

$h_t = (s_0, a_0, \ldots, s_t, a_t)$0

In ablation experiments, the paper also reports a composite score,

$h_t = (s_0, a_0, \ldots, s_t, a_t)$1

and explicitly notes that Table 5 values match this arithmetic mean. All metrics lie in $h_t = (s_0, a_0, \ldots, s_t, a_t)$2, though the main results table presents them as percentages (Xiao et al., 16 Feb 2026).

Several implementation-relevant details follow from the metric definitions. Partial credit is built directly into the factuality rubric. Invalid or format-misaligned outputs are not penalized by a separate rule; instead, they depress both metrics because the judges score the output as produced. In the long-horizon dimension, full interaction history is provided, and scoring averages uniformly over steps before averaging across trajectories and dimensions (Xiao et al., 16 Feb 2026).

4. Evaluation protocol and judge models

The benchmark operates on logged trajectories rather than live browser execution. States come from the browser’s accessibility layer via Playwright and BrowserGym. For each test case defined by dimension $h_t = (s_0, a_0, \ldots, s_t, a_t)$3, trajectory $h_t = (s_0, a_0, \ldots, s_t, a_t)$4, and step $h_t = (s_0, a_0, \ldots, s_t, a_t)$5, the model receives the instruction $h_t = (s_0, a_0, \ldots, s_t, a_t)$6 if present and the history $h_t = (s_0, a_0, \ldots, s_t, a_t)$7, predicts a simulated next state $h_t = (s_0, a_0, \ldots, s_t, a_t)$8, and is then evaluated against the recorded browser next state $h_t = (s_0, a_0, \ldots, s_t, a_t)$9 (Xiao et al., 16 Feb 2026).

Each step receives two judgments. The first is the factuality judgment, which assesses whether $\hat{s}_{t+1}$0 exhibits the main causal effect of action $\hat{s}_{t+1}$1 given the history. The second is the Turing judgment, which compares $\hat{s}_{t+1}$2 and $\hat{s}_{t+1}$3 pairwise and returns a binary preference for the more believable state. Per-step outputs are then aggregated to trajectory-level, dimension-level, and overall scores according to the formulas above (Xiao et al., 16 Feb 2026).

The primary judge is GPT-4o. The paper also reports a cross-judge consistency study using Claude-Opus-4.1 and states that the ranking of models is consistent across judges, although absolute values shift. There is no human-in-the-loop review in the main protocol; success criteria are embedded in the judge prompts and scoring rubrics. The prompts for both metrics are provided verbatim in the appendix, and API model versions for judges are listed, but random seeds, hardware, and explicit evaluation config files are not specified (Xiao et al., 16 Feb 2026).

A concrete example in the paper illustrates the scoring procedure with a checkbox interaction on an account settings page. If action $\hat{s}_{t+1}$4 is click('b42') on an unchecked “Subscribe to newsletter” checkbox and the predicted next state correctly changes checked=false to checked=true, then the per-step factuality score is $\hat{s}_{t+1}$5 because the main causal effect is present and unambiguous. If the model changes the wrong control but still indicates a checkbox change, the rubric may yield $\hat{s}_{t+1}$6. The corresponding Turing judgment is then obtained by pairwise comparison of the predicted and real next states (Xiao et al., 16 Feb 2026).

5. Empirical results and analytical findings

The main comparison includes proprietary LLMs, open-source LLMs, and WebWorld model variants. The paper reports the following overall averages across all nine dimensions:

Model	Fact	Tur
WebWorld-32B	71.0	45.6
WebWorld-14B	70.7	44.7
WebWorld-8B	70.1	42.2
Gemini-3-Pro	70.3	43.2
Claude-Opus-4.1	71.3	47.4
GPT-4o	59.5	35.4
Qwen3-14B	38.0	22.7
Qwen3-32B	40.1	23.0

The paper states that WebWorld achieves average factuality comparable to Claude-Opus-4.1 and Gemini-3-Pro, with consistently strong performance across dimensions, including long-horizon and multi-format settings. This is the central empirical claim of WebWorld-Bench: open-weight web simulators can approach leading proprietary LLMs on intrinsic simulation fidelity (Xiao et al., 16 Feb 2026).

The long-horizon and multi-format results refine that picture. On Long-Horizon Consistency, WebWorld-32B obtains $\hat{s}_{t+1}$7 factuality and $\hat{s}_{t+1}$8 Turing, WebWorld-8B obtains $\hat{s}_{t+1}$9 and $s_{t+1}$0, Gemini-3-Pro obtains $s_{t+1}$1 and $s_{t+1}$2, and Claude-Opus-4.1 obtains $s_{t+1}$3 and $s_{t+1}$4. On multi-format robustness, WebWorld-32B scores XML $s_{t+1}$5, HTML $s_{t+1}$6, Markdown $s_{t+1}$7, and Playwright $s_{t+1}$8; Gemini-3-Pro scores XML $s_{t+1}$9, HTML $[0,1]$0, Markdown $[0,1]$1, and Playwright $[0,1]$2 (Xiao et al., 16 Feb 2026).

The benchmark also supports methodological analyses. Judge consistency is reported as robust in terms of ranking, even though absolute values change. For example, Gemini-3-Pro is reported at $[0,1]$3 under GPT-4o judging and $[0,1]$4 under Claude-Opus-4.1 judging. The paper further notes that low baseline scores often reflect output format misalignment rather than model capacity, indicating that schema compatibility is a substantial confound in intrinsic web-simulation evaluation (Xiao et al., 16 Feb 2026).

Ablation results connect benchmark scores to training choices. Adding a small amount of chain-of-thought data—$[0,1]$5k samples—after large-scale transition training maximizes Total Score at $[0,1]$6, which is higher than directly reasoning-tuning Qwen3-8B with $[0,1]$7 more chain-of-thought data, reported at $[0,1]$8. The paper also states that excessive chain-of-thought can degrade performance (Xiao et al., 16 Feb 2026).

6. Relation to downstream use, limitations, and reproducibility

WebWorld-Bench is intrinsic by design, but the paper explicitly compares intrinsic findings with extrinsic outcomes. Using WebWorld-synthesized trajectories improves downstream agents on MiniWob++ by $[0,1]$9 for Qwen3-8B and on WebArena by $30+$0. In addition, WebWorld used as a world model supports lookahead search on MiniWob, with improvements over GPT-5 as a simulator under comparable settings; the paper attributes gains to pairwise scoring and better value models, reporting $30+$1 with a GPT-5 value model and $30+$2 with GPT-4o. These results do not make WebWorld-Bench itself an extrinsic benchmark, but they qualitatively align higher intrinsic simulator quality with practical utility (Xiao et al., 16 Feb 2026).

The benchmark also has explicit limitations. The exact size of each evaluation dimension is not disclosed. Because the held-out evaluation domains align with the training pipeline sources, potential domain bias remains. Absolute scores vary across judge models, even though rankings are stable, so judge variance is a real source of metric brittleness. The benchmark is text-centric, operating over accessibility trees and converted textual representations such as HTML, XML, Markdown, and Playwright extraction; it therefore does not evaluate pixel-level or vision-grounded fidelity. The authors also note practical brittleness to output schema mismatch, since baseline models trained in incompatible formats underperform (Xiao et al., 16 Feb 2026).

Reproducibility is partial rather than fully packaged. The paper provides full judge prompts, action-space definitions, format-conversion details, and judge model versions, and it outlines how to run the protocol locally: assemble held-out sets per dimension, predict $30+$3 from history, call the Factuality and Turing judges, and aggregate scores according to the benchmark formulas. GPT-4o is the default judge, with Claude-Opus-4.1 recommended for rank-robustness checks. However, the paper does not provide a public evaluation server, a leaderboard, a public URL or repository for WebWorld-Bench artifacts, or seed and hardware specifications for evaluation (Xiao et al., 16 Feb 2026).

Within the broader evaluation landscape, the benchmark is positioned against WebArena, AgentBench, prior intrinsic text-similarity metrics, and simulators trained in closed environments. WebArena is described as a live environment for extrinsic evaluation based on task success rate, while AgentBench and similar suites evaluate overall agent competence. WebWorld-Bench instead isolates web world model quality by scoring state transitions with judge LLMs, and it extends intrinsic evaluation to open-web dynamics, long-horizon contexts exceeding 10 steps, and multi-format robustness (Xiao et al., 16 Feb 2026).