Vision2Web: Hierarchical Web Dev Benchmark

• Vision2Web is a hierarchical benchmark for visual website development that evaluates static, interactive, and full-stack tasks.
• It curates 193 real-world tasks across 16 categories with prototype images and multiple test cases to gauge performance.
• The evaluation protocol integrates GUI agent verifiers and VLM judges in a workflow-based system to assess both functional correctness and visual fidelity.

Searching arXiv for the cited Vision2Web paper and directly related web-agent/web-code-generation papers to ground the article. Vision2Web is a hierarchical benchmark for visual website development with agent verification, introduced to evaluate the end-to-end capabilities of modern multimodal coding agents on realistic website-development tasks. It spans static UI-to-code generation, interactive multi-page frontend reproduction, and long-horizon full-stack website development. The benchmark is constructed from real-world websites and comprises 193 tasks across 16 categories, with 918 prototype images and 1,255 test cases. Its evaluation protocol is organized around a workflow-based agent verification paradigm that combines a GUI agent verifier with a VLM-based judge, with the stated goal of providing flexible, thorough, and reliable assessment of both functional behavior and visual fidelity (He et al., 27 Mar 2026).

1. Objectives and problem formulation

Vision2Web was proposed in response to a gap in existing coding-agent evaluation: current benchmarks often focus on localized code edits or text-only tasks and do not cover the full multimodal, long-horizon workflow of real-world website development. Its objectives are threefold: to disentangle capabilities from visual understanding through to full-stack integration, to ground tasks in publicly accessible, real-world websites, and to introduce a reproducible, agent-based verification paradigm combining functional GUI testing with visual fidelity judging (He et al., 27 Mar 2026).

A central feature of the benchmark is its separation of website development into progressively harder strata rather than treating the problem as a single monolithic task. This design encodes a capability ladder: faithful visual reproduction at the static level, correct navigation and interaction across pages at the frontend level, and end-to-end system design with deployment and back-end integration at the full-stack level. This suggests that Vision2Web is intended not merely as a leaderboard dataset, but as an instrument for capability disentanglement and failure localization.

The benchmark also formalizes website development as a multimodal problem in which prototype images, textual specifications, interaction traces, and rendered outputs must be jointly considered. That framing is consequential because it places visual understanding, code synthesis, stateful execution, and deployment reliability within a single evaluation envelope rather than isolating them into separate subfields.

2. Hierarchical task organization

Vision2Web organizes tasks into three levels of increasing complexity. Level 1 targets static webpage generation, Level 2 targets interactive multi-page frontend reproduction, and Level 3 targets long-horizon full-stack development. The same 16 subcategories appear across the benchmark, grouped under four major website types: Content, Transaction, SaaS, and Public Service (He et al., 27 Mar 2026).

Level	Scope	Key statistics
Level 1	Static Webpage Generation	100 tasks; 300 prototype images; visual-only test cases
Level 2	Interactive Multi-Page Frontend	66 tasks; average 5.9 ± 0.5 prototype images per task; average 7.5 ± 4.5 functional-interaction cases per task
Level 3	Long-Horizon Full-Stack Development	27 tasks; average 8.5 ± 2.3 prototype images per task; average 28.2 ± 12.0 test cases per task

At Level 1, each task includes prototype images at three resolutions—desktop, tablet, and mobile—for a total of 300 images. Evaluation is visual-only, using pixel-and-component comparisons. The core skill is faithful reproduction of layout, style, and responsiveness. At Level 2, prototype images average approximately 6 per task, totaling approximately 389 images, and the benchmark introduces functional-interaction cases, averaging 7.5 ± 4.5 per task for a total of approximately 495 cases. The core skill shifts to correct navigation, component consistency, and inter-page interactions. At Level 3, tasks average 8.5 ± 2.3 prototype images and 28.2 ± 12.0 test cases per task, totaling approximately 230 images and approximately 760 cases, with the core skill defined as end-to-end system design, back-end APIs, state management, deployment, and visual fidelity (He et al., 27 Mar 2026).

Dataset-wide statistics reinforce the scale and heterogeneity of the benchmark: 193 total tasks, 918 prototype images, and 1,255 test cases. The average input text length is 1.0 k tokens for Level 2 and 4.3 k tokens for Level 3. A plausible implication is that the progression from Level 1 to Level 3 is not only a progression in interaction complexity but also in specification length, planning burden, and context management.

3. Workflow-based agent verification

Vision2Web evaluates submissions through structured test workflows, each defined as a directed acyclic graph of verification nodes. Two node types are used: Functional Verification Nodes, executed by a GUI agent verifier, and Visual Verification Nodes, evaluated by a VLM-based judge. The two components are orchestrated under a shared application state by Algorithm 1, termed Workflow-Based Agent Verification (He et al., 27 Mar 2026).

For functional verification, each node is formally defined as

$n_i = \langle O_i, A_i, V_i \rangle,$

where $O_i$ is the test objective, $A_i$ is a constrained sequence of allowed actions, and $V_i$ is the validation criterion, implemented as logical assertions or state queries. The verifier’s context at node $i$ is

$$\mathcal{C}_i = \{\mathcal{H}_{<i},\, O_i,\, A_i,\, V_i\},$$

where $\mathcal{H}_{<i}$ records the history of previous node objectives and actions. The GUI agent verifier, instantiated with a multimodal LLM such as GLM-4.6V, executes the action sequence and returns a pass/fail signal $F_i \in \{0,1\}$. The level-wise Functional Score is

$$\mathrm{FS} = \frac{1}{N_{\mathrm{func}}}\sum_{i=1}^{N_{\mathrm{func}}} F_i \times 100.$$

For visual verification, each node is defined as

$n_i = \langle P_i \rangle,$

where $O_i$0 is the target prototype image. The VLM judge segments both the prototype and the rendered page into logical components and assigns each block $O_i$1 a score in $O_i$2 according to strict rubrics. The per-node visual score $O_i$3 is the average over blocks, and the level-wise Visual Score is

$O_i$4

This verification design is significant because it separates executable correctness from appearance matching while retaining a unified workflow abstraction. It also addresses a recurrent ambiguity in visual code-generation benchmarks: high pixel similarity does not imply correct behavior, and correct behavior does not imply faithful reproduction. Vision2Web makes that distinction explicit at the metric and infrastructure levels.

4. Benchmark construction and data curation

The 193 tasks in Vision2Web were curated through a three-stage pipeline on the C4 validation corpus to prevent leakage. The first stage, structural assessment by DOM metrics such as tag distribution, depth, and token count, yielded 63,515 candidates. The second stage, content screening via VLM scoring of visual and functional richness, reduced the set to 7,391 pages. The third stage, manual review by annotators, filtered pages for cross-device consistency, design quality, interaction richness, and category balance. For each retained task, multimodal resource libraries, including images, icons, and fonts, together with unambiguous textual specifications, were assembled (He et al., 27 Mar 2026).

The reliance on real-world websites rather than synthetic constructions is methodologically important. It means the benchmark inherits actual design conventions, cross-device rendering patterns, interaction structures, and content compositions that are difficult to replicate with templated generators. It also creates a closer alignment between benchmark difficulty and deployment difficulty in real software workflows.

The dataset design additionally embeds balance constraints at multiple levels: across the four major website types, across the 16 subcategories, and across device form factors. This indicates that Vision2Web is intended to test generality rather than narrow specialization on a single design regime such as marketing pages or dashboards.

5. Evaluation protocol and empirical findings

Vision2Web evaluates multiple visual LLMs instantiated under two coding-agent frameworks, OpenHands and Claude Code. The reported multimodal backbones are Claude-Opus-4.5, Claude-Sonnet-4.5, GPT-5, Gemini-3-Pro-Preview, Gemini-3-Flash-Preview, Seed-1.8-VL, and Qwen3-VL-32B/8B. The GUI verifier uses GLM-4.6V, and the VLM judge uses Gemini-3-Pro-Preview. Metrics include Static layout fidelity measured by VS only, Interactive and Full-Stack Visual Score, Functional Score, and Deployment Success Rate (He et al., 27 Mar 2026).

Representative results under the OpenHands framework are as follows.

Coding agent	Static Avg VS	Interactive VS / FS	Full-Stack VS / FS
Claude-Opus-4.5	53.4	46.5 / 66.7	38.4 / 57.6
Gemini-3-Pro	55.8	29.7 / 40.7	11.7 / 22.6
GPT-5	49.6	23.9 / 61.4	18.3 / 49.7

Several benchmark-level observations are reported. Performance declines sharply from Level 1 to Level 3 across all agents. Device form factor and prototype complexity further reduce VS by 10–20%. Claude-Opus-4.5 is the strongest model overall, but still reaches only approximately 38 VS and 58 FS on full-stack tasks. Framework choice matters: OpenHands consistently outperforms Claude Code for most models. Category-wise, Public Service sites yield higher VS and FS, while SaaS Platforms remain the hardest. Test-case-level analysis shows low success on state management, CRUD, and media operations (He et al., 27 Mar 2026).

These findings delimit the current capability frontier. The strongest systems can reproduce static designs moderately well, but full-stack execution remains substantially more difficult. The gap between visual reproduction and functional correctness is also nontrivial, as seen in result pairs where FS materially exceeds VS or vice versa. This suggests that website development benchmarks should continue to report both axes rather than collapsing them into a single scalar.

6. Relation to adjacent research, limitations, and future directions

Vision2Web sits at the intersection of at least three neighboring research lines: visual web navigation, design-to-code generation, and vision-first web agents. "Dual-View Visual Contextualization for Web Navigation" introduces Dual-View Contextualized Representation, which pairs HTML elements with screenshot bounding boxes and visual content, contextualizes each element with nearby elements, and reports consistent improvements on Mind2Web, including Recall@1/5/10/50 of 38.4 / 71.6 / 79.7 / 90.1% relative to a baseline of 25.4 / 61.0 / 73.5 / 88.9% (Kil et al., 2024). For Vision2Web, this suggests that integrating rendered layout cues into structured web representations is likely to be important for Level 2 interaction and possibly for verifier design.

For static webpage generation, "WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs" provides a 2.56 million-instance corpus of design images, HTML+CSS, and layout annotations, together with the WebCoder baseline and the TreeBLEU metric for structural hierarchy recall. On the WebCode2M test split, WebCoder reports BLEU 48.2 and TreeBLEU 0.84, compared with 37.5 and 0.72 for Pix2Struct-1.3B and 15.6 and 0.32 for ChatGPT-4V (one-shot) (Gui et al., 2024). This defines a closely related but narrower regime than Vision2Web Level 1, since WebCode2M is centered on code generation from webpage designs rather than hierarchical evaluation spanning interaction and deployment.

At the opposite end of the stack, "WebSight: A Vision-First Architecture for Robust Web Agents" describes a vision-only, modular multi-agent web agent with planning, reasoning, vision-action, and verification components, reporting 58.84% top-1 click accuracy on Showdown/Clicks and 68.0% success on the filtered 50-task WebVoyager benchmark (Bhathal et al., 23 Aug 2025). That work is directly relevant to Vision2Web’s emphasis on visual interaction and verification, although it addresses autonomous web navigation rather than website construction.

Vision2Web’s own limitations are explicitly identified. Annotation and verification still rely on current agents, including Claude Code and the VLM judge, whose errors may introduce bias. Full-stack tasks remain extremely challenging, and agent resource constraints such as time and compute may limit scale. The current VLM judge, while strongly correlated with humans with Spearman $O_i$5, can still disagree on subtle visual details (He et al., 27 Mar 2026). These limitations are not peripheral: they bear directly on benchmark validity, especially when the evaluator is itself an imperfect agent.

The future directions proposed for Vision2Web follow from these constraints. They include improving agent planning and self-verification for long-horizon code tasks through dynamic feedback loops; enhancing VLM judges with learned visual-semantic alignment; extending the benchmark to mobile-only or hybrid-app scenarios and richer back-end integrations such as real-time data; and investigating multi-agent coordination in which specialized sub-agents handle UI, back-end, testing, and deployment in parallel (He et al., 27 Mar 2026). Taken together, these directions indicate that Vision2Web is designed as an extensible benchmark framework for multimodal coding-agent research rather than as a fixed, closed evaluation suite.