WebMMU: Multilingual Multimodal Web Benchmark

• WebMMU is a benchmark for multimodal website understanding and code generation, integrating tasks like visual QA, code editing, and mockup-to-code into one unified framework.
• It is constructed from 2,059 webpage screenshots across 20 domains and four languages, using expert annotations to provide precise element grounding and functional code edits.
• The benchmark evaluates complex multi-step reasoning, UI element localization, and hierarchical design-to-code synthesis, highlighting the current limits of multimodal large language models.

Searching arXiv for the WebMMU paper and directly related work mentioned in the provided data. WebMMU is a multilingual benchmark for multimodal website understanding and code generation that evaluates three core web tasks: website visual question answering, code editing involving HTML/CSS/JavaScript, and mockup-to-code generation (Awal et al., 22 Aug 2025). It unifies tasks that prior benchmarks treat separately, using expert-annotated, real-world web data to assess complex multi-step reasoning, precise element grounding, and functional UI comprehension and coding. The benchmark is explicitly designed to expose the current limits of multimodal LLMs (MLLMs), particularly in reasoning and grounding, editing code while preserving functionality, and generating design-to-code outputs that maintain hierarchy and support multilingual content (Awal et al., 22 Aug 2025).

1. Scope and dataset construction

WebMMU is built from 2,059 unique webpage screenshots drawn from 20 real-world domains, including e-commerce, booking, news, finance, etc., and covers four languages: English (En), Spanish (Es), German (De), French (Fr) (Awal et al., 22 Aug 2025). Each example contains three modalities: a visual representation, code assets, and task-specific annotations. The visual component is a full-page screenshot or collage capturing multiple scroll depths/interactions. The code component retains raw HTML/CSS/JavaScript assets. The annotation component spans three task families: WebQA with 6,102 samples, Mockup2Code with 436 hand-drawn and high-fidelity mockups, and Web Code Editing with 1,602 paired (source, edit-instruction, target) triples, including diff-style annotations (Awal et al., 22 Aug 2025).

The collection pipeline begins from URLs selected from the FineWeb crawl, while respecting robots.txt and preserving coverage across the 20 domains (Awal et al., 22 Aug 2025). Screenshots are captured at multiple scroll depths and interaction states and then collaged into single images. Matching HTML/CSS/JS assets are retained for grounding. Annotation quality control is unusually explicit: 127 professional annotators participate in three QA stages (trainer review, primary QA1, secondary QA2) to ensure semantic correctness, completeness, and consistency across languages (Awal et al., 22 Aug 2025).

This construction suggests that WebMMU is intended not merely as a perception benchmark but as a tightly grounded testbed spanning rendered appearance, executable web artifacts, and multilingual instruction following. A plausible implication is that the benchmark targets the gap between surface-level screenshot understanding and operational web competence.

2. Formal task definitions

WebMMU defines three tasks using explicit input–output mappings. Let $I$ denote an input webpage screenshot or mockup, $q$ a natural-language question, $C^{orig}$ the original code snippet, $e$ an edit instruction, and $C$ the code output (Awal et al., 22 Aug 2025).

For Website Visual Question Answering (WebQA), the objective is $f_{VQA}(I,q)=a$, formally defined as

$f_{\mathrm{VQA}: \{I\}\times\{q\}\;\longrightarrow\;\{a\},$

where $a$ may be free-form text, a multiple-choice label, or coordinate tuples $(x_{\min},y_{\min},x_{\max},y_{\max})$ when spatial grounding is required (Awal et al., 22 Aug 2025). The input is an image $I$ and a question $q$0 in En/Es/De/Fr. The output is an answer string with an optional bounding box. For “Agentic Action” questions, the model must deliver exact UI coordinates for the interactive element (Awal et al., 22 Aug 2025).

For Code Editing, the objective is $q$1, formally defined as

$q$2

with $q$3 represented as a concise git-diff using “+” for additions and “–” for deletions, rather than the full file (Awal et al., 22 Aug 2025). The input is source HTML/CSS/JS + NL instruction $q$4. The output is a patch diff. The grounding requirement is stringent: edits must preserve original functionality and modify only the specified segments (Awal et al., 22 Aug 2025).

For Mockup-to-Code Generation, the objective is $q$5, formally defined as

$q$6

where $q$7 is a wireframe or mockup image and $q$8 is a self-contained HTML file with internal CSS replicating the design (Awal et al., 22 Aug 2025). The input is an image of a sketch or high-fidelity mockup. The output is a single HTML file with embedded <style> for CSS, using placehold.co for placeholder images. The grounding requirement is exacting: element hierarchy, spacing, and alignment must match the sketch, with no extra or missing elements (Awal et al., 22 Aug 2025).

Taken together, these definitions make WebMMU a benchmark for multimodal grounding at three levels: semantic interpretation of rendered content, localized modification of implementation artifacts, and synthesis of implementation from visual specification.

3. Evaluation protocol and metrics

WebMMU combines standard automatic metrics with an “LLM-as-Judge” approach using GPT4o-1120 for tasks that require subjective judgment (Awal et al., 22 Aug 2025). For WebQA, the primary metric is

$q$9

The benchmark also uses Exact Match (EM) for text answers,

$C^{orig}$0

and F1 over tokens for partial matches,

$C^{orig}$1

These are applied where applicable (Awal et al., 22 Aug 2025).

For code editing, WebMMU reports BLEU and TreeBLEU. BLEU is described as the standard n-gram precision score, with brevity penalty. TreeBLEU (Guo et al. 2024) measures matched subtrees in parse trees of predicted vs. reference diffs (Awal et al., 22 Aug 2025). The benchmark also adds an LLM-as-Judge score for code editing: a 1–5 functional correctness rating based on behavior, not syntax (Awal et al., 22 Aug 2025). This is important because code diffs can be textually dissimilar yet behaviorally valid, or textually close yet functionally incorrect.

For Mockup2Code, the LLM-as-Judge protocol produces three 1–5 scores: Layout Structure, Spacing/Proportions, and Alignment/Grid Consistency (Awal et al., 22 Aug 2025). These are aggregated into the composite score

$C^{orig}$2

The benchmark reports no composite metrics beyond the Mockup2Code average; instead, each metric is reported separately to isolate perceptual, structural, and functional dimensions (Awal et al., 22 Aug 2025).

This metric design suggests a deliberate attempt to separate lexical similarity, structural similarity, perceptual fidelity, and behavioral correctness rather than collapsing them into a single scalar. A plausible implication is that WebMMU is structured to diagnose failure modes rather than only to rank models.

4. Baseline models and empirical results

WebMMU evaluates 24 vision-LLMs: 4 closed-source systems—Claude-3.5-Sonnet, Gemini-2.0-Flash, OpenAI-o1, GPT4o-1120—and 20 open-source systems with sizes ranging from 4 B–72 B (Awal et al., 22 Aug 2025). The reported performance establishes a differentiated profile across subtasks, languages, and model families.

For WebQA, performance is broken down by question type and language. In Multi-step Reasoning, OpenAI-o1 reaches 68.2/55.5/46.4/62.4 for En/Es/De/Fr, while GEMINI-2.0 reaches 44.3/41.4/29.8/51.1, and the best open model, Qwen2.5VL-72B, reaches 43.4/36.8/31.2/44.9 (Awal et al., 22 Aug 2025). Smaller open (<8 B) models are typically <20%. In Agentic Action, the top closed model, OpenAI-o1, obtains 4.9/12.3/14.9/15.3, while others ≲ 5%, many near 0% (Awal et al., 22 Aug 2025). In General Comprehension, OpenAI-o1 records 72.7/69.6/49.6/60.9, and Qwen2.5VL-72B records 52.8/47.9/44.5/49.5 (Awal et al., 22 Aug 2025).

For Mockup2Code, the benchmark distinguishes simple layouts from complex layouts. On simple layouts, OpenAI-o1 scores 4.35, Claude-3.5 scores 3.81, InternVL2.5-38B scores 3.85, and Qwen2VL-72B scores 3.26 (Awal et al., 22 Aug 2025). On complex layouts, OpenAI-o1 scores 3.40, Claude-3.5 scores 3.64, InternVL2.5-38B scores 2.98, and Qwen2VL-72B scores 3.05 (Awal et al., 22 Aug 2025).

For Web Code Editing, the benchmark reports BLEU, TreeBLEU, and Judge scores for English:

Model	BLEU(En)	TreeBLEU(En)
Gemini-2.0-Flash	17.02	28.54
Claude-3.5-Sonnet	15.14	24.40
OpenAI-o1	13.95	22.94
InternVL2.5-38B	13.71	26.86
Qwen2VL-7B	7.89	22.10
Phi-3.5-VI (4B)	0.01	0.00

The corresponding Judge scores are 3.26 for Gemini-2.0-Flash, 4.62 for Claude-3.5-Sonnet, 4.26 for OpenAI-o1, 3.85 for InternVL2.5-38B, 2.20 for Qwen2VL-7B, and 1.01 for Phi-3.5-VI (4B) (Awal et al., 22 Aug 2025).

These results indicate that strong textual overlap metrics do not necessarily align with the strongest functional scores. That divergence is explicit in the benchmark design and reinforces the distinction between syntactic patch similarity and operational correctness.

5. Failure modes and benchmarked difficulty

The benchmark identifies several recurring limitations in current MLLMs. General Visual Comprehension is relatively easy, with some models > 70% accuracy (Awal et al., 22 Aug 2025). By contrast, Multi-step numerical/logical reasoning remains hard, with most models < 50% and some as low as 2% (Awal et al., 22 Aug 2025). The most difficult WebQA subtask is Agentic Action (precise UI grounding), which remains < 15% even for best models (Awal et al., 22 Aug 2025).

For Mockup2Code, closed-source LLMs handle simple sketches well but fail on deep nesting and spacing, while open models need ≥ 30 B parameters to approach mid-range performance (Awal et al., 22 Aug 2025). For Web Code Editing, proprietary models slightly outperform large open models but still struggle to output usable patch files without manual correction (Awal et al., 22 Aug 2025). Across the benchmark, multilingual drops are observed across all tasks, and performance on Spanish/French/German is often 10–20 points below English (Awal et al., 22 Aug 2025).

These findings matter because the benchmark is not limited to coarse semantic recognition. It requires alignment between visual evidence, UI structure, code locality, and multilingual instruction following. This suggests that current systems are stronger at local information extraction than at maintaining a consistent internal representation across perceptual, structural, and behavioral constraints.

6. Research significance and projected extensions

WebMMU is presented as a benchmark that provides a comprehensive stress-test for future multimodal web agents, precisely because it unifies visual understanding, design-to-code, and code editing in a multilingual setting (Awal et al., 22 Aug 2025). The paper’s future directions are correspondingly targeted. It proposes to improve spatial grounding and bounding-box accuracy for interactive element localization, enhance numerical and logical module integration for robust multi-step reasoning, and develop UI-aware architectures (e.g., graph-structured DOM encoding) to better capture hierarchical layouts (Awal et al., 22 Aug 2025).

Additional directions include extending the benchmark to interactive/multi-turn web navigation (dynamic agents), broadening linguistic coverage to low-resource languages and regional web designs, incorporating modern front-end frameworks (React, Vue, Angular) into code editing and generation tasks, and refining automatic metrics or devising hybrid human–LLM evaluation for subjective dimensions (Awal et al., 22 Aug 2025).

In this framing, WebMMU serves both as an evaluation instrument and as a decomposition of the research agenda for autonomous web systems. A plausible implication is that progress on the benchmark would require tighter coupling between screenshot-level vision, DOM-aware structure modeling, multilingual reasoning, and behavior-sensitive code generation rather than isolated improvements in any one of those components alone.