---
title: 'WebMMU: Multilingual Multimodal Web Benchmark'
url: https://www.emergentmind.com/topics/webmmu
type: topic
---

# WebMMU: Multilingual Multimodal Web Benchmark

Searching arXiv for the WebMMU paper and directly related work mentioned in the provided data.
WebMMU is a multilingual benchmark for multimodal website understanding and code generation that evaluates three core web tasks: website visual question answering, code editing involving HTML/CSS/JavaScript, and mockup-to-code generation [2508.16763]. It unifies tasks that prior benchmarks treat separately, using expert-annotated, real-world web data to assess complex multi-step reasoning, precise element grounding, and functional UI comprehension and coding. The benchmark is explicitly designed to expose the current limits of multimodal large language models (MLLMs), particularly in reasoning and grounding, editing code while preserving functionality, and generating design-to-code outputs that maintain hierarchy and support multilingual content [2508.16763].

## 1. Scope and dataset construction

WebMMU is built from **2,059 unique webpage screenshots** drawn from **20 real-world domains**, including **e-commerce, booking, news, finance, etc.**, and covers **four languages: English (En), Spanish (Es), German (De), French (Fr)** [2508.16763]. Each example contains **three modalities**: a visual representation, code assets, and task-specific annotations. The visual component is a **full-page screenshot or collage capturing multiple scroll depths/interactions**. The code component retains **raw HTML/CSS/JavaScript assets**. The annotation component spans three task families: **WebQA** with **6,102 samples**, **Mockup2Code** with **436 hand-drawn and high-fidelity mockups**, and **Web Code Editing** with **1,602 paired (source, edit-instruction, target) triples**, including **diff-style annotations** [2508.16763].

The collection pipeline begins from **URLs selected from the FineWeb crawl**, while **respecting robots.txt** and preserving coverage across the 20 domains [2508.16763]. Screenshots are captured at **multiple scroll depths and interaction states** and then **collaged into single images**. Matching **HTML/CSS/JS assets** are retained for grounding. Annotation quality control is unusually explicit: **127 professional annotators** participate in **three QA stages (trainer review, primary QA1, secondary QA2)** to ensure **semantic correctness, completeness, and consistency across languages** [2508.16763].

This construction suggests that WebMMU is intended not merely as a perception benchmark but as a tightly grounded testbed spanning rendered appearance, executable web artifacts, and multilingual instruction following. A plausible implication is that the benchmark targets the gap between surface-level screenshot understanding and operational web competence.

## 2. Formal task definitions

WebMMU defines three tasks using explicit input–output mappings. Let $I$ denote an input webpage screenshot or mockup, $q$ a natural-language question, $C^{orig}$ the original code snippet, $e$ an edit instruction, and $C$ the code output [2508.16763].

For **Website Visual Question Answering (WebQA)**, the objective is $f_{VQA}(I,q)=a$, formally defined as
$$
f_{\mathrm{VQA}: \{I\}\times\{q\}\;\longrightarrow\;\{a\},
$$
where $a$ may be **free-form text**, a **multiple-choice label**, or **coordinate tuples $(x_{\min},y_{\min},x_{\max},y_{\max})$** when spatial grounding is required [2508.16763]. The input is an image $I$ and a question $q$ in **En/Es/De/Fr**. The output is an answer string with an optional bounding box. For **“Agentic Action”** questions, the model must deliver **exact UI coordinates for the interactive element** [2508.16763].

For **Code Editing**, the objective is $f_{edit}(C^{orig},e)=C^{new}$, formally defined as
$$
f_{\mathrm{edit}: \{C_{\mathrm{orig}}\}\times\{e\}\;\longrightarrow\;\{C_{\mathrm{new}}\},
$$
with $C^{new}$ represented as a **concise git-diff** using **“+”** for additions and **“–”** for deletions, rather than the full file [2508.16763]. The input is **source HTML/CSS/JS + NL instruction $e$**. The output is a **patch diff**. The grounding requirement is stringent: edits must **preserve original functionality** and **modify only the specified segments** [2508.16763].

For **Mockup-to-Code Generation**, the objective is $f_{gen}(M)=C$, formally defined as
$$
f_{\mathrm{gen}:\{M\}\;\longrightarrow\;\{C\},
$$
where $M$ is a **wireframe or mockup image** and $C$ is a **self-contained HTML file with internal CSS replicating the design** [2508.16763]. The input is an image of a **sketch or high-fidelity mockup**. The output is a **single HTML file with embedded `<style>` for CSS**, using **placehold.co** for placeholder images. The grounding requirement is exacting: **element hierarchy, spacing, and alignment must match the sketch**, with **no extra or missing elements** [2508.16763].

Taken together, these definitions make WebMMU a benchmark for multimodal grounding at three levels: semantic interpretation of rendered content, localized modification of implementation artifacts, and synthesis of implementation from visual specification.

## 3. Evaluation protocol and metrics

WebMMU combines standard automatic metrics with an **“LLM-as-Judge”** approach using **GPT4o-1120** for tasks that require subjective judgment [2508.16763]. For WebQA, the primary metric is
$$
\mathrm{Accuracy}_{\mathrm{VQA}} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}\bigl[\hat a_i = a_i\bigr].
$$
The benchmark also uses **Exact Match (EM)** for text answers,
$$
\mathrm{EM}=\frac1N\sum_i\mathbf1[\hat a_i\equiv a_i],
$$
and **F1 over tokens** for partial matches,
$$
\mathrm{F1}=2\cdot\frac{\mathrm{Precision}\times\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}.
$$
These are applied **where applicable** [2508.16763].

For code editing, WebMMU reports **BLEU** and **TreeBLEU**. BLEU is described as the **standard n-gram precision score, with brevity penalty**. **TreeBLEU (Guo et al. 2024)** measures **matched subtrees in parse trees of predicted vs. reference diffs** [2508.16763]. The benchmark also adds an **LLM-as-Judge** score for code editing: a **1–5 functional correctness rating based on behavior, not syntax** [2508.16763]. This is important because code diffs can be textually dissimilar yet behaviorally valid, or textually close yet functionally incorrect.

For Mockup2Code, the **LLM-as-Judge** protocol produces three **1–5 scores**: **Layout Structure**, **Spacing/Proportions**, and **Alignment/Grid Consistency** [2508.16763]. These are aggregated into the composite score
$$
S_{\mathrm{mockup}} = \frac{S_{\mathrm{layout}} + S_{\mathrm{spacing}} + S_{\mathrm{alignment}}}{3}.
$$
The benchmark reports **no composite metrics beyond the Mockup2Code average**; instead, **each metric is reported separately to isolate perceptual, structural, and functional dimensions** [2508.16763].

This metric design suggests a deliberate attempt to separate lexical similarity, structural similarity, perceptual fidelity, and behavioral correctness rather than collapsing them into a single scalar. A plausible implication is that WebMMU is structured to diagnose failure modes rather than only to rank models.

## 4. Baseline models and empirical results

WebMMU evaluates **24 vision-language models**: **4 closed-source** systems—**Claude-3.5-Sonnet, Gemini-2.0-Flash, OpenAI-o1, GPT4o-1120**—and **20 open-source** systems with sizes ranging from **4 B–72 B** [2508.16763]. The reported performance establishes a differentiated profile across subtasks, languages, and model families.

For **WebQA**, performance is broken down by question type and language. In **Multi-step Reasoning**, **OpenAI-o1** reaches **68.2/55.5/46.4/62.4** for **En/Es/De/Fr**, while **GEMINI-2.0** reaches **44.3/41.4/29.8/51.1**, and the **best open** model, **Qwen2.5VL-72B**, reaches **43.4/36.8/31.2/44.9** [2508.16763]. **Smaller open (<8 B)** models are **typically <20%**. In **Agentic Action**, the **top closed** model, **OpenAI-o1**, obtains **4.9/12.3/14.9/15.3**, while **others ≲ 5%, many near 0%** [2508.16763]. In **General Comprehension**, **OpenAI-o1** records **72.7/69.6/49.6/60.9**, and **Qwen2.5VL-72B** records **52.8/47.9/44.5/49.5** [2508.16763].

For **Mockup2Code**, the benchmark distinguishes **simple layouts** from **complex layouts**. On **simple layouts**, **OpenAI-o1** scores **4.35**, **Claude-3.5** scores **3.81**, **InternVL2.5-38B** scores **3.85**, and **Qwen2VL-72B** scores **3.26** [2508.16763]. On **complex layouts**, **OpenAI-o1** scores **3.40**, **Claude-3.5** scores **3.64**, **InternVL2.5-38B** scores **2.98**, and **Qwen2VL-72B** scores **3.05** [2508.16763].

For **Web Code Editing**, the benchmark reports BLEU, TreeBLEU, and Judge scores for English:

| Model | BLEU(En) | TreeBLEU(En) |
|---|---:|---:|
| Gemini-2.0-Flash | 17.02 | 28.54 |
| Claude-3.5-Sonnet | 15.14 | 24.40 |
| OpenAI-o1 | 13.95 | 22.94 |
| InternVL2.5-38B | 13.71 | 26.86 |
| Qwen2VL-7B | 7.89 | 22.10 |
| Phi-3.5-VI (4B) | 0.01 | 0.00 |

The corresponding **Judge** scores are **3.26** for **Gemini-2.0-Flash**, **4.62** for **Claude-3.5-Sonnet**, **4.26** for **OpenAI-o1**, **3.85** for **InternVL2.5-38B**, **2.20** for **Qwen2VL-7B**, and **1.01** for **Phi-3.5-VI (4B)** [2508.16763].

These results indicate that strong textual overlap metrics do not necessarily align with the strongest functional scores. That divergence is explicit in the benchmark design and reinforces the distinction between syntactic patch similarity and operational correctness.

## 5. Failure modes and benchmarked difficulty

The benchmark identifies several recurring limitations in current MLLMs. **General Visual Comprehension is relatively easy**, with **some models > 70% accuracy** [2508.16763]. By contrast, **Multi-step numerical/logical reasoning remains hard**, with **most models < 50%** and **some as low as 2%** [2508.16763]. The most difficult WebQA subtask is **Agentic Action (precise UI grounding)**, which remains **< 15% even for best models** [2508.16763].

For **Mockup2Code**, **closed-source LLMs handle simple sketches well but fail on deep nesting and spacing**, while **open models need ≥ 30 B parameters to approach mid-range performance** [2508.16763]. For **Web Code Editing**, **proprietary models slightly outperform large open models but still struggle to output usable patch files without manual correction** [2508.16763]. Across the benchmark, **multilingual drops are observed across all tasks**, and performance on **Spanish/French/German is often 10–20 points below English** [2508.16763].

These findings matter because the benchmark is not limited to coarse semantic recognition. It requires alignment between visual evidence, UI structure, code locality, and multilingual instruction following. This suggests that current systems are stronger at local information extraction than at maintaining a consistent internal representation across perceptual, structural, and behavioral constraints.

## 6. Research significance and projected extensions

WebMMU is presented as a benchmark that **provides a comprehensive stress-test for future multimodal web agents**, precisely because it unifies **visual understanding, design-to-code, and code editing in a multilingual setting** [2508.16763]. The paper’s future directions are correspondingly targeted. It proposes to **improve spatial grounding and bounding-box accuracy for interactive element localization**, **enhance numerical and logical module integration for robust multi-step reasoning**, and **develop UI-aware architectures (e.g., graph-structured DOM encoding) to better capture hierarchical layouts** [2508.16763].

Additional directions include extending the benchmark to **interactive/multi-turn web navigation (dynamic agents)**, broadening linguistic coverage to **low-resource languages** and **regional web designs**, incorporating **modern front-end frameworks (React, Vue, Angular)** into code editing and generation tasks, and refining automatic metrics or devising **hybrid human–LLM evaluation** for subjective dimensions [2508.16763].

In this framing, WebMMU serves both as an evaluation instrument and as a decomposition of the research agenda for autonomous web systems. A plausible implication is that progress on the benchmark would require tighter coupling between screenshot-level vision, DOM-aware structure modeling, multilingual reasoning, and behavior-sensitive code generation rather than isolated improvements in any one of those components alone.

Source: https://www.emergentmind.com/topics/webmmu