Benchmarking Multimodal LLMs on Code Generation for Complex Interactive Webpages

Abstract: Recent advancements in multimodal LLMs (MLLMs) have achieved remarkable progress in multimodal reasoning and code generation, catalyzing a new paradigm for front-end development. In particular, these models can directly transform visual designs into executable code, significantly improving the efficiency and adaptability of web development. Modern web applications are dynamic and interactive, featuring frequent user-page interactions. However, existing benchmarks largely evaluate the code generation of static webpages, ignoring the complex interactive behaviors in real-world applications. Besides, their evaluation criteria remain confined to visual fidelity and code structure, overlooking the interaction consistency between the generated and the reference webpages. To address these limitations, we introduce WebIGBench, the first benchmark designed to evaluate code generation for interactive webpages with complex interactions. By combining manually designed interaction paths with UI automation, we collected 103 complex webpages from real-world websites. This benchmark covers 5 popular interactive action types (e.g., click, input) involving 871 distinct interactive actions. Moreover, we propose a novel evaluation pipeline to address the gap in automated assessment of interactive actions. Extensive experiments on several representative MLLMs reveal the performance boundaries of current models in interactive webpage code generation using WebIGBench. The proposed benchmark is available at https://github.com/anoa12159-hue/WebIGBench_eval.

• The paper introduces WebIGBench, featuring 103 real-world webpages and 871 annotated interactions to evaluate dynamic, interactive code generation.
• It proposes a two-stage automated evaluation combining visual and behavioral assessments, reporting metrics like Success Rate and Action Completion Rate.
• The study finds that explicit action prompts boost functional metrics, yet current models still struggle with long interaction sequences and state management.

Benchmarking Multimodal LLMs for Interactive Webpage Code Generation: An Expert Summary

Introduction and Motivation

The increasing sophistication of multimodal LLMs (MLLMs) has driven rapid advancements in code generation from visual and textual inputs, catalyzing progress in automated front-end web development. Despite advancements in static web code generation, the automatic synthesis of executable, interactive web applications—mirroring the dynamic behaviors and complex state transitions of modern websites—remains a substantial challenge. Current benchmarks either lack coverage of complex interactivity or provide no systematic automated evaluation of functional fidelity, limiting their utility in assessing the functional depth of MLLMs.

WebIGBench: Dataset Construction and Problem Formulation

To address these gaps, the authors introduce WebIGBench, the first benchmark explicitly designed to evaluate interactive webpage code generation, targeting continuous, complex user-page interaction chains. WebIGBench steps beyond prior resources by curating 103 diverse and interaction-rich webpages from real-world domains, annotated with 871 atomic interaction actions spanning five types (click, input, scroll, key press, switch-tab). Each task involves multi-step, human-authored interaction paths designed to reflect goal-oriented behavior rather than isolated events.

The benchmark’s core representation models the interactive webpage as a sequence of visual states $L_{state}$ and corresponding user actions $L_{action}$—with state transitions induced by the action sequence—captured via aligned screenshots $L_{image}$. The code generation objective is formalized as mapping the entire visual trajectory, accompanied optionally by action annotations and natural language prompts, to an executable code artifact that faithfully reconstructs both appearance and interactivity: $$C_{dyn} = \mathrm{MLLM}(L_{image}, L_{action}, P_{dyn})$$

Figure 1: A typical example of an interactive webpage where user actions induce visual state transitions.

The dataset construction pipeline involves: (1) webpage selection for diverse, complex interactivity; (2) human-in-the-loop task design for realistic multi-step scenarios; (3) automated UI-agent-based execution to collect structured behavioral logs and synchronized visual captures; and (4) action sequence parsing and alignment for high-fidelity annotation.

Figure 2: Details of the WebIGBench benchmark construction pipeline, involving Webpage Selection, Task Design, UI-Agent Interaction Execution, and Interaction Extraction.

Figure 3: Distributions of key features in WebIGBench.

Compared to previous benchmarks (e.g., Design2Code, Web2Code, Interaction2Code), WebIGBench demonstrates markedly higher action density (~8.5 actions/page vs. ~2.9), substantially richer interaction continuity, and domain diversity across 15 real-world categories.

Automated Evaluation Pipeline

Traditional approaches assess generated webpages via visual or code-level similarity, failing to capture functional consistency—whether the code exhibits the correct sequence of state transitions under user interactions. WebIGBench introduces a novel two-stage automated evaluation pipeline that couples visual and behavioral assessment. The process includes:

• Serving the Generated Webpage via an HTTP Server: Ensures equivalence in agent anchoring between reference and generated pages.
• UI-Agent Execution: Systematically executes the user interaction trajectory on the generated page, extracting both the action sequence and post-step screenshots.
• Action Semantic Matching: Employs deterministic, LLM-based semantic alignment to pair the reference and generated action trajectories, allowing for non-strict but semantically correct matching in presence of order or length variation. Agreement between the matcher and human raters is high (Cohen's $\kappa=0.853$).
• Metrics Computation: Three main families of metrics are proposed:
  • Functional: Success Rate (SR), Action Completion Rate (ACR), Average Step Try Cost (ASTC).
  • Visual: CLIP similarity, SSIM, OCR-BLEU between aligned screenshots.
  • Efficiency: Token and wall-clock time costs.
  • 

  • Figure 4: Illustration of the WebIGBench evaluation pipeline.

Experimental Evaluation

Six MLLMs are rigorously evaluated: three closed-source (GPT-5, Gemini-2.5-Pro, Grok-4) and three open-weight models (InternVL3.5-241B, Qwen-VL-Max, GLM-4.5V), covering the current spectrum of leading architectures. Prompts are tested under two regimes: “Direct” (text-only instructions) and “Action” (instructions augmented with explicit action lists).

Key empirical findings:

• Explicit action guidance consistently improves functional metrics across all models. For example, Gemini-2.5-Pro's SR rises from 0.524 to 0.699, and ACR from 0.754 to 0.853, when using enriched “Action” prompting.
• Visual fidelity remains decoupled from functional correctness. The best models reach high CLIP similarity (>0.79), but Success Rates often lag far behind (best case 0.699). This exposes the inability of image-only benchmarks to reveal the failures of interaction logic, specifically the "looks-right-but-does-not-work" phenomenon.
• Step Limit Exceeded and Functional Deviation dominate error cases. Anticipated error types such as No Output or Execution Failure are rare; the main bottleneck is miswiring of DOM-event handlers or incomplete state management.

  Figure 5: Percentage of four error types across different models and prompts.

Ablative and Error Analyses

Further dissection reveals that augmenting the prompt with both visual and action cues does not necessarily outperform structured action prompts alone—visual emphasis can introduce ambiguity. Error breakdowns by action types indicate that while click actions dominate failures (due to prevalence), a non-negligible fraction stems from input, scroll, or key-press, which are poorly modeled by current MLLMs.

Figure 6: Percentage of four error types across input modalities using GPT-5.

Analysis of failure cases illustrates two primary pitfalls:

• Functional Deviation: Generated code renders and interacts, but deviates in logic from the reference (e.g., incorrect order, improper state update).
• Step Limit Exceeded: Agent enters infinite loops or dead ends due to missing or erroneous event wiring.

  Figure 7: Failure case 1 for Functional Deviation.

  

  Figure 8: Failure case 1 for Step Limit Exceeded.

The impact of interaction step length on performance is also measured: longer action chains exacerbate performance degradation under direct prompts but can be partially compensated via explicit action cues, contingent on model capability.

Figure 9: The impact of interaction step length on interactive webpage generation.

Implications, Limitations, and Future Directions

Functionally accurate code generation for interactive webpages remains unresolved by current MLLMs, even with explicit behavioral supervision and strong multimodal perception. The clear separation between visual similarity and functional correctness observed on WebIGBench necessitates a refocusing of benchmark design and model training objectives towards behavioral fidelity and dynamic logic synthesis, rather than static mimicry.

Practical implications:

• For researchers measuring progress in web code generation, reliance on visual/structural metrics alone is fundamentally inadequate for capturing true system utility.
• Automated, agent-based evaluation and trajectory semantic alignment are essential for fine-grained functional diagnosis.

Theoretical and methodological limitations:

• WebIGBench scopes to single-file, front-end only code, omitting back-end, SPA frameworks, or complex multi-page navigation. Dataset size, while high in interaction density, remains limited by manual annotation.
• The evaluation pipeline’s reliance on a specific UI-agent presents potential for error if coverage is incomplete or if browser variance is high; however, high agreement with human annotators partly mitigates this concern.

Potential future developments:

• Extension to multi-file, component-based, or full-stack interactive generation, integrating back-end state and server-client logic.
• Benchmarks modeling richer equivalence classes in user interaction sequences (i.e., DAG-based action alignment).
• Model architectures explicitly tuned for DOM event semantics and long-horizon action planning.

Conclusion

WebIGBench sets a new standard for evaluating interactive web code generation by MLLMs, highlighting strongly that visual fidelity does not guarantee functional correctness. The benchmark and its accompanying evaluation pipeline expose fundamental model limitations and enable targeted, automated diagnosis of logic errors—the primary barrier to real-world web automation. Addressing these methodological gaps is now critical for further advances in trustworthy, automation-capable multimodal foundation models.