WebCoderBench Benchmark

• WebCoderBench is a benchmark framework designed to measure LLM performance in generating web app code based on diverse real-world user requirements.
• It employs 24 detailed metrics across code, visual, content, performance, accessibility, and maintainability to deliver granular diagnostic insights.
• The framework leverages automated evaluation pipelines and human-aligned weighting to enable objective comparisons among open- and closed-source models.

WebCoderBench is a publicly documented benchmark designed to rigorously evaluate the code generation capabilities of LLMs and LLM-based agents in the domain of web application (web app) generation. The framework is structured to address the complexities inherent in benchmarking web apps: capturing real-world user requirements, imposing generalizable evaluation metrics without dependence on ground-truth implementations or test cases, and producing interpretable diagnostic reports. WebCoderBench comprises a dataset of 1,572 authentic user requirements sourced from anonymized industrial logs and provides 24 fine-grained metrics across 9 distinct perspectives, augmented with human-preference–aligned aggregation to yield meaningful overall scores. This benchmark facilitates comprehensive, objective, and interpretable comparisons between models and agents, offering granular insights into system strengths and weaknesses (Liu et al., 5 Jan 2026).

1. Dataset Construction and Curation

The WebCoderBench dataset originates from one-week anonymized logs of an industrial LLM service. To ensure faithful reflection of real-world intentions, multi-turn interactions are automatically merged using Gemini-2.5-pro, with a prompt filtering only “Functionality Addition” turns; expert validation guarantees retention of user intent and prevention of hallucinations. Filtering is conducted by three expert reviewers for each requirement, discarding incoherent requests and entries lacking requisite materials (e.g., referenced images), as well as non-web scenarios. Text-level de-duplication employs MinHash, while semantic similarity elimination uses MiniLM embeddings and cosine similarity.

Each requirement is supported by ground-truth checklists enumerating functional, visual, and content criteria, initially drafted by three LLMs—GPT-5-Chat, Gemini-2.5-pro, Doubao-Seed—and finalized via expert consensus.

Dataset diversity includes:

• Modalities: Text only (1,413), Text + image (123), Text + URL (36)
• Clarity levels: C1 (Clear, 764), C2 (Intermediate, 730), C3 (Vague, 78)
• Expression styles: S1 (Technical, 683), S2 (Colloquial, 724), S3 (Role-playing, 60), S4 (Analogy, 105)
• Artifact complexity levels: L1 (Highly Simple, 179), L2 (Simple, 433), L3 (Medium, 658), L4 (Complex, 259), L5 (Highly Complex, 43)

This indicates substantial coverage of modality, clarity, style, and complexity, enhancing benchmark generalizability.

2. Multi-Faceted Evaluation Framework

WebCoderBench adopts 24 metrics distributed across 9 perspectives, subdivided under two overarching categories: “General Quality” and “Alignment Quality.” Each metric is operationalized as a scalar in the [0,100] interval. Rule-based metrics apply deterministic tooling, whereas LLM-as-judge metrics leverage prompts and model scoring for subjective aspects.

General Quality (21 metrics, 6 perspectives):

• Code Quality (5 metrics): General Functionality Correctness (LLM-judge), Best Practices (Lighthouse), Error Handling, Runtime Console Errors, Static Syntax Checking (htmlhint, eslint, stylelint)
• Visual Quality (6 metrics): General Visual Experience (LLM-judge), Component Style Consistency, Icon Style Consistency, Layout Consistency, Layout Sparsity, Visual Harmony Degree (K-means in HSV space)
• Content Quality (4 metrics): Copywriting Quality, Media Quality (clarity, accessibility), Placeholder Quality, Resource Validity (HTTP status checks)
• Performance Quality (1 metric): General Performance (Lighthouse)
• Accessibility (3 metrics): Accessibility Core Metrics (Lighthouse), Cross-Browser Compatibility (Playwright, MDN BCD), Mobile Device Compatibility (horizontal overflow)
• Maintainability (2 metrics): Code Redundancy Rate (unused JS/CSS), Comment Rate

Alignment Quality (3 metrics):

Metrics employ GPT-5-Chat, referencing artifact similarity to ground-truth checklists derived for each requirement:

• Functional Alignment
• Visual Alignment
• Content Alignment

For each alignment metric: $$\text{score} = \left(\frac{\text{passed\_points}}{\text{total\_points}}\right) \times 100$$

3. Scoring, Aggregation, and Human Alignment

Metric weighting is determined through a survey of 899 participants (filtered to 141 valid), encompassing multifaceted professional roles. A two-stage process ranks the nine evaluation perspectives and subsequently the contained metrics within each perspective; Borda Count is used to aggregate rankings:

$$\text{BordaScore}(m) = \sum_{r \in \text{respondents}} [\#\text{items} - \text{rank}_r(m) + 1]$$

Weights are normalized so that $\sum_m w_m = 1$, with final per-metric weight $$w_i = w_{\text{perspective}(p(i))} \times \text{normalized weight}(i)$$.

Overall model scores are computed by normalizing each metric score via Z-scores over all models and samples:

$z_{i,j} = \frac{x_{i,j} - \mu_j}{\sigma_j}$

where $x_{i,j}$ is model $M$'s raw score for sample $i$ on metric $j$, with per-metric mean $\mu_j$ and standard deviation $\sigma_j$ across all samples ($N=1572$). The final score for model $M$ is:

$$S(M) = \sum_{j=1}^{24} w_j \cdot \left[ \frac{1}{N} \sum_{i=1}^{N} z_{i,j} \right]$$

This scoring protocol integrates human preferences, enforces cross-metric comparability, and enables interpretable fine-grained diagnoses.

4. Benchmark Execution and Experimental Protocol

WebCoderBench supports evaluation of 12 representative LLMs, both open-source (DeepSeek-R1, DeepSeek-V3.1, Qwen3-Coder-Plus, Qwen3-Instruct, StarCoder, MiniMax-M2, GLM-4.5, Gemini-2.5-pro) and closed-source (GPT-4o, GPT-5-Codex-High, GPT-5-High), along with 2 LLM-based agents (Manus, MiniMax Agent). All models are accessed via standard APIs under identical system prompts; agents are controlled in a web UI with enforced restrictions to “native HTML/CSS/JS only.”

Multi-modal requirements are downsampled for uni-modal models, with images converted to alt-text by Gemini-2.5-pro. URLs are treated as references. Each requirement prompts a single artifact generation using:

"You are a professional front-end engineer. Given user requirements, output only native HTML, CSS, JS that fulfills them."

Artifacts undergo automated pipeline postprocessing: headless browser rendering (Selenium), logging, linting (htmlhint, eslint, stylelint), Lighthouse analysis, documentation, and LLM-as-judge metric invocation via standardized prompts. Cases unsuited for specific metrics (e.g., no media present) are excluded from averages.

5. Key Findings and Diagnostic Insights

Experiments reveal no single model dominates across all 24 metrics. GPT-5-High achieves the highest overall score, but best-in-class open-source models outperform it on particular metrics. GLM-4.5 and MiniMax-M2 markedly close the gap between closed- and open-source systems to less than approximately 6%.

Model strengths are distributed: some excel at code quality but underperform in visual design, while others show the opposite trend. Agents exhibit robust alignment on planning tasks but at the expense of performance, visual consistency, and accessibility—attributable to the complexity of external function invocation.

Alignment scores are also sensitive to requirement clarity: vague requirements lead to higher alignment due to increased interpretive latitude, whereas clear requirements more harshly penalize minor divergences. Uni-modal LLMs consistently underperform on requirements incorporating images.

User-persona–specific weighting stratifies areas of interest: designers prioritize visual quality, developers emphasize code quality and maintainability.

6. Significance and Prospects

WebCoderBench is distinguished as the first benchmark for web app code generation built from real user requirements, offering fully automated, objective, and interpretable evaluation metrics. It enables end-to-end assessment of LLMs and agents, provides actionable diagnostics for targeted model optimization, and advances the field’s capacity to rigorously evaluate generative systems in practical, commercial, and research-sensitive web domains (Liu et al., 5 Jan 2026). This suggests ongoing application as a comparative standard and potential refinement as user requirements and model architectures evolve.