WebGen-Bench: Full-Stack Code Evaluation

• WebGen-Bench is a benchmark that tests LLM agents’ ability to autonomously generate fully functional, multi-file web applications from scratch using natural language instructions.
• It features 101 curated tasks across content presentation, user interaction, and data management, with each task evaluated by automated GUI-based test cases.
• The benchmark employs a multi-dimensional scoring protocol that assesses both functional accuracy and visual appearance, highlighting areas for improvement in full-stack synthesis.

WebGen-Bench is a curated benchmark for evaluating the capability of LLM code agents to autonomously generate fully functional, multi-file web applications from natural-language instructions. Distinct from prior datasets that focus on patching or extending existing repositories, WebGen-Bench exclusively tests an agent's proficiency in full-stack, from-scratch website creation, including frontend, backend, and integration logic, under high-level, tool-agnostic user requests. The benchmark encompasses both functional and visual requirements, employing automated, reproducible, GUI-based test suites and appearance evaluations to holistically assess code quality and user experience fidelity (Lu et al., 6 May 2025, Lu et al., 26 Sep 2025).

1. Benchmark Structure and Coverage

WebGen-Bench consists of 101 realistic, hand-curated website-generation tasks, each specified as a natural-language instruction that intentionally avoids explicit references to any frameworks, programming languages, or low-level artifacts. The instructions probe the agent's ability to plan architectural structure, select appropriate technologies, and implement end-to-end logic. Each instruction maps to a small- to medium-complexity web app and is categorized across three major axes:

• Content Presentation (28 tasks): static site generation, dynamic content rendering, data visualization, media display.
• User Interaction (49 tasks): form workflows, authentication, real-time messaging, e-commerce cart systems, simple AI assistant integration.
• Data Management (24 tasks): CRUD dashboards, REST API integration, large-scale logging and statistics endpoints, cloud file handling.

The instruction set was compiled via multi-stage human and LLM synthesis. Computer science PhD students systematically analyzed common tasks on freelance platforms, proposing over 10,000 detailed project descriptions. These were expanded via GPT-4o to yield nuanced, context-rich instructions, then filtered and refined to avoid ambiguity and ensure technical feasibility (Lu et al., 6 May 2025).

2. Test Case Design and Automated Evaluation

Each task in WebGen-Bench is accompanied by a suite of meticulously constructed test cases, totaling 647 across the benchmark. Test cases are atomic, each specifying a single user-level operation (e.g., submitting a form, clicking a menu, triggering an API call) and the precise resulting state to be observed in the UI or backend.

• Test Case Generation: Initial drafts were produced via LLM reasoning over task specifications, then validated and de-duplicated by multiple subject-matter experts to ensure coverage, atomicity, and clarity.
• Test Case Execution: The evaluation harness employs a web-navigation “GUI-agent” that launches the generated codebase in a headless browser (e.g., WebContainer for certain frameworks), performs scripted user interactions, and determines pass/fail status by programmatic inspection of the DOM, visual screenshot, or observable application state (Lu et al., 6 May 2025, Lu et al., 26 Sep 2025).

This pipeline supports reproducible, unbiased measurement of both functional correctness and appearance, critical for fair comparison between modeling approaches.

3. Metrics and Scoring Protocol

WebGen-Bench introduces a multi-dimensional, interpretable evaluation protocol:

• Functional Accuracy: The fundamental metric is accuracy over all test cases, with partial credit for incomplete solutions. For $N_{\mathrm{Total}}$ test cases, $N_{\mathrm{Yes}}$ fully passed, and $N_{\mathrm{Partial}}$ partially passed:

$$\text{Accuracy} = \frac{N_{\mathrm{Yes}} + 0.5\,N_{\mathrm{Partial}}}{N_{\mathrm{Total}}} \times 100\%$$

• Outcome Categories: For each task, results are classified as Yes (all checks passed), Partial (some passed), No (none passed), and Start Failed (server/build errors prevent evaluation).
• Appearance Score: Each successfully-rendered website is graded by a state-of-the-art vision-LLM (VLM; e.g., GPT-4o) on a 0–5 scale, considering rendering quality, layout, content appropriateness, and "modernness." Zero indicates a blank or broken page; a five represents highly polished, contemporary design (Lu et al., 26 Sep 2025).
• Categorical Breakdown: Performance is additionally analyzed by instruction class (presentation, interaction, data) and test-case type (functional, data-display, design-validation), enabling granular diagnosis of model strengths and weaknesses.

These metrics facilitate robust cross-system benchmarking and progress tracking.

4. Baseline Agents and Empirical Findings

Extensive empirical studies were conducted on WebGen-Bench using multiple agent architectures and LLMs:

Model	Yes	Partial	No	StartFailed	Accuracy (%)	Appearance
Bolt.diy + Claude-3.5	22.6	7.6	64.1	5.7	26.4	3.0
Bolt.diy + DeepSeek-V3	18.5	4.5	73.9	3.1	20.8	2.0
WebGen-Agent + Claude-3.5	45.6	12.7	40.6	1.1	51.9	3.9
WebGen-Agent + DeepSeek-V3	46.1	13.1	40.6	0.2	52.6	3.8

The best-performing code-centric LLM-agents (Bolt.diy with DeepSeek-R1) achieve only ~27.8% accuracy, demonstrating the challenge posed by full-stack, from-scratch synthesis. Open-source fine-tuning on decontaminated data (WebGen-Instruct; 6,667 instructions) can raise accuracy to 38.2% (Qwen2.5-Coder-32B-Instruct), while advanced agentic frameworks that integrate multi-level feedback loops and visual reward signals achieve scores above 50% and significantly higher appearance evaluations (Lu et al., 26 Sep 2025).

Category-wise analysis reveals that "Content Presentation" is generally more tractable for current models (~40% accuracy), while complex stateful flows (real-time, deep backend integration) remain bottlenecks at <25% accuracy.

5. Challenges, Analysis of Failure Modes, and Development Directions

WebGen-Bench systematically uncovers persistent limitations in LLM-based code generation for web applications:

• Multi-file Orchestration: Over 50% of outright failures are attributed to improperly initialized project structures (missing files, misconfigured servers, import path errors).
• Complex User Interactions: Tasks requiring robust state management, real-time event propagation, or distributed control (e.g., authentication, live chat, dashboards) expose fragility in current agents' reasoning and memory.
• Design Consistency: Visual fidelity is often compromised in advanced layouts or responsive designs. While models now clear basic color and positioning checks, high-order constraints (e.g., adaptive CSS, accessibility) remain challenging (Lu et al., 6 May 2025, Lu et al., 26 Sep 2025).

Suggested research directions to address these gaps include expanding the benchmark to additional tech stacks (Python, database backends), incorporating RL or direct preference optimization for multi-step assembly, developing richer appearance and usability tests (e.g., cross-browser, accessibility audits), and prototyping mixed-initiative dialogue protocols for clarifying underspecified requirements.

6. Relationship to Adjacent Benchmarks and Research Ecosystem

WebGen-Bench is differentiated from preceding code generation benchmarks (e.g., SWE-Bench, APPS, HumanEval) by focusing on multi-file, full-stack web development rather than single-file algorithms or patching workflows. In contrast to placeholder-based HTML/CSS benchmarks or UI-image-to-code tasks, it directly evaluates functional and visual integration of realistic applications.

Discussion in (Xu et al., 12 May 2025) highlights the value of adopting sequential, interdependent task workflows (as in Web-Bench's 20-step structure), and the necessity of standards- and framework-aware evaluation for LLMs to be productive in contemporary web engineering. Subsequent datasets and frameworks (e.g., WebGen-Instruct, WebGen-Agent) have built on these design principles to demonstrate measurable progress in agentic code synthesis.

7. Significance and Continuing Evolution

WebGen-Bench sets a foundational standard for measuring holistic web-development competence in LLM-based agents. Its rigorous combination of hand-curated tasks, atomic GUI-based evaluation, and multi-modal appearance analysis provides a challenging yet interpretable testbed for both general and specialized model architectures. Performance trends suggest that continuous advances in agentic feedback, RL-fine-tuning, and broader tech-stack coverage will be required to approach robust, autonomous web app generation. The benchmark's structure and findings continue to inform the design of newer datasets such as MM-WebGEN-Bench and WebGen-V, ensuring continued relevance within the research community (Lu et al., 6 May 2025, Xu et al., 12 May 2025, Lu et al., 26 Sep 2025).