Vibe Code Bench for AI-driven Web Apps

Updated 4 July 2026

Vibe Code Bench is an evaluation framework that benchmarks AI-mediated, natural-language-driven development for deployable web applications.
It standardizes the development environment using a fixed stack (React, Vite, Tailwind, Supabase) and assesses performance via 964 browser-based workflows.
The benchmark reveals challenges in feature completeness, security, and evaluator alignment, with top models achieving around 58-62% accuracy.

Vibe Code Bench denotes a benchmarked approach to evaluating “vibe coding,” a software-development workflow in which high-level natural-language intent is translated into executable systems by large language models and related agents. In one prominent and specific usage, it refers to a benchmark for end-to-end web application development that measures whether a model can go from a natural-language “zero-to-one” specification to a deployable application, with success judged through browser-based workflows rather than code-coupled assertions [2603.04601]. In adjacent literature, the term is also used more broadly or analogically for benchmark suites targeting feature implementation, molecular program synthesis, instruction following, educational outcomes, security, and automated review, indicating that “Vibe Code Bench” has become both a named benchmark and a wider evaluation motif within the study of AI-mediated programming [2509.22237].

1. Conceptual background in vibe coding

Vibe coding is defined as a style of programming in which the developer “orchestrates” code production almost entirely via conversational prompts to an agentic code-generating LLM, rather than by typing and editing code directly. Its key characteristics include “material disengagement,” an iterative conversational loop, “context momentum,” and “trust calibration.” Sarkar and Drosos describe an 8-step iterative goal-satisfaction cycle: formulate a subgoal, generate a prompt, receive a code diff, review the diff, accept or reject it, test the running application, identify bugs or refinements, and then either refine the prompt or switch to manual editing. They also emphasize that vibe coding does not eliminate the need for programming expertise; instead, expertise is redistributed toward context management, rapid code evaluation, and decisions about when to transition between AI-driven and manual manipulation of code [2506.23253].

A practical demonstration of the paradigm appears in a proteomics setting. Meyer reports that a fully functional proteomics data analysis website capable of performing standard tasks, including data normalization, differential expression testing, and volcano plot visualization, was developed in less than ten minutes using only four natural-language prompts, without any manual coding, at a cost of under \$2. In the detailed walk-through, the development cost is \$1.96, the code volume is approximately 1,400 lines across four modules, and the application reproduced published PCA and volcano results within published tolerances, with summary statistics matching within 1–2% of the published values [2510.09804]. This suggests why end-to-end evaluation became salient: the bottleneck shifts from isolated code synthesis to the reliability of complete AI-mediated development workflows.

2. The end-to-end web-application benchmark

The benchmark most directly titled Vibe Code Bench is designed to fill a gap left by code-generation benchmarks that measure isolated tasks rather than complete application construction. It consists of 100 total web-app specifications, split evenly into a public validation set of 50 tasks and a held-out test set of 50 tasks; the splits are disjoint. Each specification is paired with user-like UI tests, yielding 964 browser-based workflows and 10,131 individual substeps [2603.04601].

Its core object of measurement is the ability to build a working application from scratch. Models operate inside isolated Docker-in-Docker containers with a 5 hr wall-clock cap per task. The harness provides a terminal for shell commands, a headless browser for both documentation lookup and self-testing, and integrated services including Supabase, MailHog, and Stripe test-mode endpoints. A fixed system prompt mandates a React + Vite + Tailwind + Supabase stack, a Docker Compose entry point, and explicit self-testing before submission. The benchmark therefore standardizes both the target stack and the development environment, making “zero-to-one” application development comparable across models [2603.04601].

The benchmark’s stated contributions are a novel dataset of 100 natural-language web-app specifications paired with realistic browser workflows, an open reproducible agentic development harness based on a fork of OpenHands, an automated browser-based evaluation pipeline, a large-scale evaluation across 16 state-of-the-art models, and a human-alignment and evaluator-alignment protocol. Within this framing, Vibe Code Bench is not a unit-test corpus or a repository-patching suite; it is a deployed-application benchmark whose unit of success is observable user-facing behavior [2603.04601].

3. Browser-based evaluation and formal metrics

The evaluation protocol is deployment-centered. Each submitted application is brought up via docker compose in an isolated environment, and failure to start immediately marks all workflows as failed. For each workflow, the benchmark launches a fresh headless browser at 1920×1200, driven by the vision-enabled Browser Use agent, with Claude Sonnet 4.5 as the default evaluator. The browser agent follows natural-language substeps and emits structured pass/fail judgments for each substep [2603.04601].

A workflow is counted as passing if at least 90% of its substeps succeed. Application accuracy is then defined as the percentage of workflows passing. The paper formalizes per-model test-split accuracy as

$$
\text{Accuracy}m \;=\; \frac{\sum{i=1}ⁿ \mathbf{1}[\text{workflow}_i^{\text{pass}]}{n}\times} 100\%,
$$

where (n=50) tasks and each task’s workflows are aggregated through the (\ge 90\%) substep rule. The benchmark also reports step-level success rate, defined as raw substeps passed over total substeps, together with cost in US dollars and latency in minutes. For association analyses, such as relating browser self-testing frequency to final accuracy, it uses the Pearson correlation coefficient

$$
r = \frac{\sum_i (x_i-\bar x)(y_i-\bar y)}{\sqrt{\sum_i(x_i-\bar x)^{2}\,\sqrt{\sum_i(y_i-\bar} y)^2}}.
$$

This evaluation design is consequential because it separates application success from internal implementation details. A model is rewarded for delivering a deployable system that behaves correctly under realistic user workflows, not for matching a reference patch or satisfying code-level assertions [2603.04601].

4. Reported performance, failure modes, and evaluator alignment

Across 16 frontier models, the benchmark reports a substantial performance ceiling. The abstract states that “the best achieves only 58.0% accuracy on the test split,” while the detailed results section reports that GPT-5.3-Codex achieves (61.77 \pm 4.71\%), Claude Opus 4.6 achieves (57.57 \pm 4.37\%), and GPT-5.2 achieves (53.50 \pm 5.07\%) on the test split. Cost and latency expose clear trade-offs: GPT-5.3-Codex is reported at \$11.91 and 75.8 min, whereas Claude Opus 4.6 is reported at \$8.69 and 21.3 min. The study notes that models spending more time or money generally do better, but with diminishing returns [2603.04601].

The failure taxonomy is dominated by omitted functionality rather than subtle logic defects. Among failed substeps for applications that do start, the aggregate breakdown across the top five models is: Missing Feature, 46.7%; Authorization Issue, 20.4%; Validation/Policy Block, 14.8%; UI Rendering/Navigation, 6.0%; Data Consistency/Logic, 1.9%; and Other, 10.2%. This pattern suggests that the main obstacle in end-to-end generation is not merely syntax or local correctness, but complete specification coverage and robust integration [2603.04601].

A central analytical result concerns self-testing. Browser-call volume during development correlates strongly with end-to-end accuracy, with Pearson (r=0.72), and this remains (r=0.72) after controlling for total runtime. By contrast, raw edit volume shows almost no correlation with final accuracy, at (r=0.09). The benchmark therefore identifies self-testing during generation as a strong performance predictor rather than a secondary convenience [2603.04601].

The evaluator-alignment study is equally important. It uses a 1,401-substep panel derived from 18 generated applications, with judgments from four model-based evaluators and three human reviewers. Pairwise step-level agreement ranges from 31.8% up to 93.6%; human–human agreement lies in ([88.6\%, 93.6\%]); model–model agreement ranges from 33.1% to 87.8%; and mean model–human agreement is 86.4% for Claude Sonnet 4.5, 86.3% for Claude Sonnet 4.6, 84.7% for Gemini 3.1 Pro, and 36.1% for GPT 5.2. The benchmark concludes that evaluator choice materially affects reported outcomes, and that some LLM evaluators align very closely with expert humans [2603.04601]. A common misconception is therefore that browser-based automatic evaluation is evaluator-independent; the reported agreement ranges show that it is not.

5. Related benchmark families and the spread of the term

The phrase “Vibe Code Bench” also appears in adjacent benchmark programs that target different parts of the vibe-coding problem. Some focus on repository-scale feature implementation from pure natural language, some on executable chemistry workflows, and some on the gap between functional correctness and human preference.

Benchmark	Scope	Key facts
Vibe Code Bench	End-to-end web application development	100 specifications; 964 workflows; 10,131 substeps [2603.04601]
FeatBench	Feature implementation in existing repositories	157 tasks; best success rate (29.94\%) [2509.22237]
MolViBench	Molecular vibe coding	358 tasks; no model exceeds 40% Pass@1 overall [2605.02351]
Vibe Checker	Functional correctness plus instruction following	30 verifiable instructions; 1,140 BigVibeBench and 1,055 LiveVibeBench instances [2510.07315]
SusVibes	Security of agent-generated implementations	200 tasks; best FuncPass (61.0\%), SecPass (10.5\%) [2512.03262]

FeatBench is explicitly motivated by the claim that existing evaluation benchmarks are misaligned with vibe coding because they either require code-level specifications or focus on issue solving rather than feature implementation. It uses pure natural-language prompts that begin with first-person requests, draws 157 high-quality tasks from 27 actively maintained Python repositories, and validates solutions with both Fail-to-Pass and Pass-to-Pass tests. Its best configuration, Trae-agent + GPT-5, reaches only (29.94\%) success rate, and all agents exhibit regression-test pass rates below 50% [2509.22237].

MolViBench extends the idea into chemistry. It comprises 358 curated tasks across five cognitive levels and 12 real-world drug-discovery workflows, and evaluates both executability and chemical correctness through a three-stage framework: executability, type-aware exactness, and AST-based API-semantic fallback. Even the best configuration reaches Pass@1 (=39.7\%), executable rate (=98.9\%), and fallback pass rate (=72.6\%), with performance degrading monotonically from Level 1 to Level 5 [2605.02351].

Vibe Checker addresses a different limitation: the mismatch between pass@k and human preference. It introduces VeriCode, a taxonomy of 30 verifiable code instructions, and uses them to construct BigVibeBench from BigCodeBench and LiveVibeBench from LiveCodeBench. The benchmark defines a composite score (S_\alpha = \alpha \cdot IF + (1-\alpha)\cdot Func), where (IF) is instruction following and (Func) is functional correctness. The strongest reported correlation with human-preference signals appears when both are combined, rather than when functionality alone is used [2510.07315].

Other works broaden the family still further. “Vibe Coding Ate My Homework” presents a greenfield Python evaluation suite with five tasks at three prompt levels, recording JSON-format validity, syntax validity, a functionality score over one retry, and pass/fail verdicts; its adjusted pass rates range from 80.0% to 92.9% across four locally deployed models [2606.18293]. “Vibe Coding on Trial” studies unanimous LLM juries for text-to-SQL review on 82 MySQL prompts and recommends small unanimous committees of size (k=2)–(3) when false accepts are costlier than false rejects [2602.18492]. The Vibe-Check Protocol is a theoretical educational framework rather than a deployed benchmark corpus, defining three metrics—Cold Start Refactor (M_{CSR}), Hallucination Trap Detection (M_{HT}), and Explainability Gap (E_{gap})—to quantify cognitive offloading in AI programming [2601.02410]. Taken together, these works indicate that “Vibe Code Bench” functions both as a proper benchmark name and as a broader umbrella for evaluation of natural-language-driven software creation.

6. Limitations, safety, and broader research significance

The broader benchmark literature shows that end-to-end functional success does not settle questions of safety, alignment, or human oversight. SusVibes makes this point sharply. It evaluates multi-turn agents on 200 repository-scale feature requests derived from historical vulnerability fixes and reports that, although 61% of solutions from SWE-Agent with Claude 4 Sonnet are functionally correct, only 10.5% are secure. The study further reports that simple prompt-based security nudges either fail to improve SecPass or sharply degrade FuncPass by 5–9 percentage points. Common failures include timing side channels, CRLF/header injection, stored XSS, and expired-session acceptance [2512.03262]. A plausible implication is that a benchmark centered only on externally visible workflow completion can overestimate deployment readiness in security-sensitive settings.

A parallel limitation concerns evaluation criteria themselves. Vibe Checker argues that pass@k captures only functional correctness and omits the non-functional instructions that users routinely apply in practice. On its augmented suites, a composite of functionality and instruction following correlates with human preference better than either signal alone, with instruction following emerging as the primary differentiator on real-world programming tasks [2510.07315]. This suggests that a mature account of Vibe Code Bench cannot be restricted to “does it run?” and must also consider whether the solution preserves intent, style, documentation, error handling, and library conventions.

Educational benchmarking introduces a third axis: cognition. The Vibe-Check Protocol is explicitly theoretical and does not yet report full empirical data, but it formalizes how AI-assisted coding may redistribute or offload learning. Its proposed metrics quantify skill decay after removing AI scaffolding, the ability to detect hallucinated or buggy code, and the gap between code complexity and conceptual explanation [2601.02410]. This suggests that future Vibe Code Bench variants may evaluate not only artifact quality but also the extent to which developers retain procedural knowledge, vigilance, and explainability under AI-mediated workflows.

Within this larger research landscape, Vibe Code Bench is best understood as a transition point in benchmark design. It moves evaluation away from isolated functions and toward complete socio-technical workflows: prompt-driven development, self-testing, deployment, browser-based verification, evaluator calibration, and downstream concerns such as security and human preference. The benchmark’s central empirical message is that reliable end-to-end application development remains a frontier challenge [2603.04601], while the surrounding literature shows that feature completeness, secure implementation, instruction following, and cognitive ownership remain unresolved dimensions of the same paradigm.