---
title: 'VISTA: Visual Spec-to-App Benchmark'
url: https://www.emergentmind.com/topics/vista
type: topic
---

# VISTA: Visual Spec-to-App Benchmark

Searching arXiv for the benchmark paper and a few related benchmarks mentioned in the provided material.
{"query":"2605.26144 VISTA benchmark visual spec to web app coding agents"}
VISTA, short for VIsual Spec-To-App Benchmark, is an end-to-end benchmark and evaluation framework for “visual spec-to-web-app” coding agents: LLM-based systems tasked with building runnable multi-page web applications from mixed natural-language and visual specifications rather than isolated coding puzzles. It targets realistic UI-centric development in which agents must infer underspecified behavior, implement interactivity, survive build and runtime errors, and produce applications that are simultaneously functional and visually coherent. Its design combines ten Figma-derived web applications, five prompt-information conditions, human annotations, DOM-grounded evaluation, behavior-specific browser checks, and CLIP-based visual similarity, with the explicit goal of separating structural alignment, behavioral completeness, and visual fidelity [2605.26144].

## 1. Benchmark scope and task formulation

VISTA was introduced to address limitations in existing code benchmarks that focus on algorithmic or back-end tasks, evaluate agents in fixed stacks or single input modalities, rely heavily on LLM-as-judge or brittle browser scripts, and treat the model separately from the harness even though the harness significantly affects performance. In contrast, VISTA evaluates full coding agents operating in realistic development environments, where they inspect files, edit code, run builds or tests, and repair failures across complete web applications rather than code snippets [2605.26144].

The benchmark is organized around ten multi-page web applications with realistic UI and interaction patterns, derived from Figma designs. The task is to generate a runnable application from a product target in natural language, optionally augmented by reference screenshots and a pruned Figma JSON structure, under either free stack choice or a specified scaffold. The resulting application is expected to match intended UI structure and layout, implement behaviors such as navigation, inputs, toggles, and popouts, and remain executable after iterative repair. This makes VISTA substantially closer to front-end software engineering than to conventional program-synthesis tasks [2605.26144].

A central design choice is the use of five prompt-information conditions, defined over two axes: visual or structural fidelity, and stack constraint. The conditions are \(C_0\) text only with free stack, \(C_1\) text plus screenshots with three specified stacks, \(C_2\) text plus screenshots with free stack, \(C_3\) text plus screenshots plus pruned Figma JSON with a single specified stack, and \(C_4\) text plus screenshots plus pruned Figma JSON with free stack. This allows independent study of how visual grounding, structural grounding, and implementation freedom interact [2605.26144].

The primary task metric is the combined structure-function score
\[
S = \frac{1}{N} \sum_{i=1}^{N} L_i \cdot B_i,
\]
where \(L_i \in [0,1]\) is a localization score for the matched DOM element corresponding to annotated interaction \(i\), \(B_i \in [0,1]\) is the behavior score for that interaction, and \(N\) is the number of critical annotated interactions. This formulation makes correctness explicitly multiplicative: a well-localized but non-functional control, or a functional behavior assigned to the wrong element, both depress the score [2605.26144].

## 2. Dataset composition, annotations, and interfaces

VISTA includes 10 application categories: Newsletter, Real estate, Job board, Forum, Travel booking, Chat, Cloud storage, E-commerce, Project management, and Music streaming. Across these categories, the benchmark contains 128 pages, 3,253 interactive annotations, and 458 visual anchors. The applications cover dashboards, navigation menus, filters, tabs, lists and item detail views, search, forms, authentication-style flows, and e-commerce carts and checkouts, thereby stressing multi-page navigation, application state, and interaction logic rather than static page reconstruction alone [2605.26144].

Each page originates from a Figma design. The benchmark exports PNG screenshots for each page and a pruned Figma JSON structure with layout and hierarchy information. Raw Figma JSON is explicitly reduced by removing rendering-only metadata while retaining positions, frames, component-tree hierarchy, text labels, and interaction targets where possible. In \(C_3\) and \(C_4\), this pruned JSON functions as the structural input intended to help agents infer component boundaries, repeated patterns, and nesting [2605.26144].

Human annotation is foundational. Every page is annotated with controls that “should trigger a visible state change, navigation, or request,” including buttons, links, inputs, textareas, selects, menus, tabs, filters, toggles, and popouts. Each annotation includes a bounding box and an interaction type drawn from `navigation`, `input`, `toggle`, `external link`, `popout`, or `generic click`. In addition, each page has 2–5 visual anchors, labeled with stable IDs such as `<search>` or `<checkout>`, which are used for page-to-URL mapping, coordinate alignment, and robustness to layout shifts [2605.26144].

The benchmark also standardizes scaffolding and execution interfaces. Instead of large GitHub templates, it uses lightweight scaffolds such as `create-next-app` with TypeScript, Tailwind, ESLint, and modern routing. For each task, an LLM proposes three stacks, labeled pick A, B, and C; these support \(C_1\), and pick A is reused as the fixed template in \(C_3\). Agents run in harnesses comparable to a patch-oriented Codex harness or the Claude Code environment with `Write`, `Edit`, and `MultiEdit`, and VISTA logs file mutations, commands, tool usage, build or test attempts, and timestamps for later workflow analysis [2605.26144].

## 3. Evaluation methodology

VISTA’s evaluator combines three components: DOM-grounded reference matching and behavior tests as the primary measure, behavior-specific browser automation, and CLIP-based visual similarity. The central claim of the benchmark is that naive script-based testing is inadequate in open-ended UI generation, because small layout changes, selector differences, and alternate HTML structures can invalidate brittle scripts without reflecting true functional failure. VISTA addresses this by grounding evaluation in human annotations, spatial alignment, and semantic matching [2605.26144].

The first step is page-to-URL mapping. Generated applications may not preserve Figma page names or route structures, so the evaluator maps reference pages to app URLs by combining declared route mappings, crawling internal links, source route patterns, DOM signatures, visual anchors, and text cues. It then performs coordinate alignment between Figma-space annotations and rendered pages via a per-axis affine transform
\[
x' = s_x x + t_x,\quad y' = s_y y + t_y,
\]
which tolerates global rescaling and shifting while preserving expectations about local relative placement [2605.26144].

After alignment, the evaluator extracts visible interactive DOM elements such as `<a>`, `<button>`, `<input>`, `<select>`, `<textarea>`, checkboxes, radio controls, and ARIA-role interactive elements, ignoring hidden or zero-size elements. Localization is computed through tiered matching. Exact IoU thresholds or center-distance thresholds yield partial credit, and a text-similarity fallback yields a small residual score. The tiers are: IoU \(\ge 0.30\) gives \(L_i = 1.00\); IoU \(\ge 0.10\) gives \(0.60\); center distance \(\le 150\) px gives \(0.30\); center distance \(\le 600\) px gives \(0.15\); text similarity fallback gives \(0.10\); and a miss gives \(0.00\) [2605.26144].

Behavior scoring then triggers interactions on matched elements and checks expected outcomes. Navigation requires a URL change or meaningful page transition; input requires accepting text and firing input or change events; toggle requires state change such as checked or ARIA expanded or collapsed; popout requires opening a dialog or overlay; and external link requires exposing an external URL. VISTA also reports mean localization \(\bar{L}\) and mean behavior \(\bar{B}\), which makes it possible to separate structural from behavioral failures [2605.26144].

CLIP-based visual similarity is computed between screenshots of rendered generated pages and ground-truth Figma screenshots, using a CLIP model to embed both images and aggregate similarity over pages. A common misconception in visual UI evaluation is that screenshot similarity is sufficient. VISTA explicitly rejects that premise: CLIP is treated as complementary rather than primary, since high CLIP with low behavior can correspond to a visually plausible but non-functional UI, while high behavior with lower CLIP can indicate a functionally correct but visually inaccurate implementation [2605.26144].

## 4. Agents, harnesses, and experimental protocol

The benchmark evaluates four deployed agents drawn from two model families and two harnesses. The GPT family includes GPT-5.4 and GPT-5.5, both operating in a Codex-like harness characterized by patch-oriented editing, shell commands, and batch file changes. The Claude family includes Claude Sonnet and Claude Opus, both operating in the Claude Code harness with `Write`, `Edit`, and `MultiEdit` tools. VISTA evaluates these systems as full agents rather than raw LLMs, meaning they can inspect files, modify files, run builds or tests, and iterate on failures [2605.26144].

All ten tasks are run under all five prompt conditions \(C_0\) through \(C_4\). In \(C_1\), each task is solved once per pick A, B, and C stack, and results are averaged, with per-pick results reported separately. For every task, condition, and model combination, the agent is allowed a full end-to-end development cycle. Reported metrics include mean localization, mean behavior, combined \(S\), median \(S\), and CLIP similarity [2605.26144].

VISTA also instruments the process of software generation itself. It logs reads, writes, edits, deletes, commands, and temporal action sequences, enabling an analysis of editing style. The benchmark introduces the Surgical Diff Score:
\[
\text{Surgical Diff} = 100 \cdot (0.40 A + 0.30 B + 0.30 C),
\]
where \(A\) is edit-event share, \(B\) is diff-byte share, and \(C\) is targetedness. It also defines a stricter variant,
\[
\text{Strict Diff} = 100 \cdot A \cdot (0.5 B + 0.5 C),
\]
which penalizes agents that rarely use edit operations. These metrics are not quality metrics for the task itself; they quantify process style, especially the degree to which an agent rewrites files versus making localized edits [2605.26144].

Workflow trajectories show a recurrent inspect–write–verify rhythm, with family-level differences. Claude models exhibit stronger inspect-to-inspect transitions and high failure-to-inspect transitions, while GPT models show more diverse transitions after failures and verification. The benchmark’s interpretation is cautious: these workflow grammars are descriptive of process, not direct indicators of task success [2605.26144].

## 5. Empirical results and performance patterns

Aggregate results across prompt conditions reveal a consistent but nontrivial interaction between input richness and stack freedom. \(C_0\) yields localization \(0.489\), behavior \(0.315\), combined \(S = 0.242\), and median \(S = 0.234\). Adding screenshots while pinning stacks in \(C_1\) raises localization to \(0.577\) but lowers combined \(S\) to \(0.229\). Freeing the stack in \(C_2\) raises combined \(S\) to \(0.264\), the best mean combined score. The richest input with free stack, \(C_4\), gives localization \(0.600\), behavior \(0.329\), combined \(S = 0.261\), median \(S = 0.253\), and the best CLIP score, \(0.827\) [2605.26144].

These results motivate several conclusions. Screenshots improve localization: \(C_0 \to C_1\) raises localization by 18%, from \(0.489\) to \(0.577\). Free stack choice significantly boosts combined performance: \(C_1 \to C_2\) changes combined \(S\) from \(0.229\) to \(0.264\), and \(C_3 \to C_4\) changes it from \(0.236\) to \(0.261\). Figma structure yields the highest localization values, with \(C_3 = 0.615\) and \(C_4 = 0.600\). However, richer structural input does not automatically improve functional performance, since \(C_3\) remains below \(C_2\) on combined \(S\) despite additional Figma JSON [2605.26144].

Per-model results are even more revealing. GPT-5.5 has the best localization, \(0.602\), and best CLIP, \(0.853\), but the worst behavior, \(0.283\), and the lowest combined \(S\), \(0.223\). Claude Opus has the best combined \(S\), \(0.261\), the best median \(S\), \(0.250\), and the best behavior, \(0.336\), while its CLIP score is \(0.764\), not the best. GPT-5.4 and Sonnet occupy intermediate positions. This establishes a central empirical point of the benchmark: visual fidelity and functional correctness are partially decoupled across both models and prompt conditions [2605.26144].

Stack choice within \(C_1\) is not neutral. Pick A is consistently best, with localization \(0.622\), behavior \(0.317\), combined \(0.240\), median \(0.254\), and CLIP \(0.823\), outperforming picks B and C. This justifies the reuse of pick A as the fixed stack scaffold in \(C_3\) [2605.26144].

Editing-style analysis shows that high quality does not imply surgical editing. Opus, which has the best combined score, has the lowest Surgical Diff score, \(22.4\), and rewrite share \(0.988\). GPT-5.5, which has the lowest combined score, has the highest Surgical Diff score, \(33.6\), and Strict Diff score, \(9.0\). The correlation between Surgical Diff and combined \(S\) is low and slightly negative, \(\rho = -0.145\), and the Strict Diff correlation is \(-0.078\). A plausible implication is that localized editing is not a reliable proxy for end-task quality in agentic web-app generation [2605.26144].

## 6. Interpretation, relation to prior benchmarks, and significance

VISTA positions itself between design-to-code and agentic software-engineering benchmarks. Relative to SWE-bench, SWE-agent, and OpenHands, it shifts attention from real GitHub issues and general software engineering toward visual specs, layout, multi-page UI, and design fidelity. Relative to WebArena, VisualWebArena, and OSWorld, it changes the task from acting in existing web or operating-system environments to building new applications from scratch. Relative to Design2Code, it extends static design reconstruction into multi-page, interactive, end-to-end application generation [2605.26144].

A recurrent misconception in evaluation of coding agents is that app quality can be read off from a single scalar such as screenshot similarity, build success, or edit efficiency. VISTA systematically argues against this reduction. Its results show that structural fidelity, functional behavior, and overall visual similarity can move independently; that rich structural input can improve localization without improving behavior; that stack familiarity can dominate the effect of better specifications; and that process-level measures such as rewrite share or targeted edits are largely orthogonal to outcome quality [2605.26144].

The benchmark’s larger significance lies in reproducibility and extensibility. It fixes designs, annotations, and scaffolds; formalizes a structure-function metric \(S\); augments it with CLIP, localization, behavior, and editing-style analyses; and supports future addition of new stacks, more complex applications, and extra evaluation dimensions such as accessibility, performance, and robustness. In that sense, VISTA establishes a rigorous and reproducible foundation for studying agent-based front-end software engineering under mixed visual, structural, and textual specifications [2605.26144].

Source: https://www.emergentmind.com/topics/vista