---
title: 'SpecifyUI: Specification-Driven UI Design'
url: https://www.emergentmind.com/topics/specifyui
type: topic
---

# SpecifyUI: Specification-Driven UI Design

SpecifyUI is a generative UI design system that replaces prompt-only interaction with a specification-driven workflow for iterative intent expression and controllable generation. Its central contribution is **SPEC**, a structured, parameterized, hierarchical intermediate representation that exposes UI elements as controllable parameters and mediates between references, edits, and final rendering. The system extracts SPEC from UI references via region segmentation and vision-language models, supports composition across multiple sources, performs targeted edits at global, regional, and component levels, and renders the resulting specification into a high-fidelity React UI through a multi-agent pipeline [2509.07334].

## 1. Problem formulation and design rationale

SpecifyUI is motivated by a mismatch between how designers work and how most LLM-based UI tools operate. The paper argues that designers think in visual, structured, and iterative terms, whereas prompt-based tools typically require free-form text and often regenerate the whole UI after each edit. Two deficiencies are emphasized. First, **intent expression is weak**: designers need to communicate layout balance, hierarchy, stylistic mood, component roles, and relationships between parts of a page, but natural language is a poor medium for this. Second, **controllability is poor**: UI design is iterative, and prompt-based regeneration often changes unrelated parts, undermining continuity across revisions [2509.07334].

A formative study with **6 professional designers** is used to ground this diagnosis. The study reports that participants struggled to translate visual references into text, had difficulty expressing compositional relationships such as hierarchy and balance, and found repeated prompt rewriting tedious and error-prone. On this account, SpecifyUI is not framed as a better prompt interface, but as a change in representational substrate: the system makes design intent explicit as a machine-readable specification rather than leaving structure implicit inside prompts.

This reframes UI generation from one-shot text-conditioned synthesis to an iterative editing process over a structured object. A plausible implication is that the system treats generation less as open-ended completion and more as constrained rendering of an evolving design state.

## 2. SPEC as a hierarchical intermediate representation

SPEC is the paper’s formal intermediate representation for UI design intent. It encodes a UI through a hierarchy of **global**, **page/section**, and **component** information, combining numeric parameters with semantic tags. At the global level, the representation is defined as

$$
\mathcal{G} = \langle L, C, S, U \rangle
$$

where $L$ denotes layout structure, $C$ denotes the color system, $S$ denotes shape language, and $U$ denotes usage scenario or contextual design intent [2509.07334].

The paper gives representative examples for each field: $L = \text{Grid}(col=12, spacing=8px)$, $C = (\#CF9BDE, \text{Accent})$, and $S = (r, f)$ where $r$ is a corner radius and $f$ is a semantic descriptor such as “rounded.” This combination of parametric and semantic content is central to the design of SPEC. Numeric and enum values provide deterministic control, while short semantic phrases preserve interpretability for designers.

The page structure is represented hierarchically as

$$
\mathcal{P} = \{\, p_i \mid p_i = \langle Sec_j \rangle \,\}, \quad Sec_j = \{\, comp_k \mid comp_k \in Comp \,\}
$$

which defines a Page $\rightarrow$ Section $\rightarrow$ Component decomposition. A section schema is defined as

$$
SecSchema = \langle id, pos, layout, color, Comp \rangle
$$

and a component schema as

$$
CompSchema = \langle type, id, func, layout, color \rangle .
$$

The system also enforces a consistency relation between section-level and global attributes:

$$
\forall Sec_j \in \mathcal{P}, \quad Sec_j.layout \subseteq \mathcal{G}.L \ \wedge \ Sec_j.color \subseteq \mathcal{G}.C \ \wedge \ Sec_j.shape \subseteq \mathcal{G}.S
$$

[2509.07334].

This design makes SPEC neither a purely symbolic schema nor a purely stylistic descriptor. It is a hybrid specification that preserves hierarchy, exposes local edit targets, and maintains inheritance-like constraints from global style to regional structure. The paper’s broader claim is that such explicit structure is what prompt-only systems lack.

## 3. From references to specification: segmentation, region analysis, and global style extraction

SpecifyUI begins by converting reference UIs into SPEC. This process has three main sub-stages: region segmentation, region-wise specification extraction, and global style extraction [2509.07334].

The segmentation stage uses a custom **Co-DETR** detector trained from scratch. For this purpose, the authors curated **8,000 web UI screenshots**, and **10 annotators** spent **7 days** labeling bounding boxes. The sole annotation rule was that the union of all boxes must cover the full page. The paper argues that off-the-shelf parsers often miss blocks or incorrectly split semantically unified areas, especially in dense or irregular layouts. After detection, a post-processing stage merges regions using Gestalt cues including **alignment**, **proximity**, and **enclosure**.

Each cropped region $r_j \in R$ is then passed to a multimodal LLM, which outputs a **Region SPEC Unit (RSU)**:

$$
RSU_j = Sec_j = \langle id_j, pos_j, layout_j, color_j, Comp_j \rangle .
$$

The model extracts region type, layout structure, components, functional role, and style attributes. The paper states that few-shot chain-of-thought prompting is used to encourage a three-step procedure: identify section type, enumerate components, and encode attributes into RSU format. This localized extraction reduces complexity relative to whole-image reasoning and makes structure explicit earlier in the pipeline.

Because local crops do not fully capture page-wide visual language, the full screenshot is processed separately to derive a **Global Design Profile**, including tone, dominant colors, rhythm, and overall style. This profile populates the `<global_specification>` field and is then integrated with the region-level RSUs into a unified SPEC. In effect, SpecifyUI combines local structural parsing with page-level style summarization rather than assuming either scale alone is sufficient.

## 4. Composition, targeted editing, and multi-agent rendering

A defining feature of SpecifyUI is that SPEC is editable and compositional. Designers can combine multiple references hierarchically, for example taking layout from one source, color from another, and components from a third. The interface exposes a **UI canvas**, a **SPEC panel**, and a **design specification editor**, enabling direct manipulation of extracted design tokens and reassembly of sections and components into a composite hierarchy [2509.07334].

Iterative refinement is represented as structured edit operations. The Edit Generator transforms natural-language or image-based intent into triplets of the form

$$
\langle operation, path, value \rangle
$$

where `operation` denotes an action such as replace or insert, `path` identifies a target node in the SPEC hierarchy, and `value` provides the replacement content. The paper gives the example

$$
\langle replace, /VisualStyle/DesignStyle, \text{"Tech-oriented dark background with neon-accented charts"} \rangle .
$$

Because edits are path-addressed, the system supports localized change at three granularities: **global**, **regional**, and **component**. Invalid instructions trigger an exception-handling loop in which the error message, original instruction, and current SPEC are returned to the LLM for repair, with retries capped at **3**.

Rendering is performed by a multi-agent generation pipeline based on **Qwen3-Coder**, targeting **React**, **Ant Design**, and **Recharts**. The pipeline includes a code agent, a retrieval component, and a debug agent. To support retrieval-augmented generation, the authors built a **SPEC–UI Code Database**: **2,500 professionally designed web UIs** were collected, SPECs were extracted, React implementations were generated using **Claude-4**, and after validation **2,000 high-quality SPEC–code pairs** remained. At inference time, the input SPEC is embedded, nearest examples are retrieved, and these are injected as few-shot demonstrations.

The debug agent closes the loop through a compile–feedback–repair cycle: generate code, compile or render it, capture errors if they occur, convert the trace into a structured report, and return that report to the LLM for repair, again with up to **3** iterations. The overall system therefore treats UI synthesis as specification rendering with retrieval and self-correction, not as unconstrained code emission.

## 5. Empirical evaluation

The technical evaluation measures fidelity to a reference UI screenshot using **MSE**, **CLIP similarity**, and **SSIM**. The baselines are **Direct Prompt**, **Text-augmented Prompt** with OCR tokens, and **Self-Revision Prompt** with iterative self-comparison. The SPEC-based variants are **Only SPEC**, **SPEC + RAG**, **SPEC + Region Crop**, and **Integrated SPEC** [2509.07334].

Integrated SPEC achieves the best reported performance.

| Metric | Integrated SPEC | Self-Revision Prompt |
|---|---:|---:|
| MSE | 40.9930 | 50.8570 |
| CLIP | 0.8871 | 0.7553 |
| SSIM | 0.854 | 0.787 |

The paper highlights three findings. First, **structured specification is the biggest gain**: Only SPEC already exceeds the prompt baselines. Second, **RAG improves semantic alignment**. Third, **region cropping improves structural and perceptual fidelity**. Together these results support the claim that explicit intermediate structure yields more faithful generation than prompt-only baselines.

The user study evaluates human-AI co-creation against **Google Stitch** in a within-subjects design. The study includes **16 participants**, described as **9 female, 7 male**, a mix of university students and professional designers, all familiar with generative AI tools. Participants completed two tasks: designing a **data monitoring dashboard for a social AI system** and designing a **safe, engaging travel service webpage for children and parents**. Each task included **10 reference UIs**, a **10-minute training**, and **20 minutes** for generation and refinement [2509.07334].

Three expert UI designers rated **32 generated designs** on **style consistency**, **layout consistency**, **component correctness**, and **task relevance**, each on a **1–7 Likert scale**. SpecifyUI significantly outperformed Stitch on all four dimensions, all with $p < .001$. The reported means were **5.52 vs. 4.18** for style consistency, **4.99 vs. 3.73** for layout consistency, **5.69 vs. 4.20** for component correctness, and **5.58 vs. 4.92** for task relevance. Participant ratings also favored SpecifyUI on **generation controllability** (**5.88 vs. 3.88**), **editing controllability** (**5.88 vs. 3.94**), **intent adherence** (**6.06 vs. 4.25**), **articulation efficiency** (**5.56 vs. 4.38**), and **ease of communication** (**5.63 vs. 4.31**). Typing effort fell from **1442 characters** on average for Stitch to **386 characters** for SpecifyUI, or about **73% less typing**.

A notable nuance in the findings is temporal distribution of effort: SpecifyUI took longer at initial generation because users built SPECs explicitly, but refinement was significantly faster, and total task time was not significantly different. The paper interprets this as a front-loading of effort in exchange for more predictable downstream iteration.

## 6. Position within UI generation research and open directions

SpecifyUI belongs to a broader movement toward explicit intermediate representations in UI generation. Several adjacent works reinforce this direction. **"Portal UX Agent -- A Plug-and-Play Engine for Rendering UIs from Natural Language Specifications"** formalizes intent-to-UI as $c = f_{\theta}(s), \ y = g(c)$ with schema compliance $c \models \Sigma$, and treats generation as a **compile-and-render** problem using typed compositions and deterministic rendering rather than free-form code emission [2511.00843]. **"Bridging Gulfs in UI Generation through Semantic Guidance"** argues that semantics should function as an intermediate representation between human intent and generated artifacts, organizing UI intent into a four-level hierarchy of Product, Design System, Feature, and Component [2601.19171]. **"UI Remix: Supporting UI Design Through Interactive Example Retrieval and Remixing"** similarly shifts emphasis from one-shot prompting to iterative search, selection, and adaptation of examples at both global and local levels [2601.18759].

Parallel trends appear in design-to-code systems. **"UICopilot: Automating UI Synthesis via Hierarchical Code Generation from Webpage Designs"** decomposes webpage generation into coarse DOM layout generation, leaf-node HTML/CSS generation, and non-leaf style synthesis plus global refinement, arguing that direct one-step generation breaks down on long, deeply nested pages [2505.09904]. **"Bridging Design and Development with Automated Declarative UI Code Generation"** introduces explicit Page Transition Graphs and compiler-driven refinement for declarative mobile UI code generation [2409.11667]. These works differ in target artifact and task setting, but they converge on the same structural claim: hierarchy and explicit intermediate structure improve fidelity, controllability, or functional correctness.

Within this landscape, SpecifyUI is distinctive in centering **designer intent expression** rather than only code synthesis or runtime rendering. Its primary object is not the DOM tree, the transition graph, or the component schema alone, but a hybrid representation that designers can inspect and edit. This suggests that its main contribution is methodological as much as algorithmic: it casts UI generation as collaborative specification editing.

The paper also identifies clear limitations and future directions. It argues for more natural input channels beyond reference images, text, and direct selection, including **rough layout sketches**, **color/shape mood sketches**, **annotation arrows for functionality**, and **drag/drop style interactions**. It suggests extending the method beyond **single-screen UIs** toward **page-level interactions**, **data bindings**, **multi-page navigation**, and **executable product prototypes**. It also notes that SPEC is especially strong once intent is clearer, whereas text prompts may remain useful for more open-ended early exploration. In that sense, SpecifyUI does not eliminate prompting; it repositions prompting inside a larger specification-driven workflow [2509.07334].

Source: https://www.emergentmind.com/topics/specifyui