SpecifyUI: Specification-Driven UI Design

• SpecifyUI is a generative UI design system that uses a structured, hierarchical intermediate representation (SPEC) to clearly capture design intent.
• The system decomposes UI elements into global, regional, and component levels using vision-language models and custom region segmentation for precise control.
• It integrates iterative editing with a multi-agent rendering pipeline to convert SPEC into high-quality React UIs while reducing manual revisions.

SpecifyUI is a generative UI design system that replaces prompt-only interaction with a specification-driven workflow for iterative intent expression and controllable generation. Its central contribution is SPEC, a structured, parameterized, hierarchical intermediate representation that exposes UI elements as controllable parameters and mediates between references, edits, and final rendering. The system extracts SPEC from UI references via region segmentation and vision-LLMs, supports composition across multiple sources, performs targeted edits at global, regional, and component levels, and renders the resulting specification into a high-fidelity React UI through a multi-agent pipeline (Chen et al., 9 Sep 2025).

1. Problem formulation and design rationale

SpecifyUI is motivated by a mismatch between how designers work and how most LLM-based UI tools operate. The paper argues that designers think in visual, structured, and iterative terms, whereas prompt-based tools typically require free-form text and often regenerate the whole UI after each edit. Two deficiencies are emphasized. First, intent expression is weak: designers need to communicate layout balance, hierarchy, stylistic mood, component roles, and relationships between parts of a page, but natural language is a poor medium for this. Second, controllability is poor: UI design is iterative, and prompt-based regeneration often changes unrelated parts, undermining continuity across revisions (Chen et al., 9 Sep 2025).

A formative study with 6 professional designers is used to ground this diagnosis. The study reports that participants struggled to translate visual references into text, had difficulty expressing compositional relationships such as hierarchy and balance, and found repeated prompt rewriting tedious and error-prone. On this account, SpecifyUI is not framed as a better prompt interface, but as a change in representational substrate: the system makes design intent explicit as a machine-readable specification rather than leaving structure implicit inside prompts.

This reframes UI generation from one-shot text-conditioned synthesis to an iterative editing process over a structured object. A plausible implication is that the system treats generation less as open-ended completion and more as constrained rendering of an evolving design state.

2. SPEC as a hierarchical intermediate representation

SPEC is the paper’s formal intermediate representation for UI design intent. It encodes a UI through a hierarchy of global, page/section, and component information, combining numeric parameters with semantic tags. At the global level, the representation is defined as

$\mathcal{G} = \langle L, C, S, U \rangle$

where $L$ denotes layout structure, $C$ denotes the color system, $S$ denotes shape language, and $U$ denotes usage scenario or contextual design intent (Chen et al., 9 Sep 2025).

The paper gives representative examples for each field: $L = \text{Grid}(col=12, spacing=8px)$, $C = (\#CF9BDE, \text{Accent})$, and $S = (r, f)$ where $r$ is a corner radius and $f$ is a semantic descriptor such as “rounded.” This combination of parametric and semantic content is central to the design of SPEC. Numeric and enum values provide deterministic control, while short semantic phrases preserve interpretability for designers.

The page structure is represented hierarchically as

$L$0

which defines a Page $L$1 Section $L$2 Component decomposition. A section schema is defined as

$L$3

and a component schema as

$L$4

The system also enforces a consistency relation between section-level and global attributes:

$L$5

(Chen et al., 9 Sep 2025).

This design makes SPEC neither a purely symbolic schema nor a purely stylistic descriptor. It is a hybrid specification that preserves hierarchy, exposes local edit targets, and maintains inheritance-like constraints from global style to regional structure. The paper’s broader claim is that such explicit structure is what prompt-only systems lack.

3. From references to specification: segmentation, region analysis, and global style extraction

SpecifyUI begins by converting reference UIs into SPEC. This process has three main sub-stages: region segmentation, region-wise specification extraction, and global style extraction (Chen et al., 9 Sep 2025).

The segmentation stage uses a custom Co-DETR detector trained from scratch. For this purpose, the authors curated 8,000 web UI screenshots, and 10 annotators spent 7 days labeling bounding boxes. The sole annotation rule was that the union of all boxes must cover the full page. The paper argues that off-the-shelf parsers often miss blocks or incorrectly split semantically unified areas, especially in dense or irregular layouts. After detection, a post-processing stage merges regions using Gestalt cues including alignment, proximity, and enclosure.

Each cropped region $L$6 is then passed to a multimodal LLM, which outputs a Region SPEC Unit (RSU):

$L$7

The model extracts region type, layout structure, components, functional role, and style attributes. The paper states that few-shot chain-of-thought prompting is used to encourage a three-step procedure: identify section type, enumerate components, and encode attributes into RSU format. This localized extraction reduces complexity relative to whole-image reasoning and makes structure explicit earlier in the pipeline.

Because local crops do not fully capture page-wide visual language, the full screenshot is processed separately to derive a Global Design Profile, including tone, dominant colors, rhythm, and overall style. This profile populates the <global_specification> field and is then integrated with the region-level RSUs into a unified SPEC. In effect, SpecifyUI combines local structural parsing with page-level style summarization rather than assuming either scale alone is sufficient.

4. Composition, targeted editing, and multi-agent rendering

A defining feature of SpecifyUI is that SPEC is editable and compositional. Designers can combine multiple references hierarchically, for example taking layout from one source, color from another, and components from a third. The interface exposes a UI canvas, a SPEC panel, and a design specification editor, enabling direct manipulation of extracted design tokens and reassembly of sections and components into a composite hierarchy (Chen et al., 9 Sep 2025).

Iterative refinement is represented as structured edit operations. The Edit Generator transforms natural-language or image-based intent into triplets of the form

$L$8

where operation denotes an action such as replace or insert, path identifies a target node in the SPEC hierarchy, and value provides the replacement content. The paper gives the example

$L$9

Because edits are path-addressed, the system supports localized change at three granularities: global, regional, and component. Invalid instructions trigger an exception-handling loop in which the error message, original instruction, and current SPEC are returned to the LLM for repair, with retries capped at 3.

Rendering is performed by a multi-agent generation pipeline based on Qwen3-Coder, targeting React, Ant Design, and Recharts. The pipeline includes a code agent, a retrieval component, and a debug agent. To support retrieval-augmented generation, the authors built a SPEC–UI Code Database: 2,500 professionally designed web UIs were collected, SPECs were extracted, React implementations were generated using Claude-4, and after validation 2,000 high-quality SPEC–code pairs remained. At inference time, the input SPEC is embedded, nearest examples are retrieved, and these are injected as few-shot demonstrations.

The debug agent closes the loop through a compile–feedback–repair cycle: generate code, compile or render it, capture errors if they occur, convert the trace into a structured report, and return that report to the LLM for repair, again with up to 3 iterations. The overall system therefore treats UI synthesis as specification rendering with retrieval and self-correction, not as unconstrained code emission.

5. Empirical evaluation

The technical evaluation measures fidelity to a reference UI screenshot using MSE, CLIP similarity, and SSIM. The baselines are Direct Prompt, Text-augmented Prompt with OCR tokens, and Self-Revision Prompt with iterative self-comparison. The SPEC-based variants are Only SPEC, SPEC + RAG, SPEC + Region Crop, and Integrated SPEC (Chen et al., 9 Sep 2025).

Integrated SPEC achieves the best reported performance.

Metric	Integrated SPEC	Self-Revision Prompt
MSE	40.9930	50.8570
CLIP	0.8871	0.7553
SSIM	0.854	0.787

The paper highlights three findings. First, structured specification is the biggest gain: Only SPEC already exceeds the prompt baselines. Second, RAG improves semantic alignment. Third, region cropping improves structural and perceptual fidelity. Together these results support the claim that explicit intermediate structure yields more faithful generation than prompt-only baselines.

The user study evaluates human-AI co-creation against Google Stitch in a within-subjects design. The study includes 16 participants, described as 9 female, 7 male, a mix of university students and professional designers, all familiar with generative AI tools. Participants completed two tasks: designing a data monitoring dashboard for a social AI system and designing a safe, engaging travel service webpage for children and parents. Each task included 10 reference UIs, a 10-minute training, and 20 minutes for generation and refinement (Chen et al., 9 Sep 2025).

Three expert UI designers rated 32 generated designs on style consistency, layout consistency, component correctness, and task relevance, each on a 1–7 Likert scale. SpecifyUI significantly outperformed Stitch on all four dimensions, all with $C$0. The reported means were 5.52 vs. 4.18 for style consistency, 4.99 vs. 3.73 for layout consistency, 5.69 vs. 4.20 for component correctness, and 5.58 vs. 4.92 for task relevance. Participant ratings also favored SpecifyUI on generation controllability (5.88 vs. 3.88), editing controllability (5.88 vs. 3.94), intent adherence (6.06 vs. 4.25), articulation efficiency (5.56 vs. 4.38), and ease of communication (5.63 vs. 4.31). Typing effort fell from 1442 characters on average for Stitch to 386 characters for SpecifyUI, or about 73% less typing.

A notable nuance in the findings is temporal distribution of effort: SpecifyUI took longer at initial generation because users built SPECs explicitly, but refinement was significantly faster, and total task time was not significantly different. The paper interprets this as a front-loading of effort in exchange for more predictable downstream iteration.

6. Position within UI generation research and open directions

SpecifyUI belongs to a broader movement toward explicit intermediate representations in UI generation. Several adjacent works reinforce this direction. "Portal UX Agent -- A Plug-and-Play Engine for Rendering UIs from Natural Language Specifications" formalizes intent-to-UI as $C$1 with schema compliance $C$2, and treats generation as a compile-and-render problem using typed compositions and deterministic rendering rather than free-form code emission (Li et al., 2 Nov 2025). "Bridging Gulfs in UI Generation through Semantic Guidance" argues that semantics should function as an intermediate representation between human intent and generated artifacts, organizing UI intent into a four-level hierarchy of Product, Design System, Feature, and Component (Park et al., 27 Jan 2026). "UI Remix: Supporting UI Design Through Interactive Example Retrieval and Remixing" similarly shifts emphasis from one-shot prompting to iterative search, selection, and adaptation of examples at both global and local levels (Wang et al., 26 Jan 2026).

Parallel trends appear in design-to-code systems. "UICopilot: Automating UI Synthesis via Hierarchical Code Generation from Webpage Designs" decomposes webpage generation into coarse DOM layout generation, leaf-node HTML/CSS generation, and non-leaf style synthesis plus global refinement, arguing that direct one-step generation breaks down on long, deeply nested pages (Gui et al., 15 May 2025). "Bridging Design and Development with Automated Declarative UI Code Generation" introduces explicit Page Transition Graphs and compiler-driven refinement for declarative mobile UI code generation (Zhou et al., 2024). These works differ in target artifact and task setting, but they converge on the same structural claim: hierarchy and explicit intermediate structure improve fidelity, controllability, or functional correctness.

Within this landscape, SpecifyUI is distinctive in centering designer intent expression rather than only code synthesis or runtime rendering. Its primary object is not the DOM tree, the transition graph, or the component schema alone, but a hybrid representation that designers can inspect and edit. This suggests that its main contribution is methodological as much as algorithmic: it casts UI generation as collaborative specification editing.

The paper also identifies clear limitations and future directions. It argues for more natural input channels beyond reference images, text, and direct selection, including rough layout sketches, color/shape mood sketches, annotation arrows for functionality, and drag/drop style interactions. It suggests extending the method beyond single-screen UIs toward page-level interactions, data bindings, multi-page navigation, and executable product prototypes. It also notes that SPEC is especially strong once intent is clearer, whereas text prompts may remain useful for more open-ended early exploration. In that sense, SpecifyUI does not eliminate prompting; it repositions prompting inside a larger specification-driven workflow (Chen et al., 9 Sep 2025).