UIPro: Dual AI Systems for UI Automation

• UIPro is a designation referring to two distinct AI systems focused on automating user interfaces through production-ready code generation and unified GUI action prediction.
• In enterprise applications, UIPro uses a Figma-embedded declarative grammar and a specialized multi-agent workflow to transform design annotations into secure, compliant, production-level code.
• The GUI-agent version leverages vision-language grounding with unified JSON action spaces to accurately predict and execute tasks across web, mobile, and desktop platforms.

UIPro is a 2025 designation applied to two distinct AI systems concerned with user interfaces. In one usage, UIPro denotes a modern enterprise embodiment of the AI4UI vision: a specialized multi-agent framework that converts annotated Figma designs into production-ready front-end code with targeted human-in-the-loop involvement at design and post-processing stages (Ganesaraja et al., 5 Dec 2025). In the other, UIPro denotes a generalist vision-language GUI agent trained on large-scale multi-platform GUI data with a unified action space, with the goal of perceiving and operating web, mobile, and related graphical interfaces (Li et al., 22 Sep 2025). The shared name reflects a common concern with UI automation, but the two systems address different layers of the stack: software production in the former case and GUI understanding and action prediction in the latter.

1. Terminological scope and disambiguation

The term UIPro refers to two separate research artifacts rather than a single unified framework. Within the AI4UI work, UIPro is presented as an enterprise pipeline for autonomous front-end development from “pixel to production,” emphasizing production readiness, compliance, security, maintainability, and enterprise workflow integration (Ganesaraja et al., 5 Dec 2025). In the GUI-agent work, UIPro is a multimodal agent that “sees” and “acts” on interfaces by combining large-scale GUI grounding pre-training with action prediction over a unified action space (Li et al., 22 Sep 2025).

The distinction is substantive rather than nominal. The enterprise UIPro begins from design artifacts and generates code artifacts, RFC-linked pull requests, and enterprise package integrations. The GUI-agent UIPro begins from screenshots, instructions, and action history, and outputs the next GUI action as JSON. This suggests that references to UIPro require immediate contextual disambiguation by task setting, model objective, and source paper.

Usage	Primary objective	Core substrate
UIPro in AI4UI	Enterprise-grade front-end generation	Figma grammar, specialized agents, knowledge graph, secure package strategy
UIPro as GUI agent	Generalist GUI interaction	GUI grounding corpus, unified action space, VLM fine-tuning

2. Enterprise UIPro: grammar-driven design capture

In the AI4UI setting, UIPro’s design interpretation centers on a lightweight, declarative grammar embedded directly into Figma layers and components, using structured JSON-like tags rather than ad-hoc text labels or free-form comments (Ganesaraja et al., 5 Dec 2025). A button component may carry an @UIProp block specifying identifiers, types, default values, validation rules, and actions; screen-state variants are marked via @State nodes specifying transitions and conditional flows. The semantics are explicit: every frame with an @Screen tag is treated as a top-level route, every @Input annotation yields a controlled field with type inference such as email, date, or number, and every @Transition block produces a finite-state machine edge in the component lifecycle.

The paper characterizes this grammar as mapping $1{:}1$ to internal UI schema definitions. By enforcing a tight syntax in which keys must match a pre-registered registry of UI types, and fixed semantic fields for validation, asset reference, and accessibility notes, the Parser agent can deterministically extract a complete “contract” of UI requirements: layout, state graph, data bindings, validation rules, and edge cases. The intended consequence is the elimination of clarification loops between design and implementation during the autonomous middle stage of the workflow.

This design places requirements formalization upstream, inside the design artifact itself. A plausible implication is that UIPro treats Figma not merely as a visual prototyping medium but as a structured specification carrier for downstream code generation and compliance-aware implementation.

3. Specialized agents, knowledge graph, and secure packaging

Once the Figma grammar is in place, UIPro instantiates a specialized multi-agent workflow in which each agent has a narrow remit (Ganesaraja et al., 5 Dec 2025). The Grammar Interpreter reads Figma’s AST, validates annotations, and emits a canonical UI schema. The Designer Agent verifies visual fidelity constraints, generates style-token palettes, and ensures adherence to brand guidelines. The Orchestrator ingests change requests as RFC artifacts and sequences tasks. The Planner maps the schema into a module-level implementation plan, including framework templates, state-management strategy, and UI-library choices. The Implementer writes scaffolded code, stubs BFF endpoints, and embeds security wrappers. The Reviewer runs static analysis, style-guide checks, accessibility audits, and unit-test scaffolding while maintaining a running delta of outstanding errors. The Integrator links generated code into the enterprise monorepo, resolves versions via locked manifests, and submits pull requests for final human approval.

This workflow is explicitly change-oriented. Every UI change is captured as an RFC artifact containing scope, dependencies, and acceptance criteria. The Orchestrator assigns subtasks to Planner and Implementer, after which Reviewer and Integrator execute their passes before the task is marked “Done.” The paper presents this choreography as auditable and reversible, with the purpose of preventing ad-hoc patches from bypassing the formal pipeline.

UIPro also maintains a domain-aware knowledge graph whose nodes represent screens, components, props, states, validations, API endpoints, and design tokens, while edges encode relations such as “uses,” “inherits,” “transitions to,” and “binds to.” An example given is a CustomerForm node linked to a TextInput component node via “uses,” to a zipCode prop node via “binds to,” and to a validateZIP rule node via “requires.” The Planner queries this graph to decide whether to reuse existing widgets or generate new ones, while also factoring in performance annotations and recent security audit flags. The stated objective is to avoid reinvention, preserve cross-screen consistency, and detect architectural drift early.

A further enterprise constraint is the handling of proprietary back-end utilities. UIPro addresses this through an abstract/package strategy consisting of a standalone UIPro.Generated package containing only framework-agnostic code, abstract interfaces for hooks to enterprise services, and a separate UIPro.EnterpriseExtensions package in the organization’s private registry implementing those interfaces with security-hardened logic, logging, and compliance checks. The paper states that this two-package approach satisfies reproducibility, because the generated package remains constant across runs, and confidentiality, because enterprise logic remains behind locked registries and private repositories. Expertise-driven architecture templates then codify best practices such as folder structures, code-splitting conventions, styling guidelines, accessibility-first markup, React hooks patterns, or Angular module layering.

4. Enterprise evaluation, readiness metrics, and deployment model

The AI4UI paper reports a head-to-head benchmark involving 500 screens and 2,000 change requests, in which UIPro achieved 97.24% platform compatibility, 87.10% compilation success, 86.98% security compliance, 78.00% feature implementation success, 73.50% code-review quality, and 73.36% UI/UX consistency (Ganesaraja et al., 5 Dec 2025). The metric definitions are specified concretely for several cases: platform compatibility is reported as $\frac{2917}{3000}\times100\% \approx 97.24\%$, compilation success as $\frac{522}{600}\times100\% \approx 87.10\%$, security compliance as $\frac{512}{589}\times100\% \approx 86.98\%$, and feature implementation success as $\frac{117}{150}\times100\% = 78.00\%$. Code-review quality is described as an average of cyclomatic complexity, style compliance, and test coverage scores, while UI/UX consistency is measured by pixel-accurate diff across renderings.

The paper also defines an overall readiness score as

$$\mathrm{OverallScore} = \sum_i w_i\,M_i,\quad \sum_i w_i = 1,$$

with weights tuned to enterprise priorities: security 25%, compilation 20%, feature completeness 20%, consistency 15%, and maintainability 20%. Under this weighted aggregation, UIPro is reported to achieve an overall readiness score above 85%.

In blind user-preference studies involving 200 expert evaluators, UIPro consistently ranked in the top three among leading commercial solutions. The same work states that AI4UI, operating asynchronously, generates thousands of validated UI screens in weeks rather than months. An illustrative deployment scenario is a customer-onboarding portal with twenty distinct user journeys. In that example, designers annotate Figma screens during initial mock-up; agents parse the grammar, populate the knowledge graph, and generate draft code in parallel; the Reviewer flags two minor style violations and one missing alt-text; security audits run automatically; and within two weeks all twenty journeys are live on staging, with 97% cross-browser parity, automated unit tests at 85% coverage, and zero high-severity vulnerabilities. The example is presented not as a prototype workflow but as production code governed by version-controlled RFCs and critical-juncture expert review.

A common misconception would be to treat the system as merely a prototyping assistant. The source text explicitly contrasts it with rapid prototyping systems and frames it as a production-oriented pipeline whose differentiators are compliance, maintainability, secure integration, and auditable change management.

5. GUI-agent UIPro: dataset, training objectives, and action unification

The second UIPro is a vision-language GUI agent designed to “see” and “act” on graphical user interfaces by unifying massive multi-platform data and heterogeneous action semantics into a single, end-to-end trainable model (Li et al., 22 Sep 2025). Its data pipeline begins with a comprehensive multi-platform GUI understanding corpus drawn from web pages rendered from the top-200 domains from Common Crawl, Android GUIs from virtual devices on Android Emulator, and public collections including RICO, MobileViews, WAE, WebUI, MultiUI, GUIEnv, SeeClick-Web, OmniAct, and MOTIF. The resulting corpus contains 2.5 M unique screenshots, 3.3 M clean elements, and 20.6 M GUI grounding tasks.

Each grounding task is a triplet $\langle I, R, y\rangle$, where $I$ is a GUI screenshot, $R$ is a referring expression from one of 13 types, and $y$ is an element location $\frac{2917}{3000}\times100\% \approx 97.24\%$0 or a text label. The referring expressions include element description, user intent, functional “why” description, pure OCR, icon class names, widget listing, GUI captioning, and GUI Q&A. The paper describes a seven-step denoising pipeline removing invalid or blank bounding boxes, oversized or undersize elements below 18 px, duplicates, and invisible text elements via Tesseract OCR, reducing raw-source noise rates of up to 29% to sub-5% in the final task collection.

Grounding pre-training uses standard multimodal maximum likelihood:

$\frac{2917}{3000}\times100\% \approx 97.24\%$1

Here, $\frac{2917}{3000}\times100\% \approx 97.24\%$2 is either a probability distribution over discrete grid cells for element grounding or the token-sequence probability for textual outputs such as captions and Q&A. No additional contrastive or margin losses are introduced; the model reads the GUI screenshot via a frozen ViT backbone $\frac{2917}{3000}\times100\% \approx 97.24\%$3, conditions on $\frac{2917}{3000}\times100\% \approx 97.24\%$4, and is trained either to point to $\frac{2917}{3000}\times100\% \approx 97.24\%$5 or generate the required text.

A central contribution is the unified action space. Because existing datasets encode GUI actions in heterogeneous formats, UIPro defines three scenario-specific spaces $\frac{2917}{3000}\times100\% \approx 97.24\%$6, each expressed as JSON objects and each a superset of the raw actions observed in its platform family:

$\frac{2917}{3000}\times100\% \approx 97.24\%$7

Examples include click(target_x,target_y), swipe(start_x,start_y; direction∈{up,down,left,right}; distance∈{short,medium,long}), input_text(text), long_press, drag, navigate_back/home/recent, wait, and status(goal_status,answer) for mobile; web replaces swipe with scroll(direction,distance) and adds actions such as hotkey, new_tab, and switch_tab; desktop further includes right_click and double_click. Raw actions are canonicalized into this space, for example AITW’s DUAL_POINT(start,end) to drag(start,end), AndroidControl’s scroll(dir) to scroll(dir,medium), and Mind2Web’s click(x,y) to click(x,y). The merged agent-task dataset then contains 380 K mobile steps from six sources and 145 K web steps from three sources in a single JSON-token format.

After grounding pre-training, the model is fine-tuned to predict the next action $\frac{2917}{3000}\times100\% \approx 97.24\%$8 given the current screenshot $\frac{2917}{3000}\times100\% \approx 97.24\%$9, instruction $\frac{522}{600}\times100\% \approx 87.10\%$0, and action history $\frac{522}{600}\times100\% \approx 87.10\%$1. Each sample is formatted as a prompt of the form “<image> Q: … H: … What is the next action?” with the label encoded as action JSON. The action prediction objective is cross-entropy over the JSON token sequence:

$\frac{522}{600}\times100\% \approx 87.10\%$2

No auxiliary RL or weighted multi-task scheme is used; all sources are simply concatenated, and the unified action space is presented as the mechanism ensuring consistency.

Architecturally, UIPro uses a high-resolution frozen ViT as vision encoder, a small MLP VL-projector into the LLM embedding space, and a language backbone. UIPro-SLiME builds on a 2 B-parameter SLiME-Gemma-2B base with maximum length 2048, while UIPro-Qwen2VL uses Qwen2-VL-7B-Instruct with maximum length 4096 and LoRA rank 128. Fusion and generation follow LLaVA-style cross-attention layers, with action JSON generated by the LLM head. Fine-tuning uses AdamW, learning rate $\frac{522}{600}\times100\% \approx 87.10\%$3, warm-up 3%, cosine decay, BFloat16, and ZeRO-2 over $\frac{522}{600}\times100\% \approx 87.10\%$4L20 GPUs.

6. Benchmarks, ablations, and limitations of the GUI agent

The GUI-agent UIPro is evaluated on mobile benchmarks including AITW, AndroidControl, and GUIAct-Smartphone, and on web benchmarks including GUIAct-Web and Mind2Web, using metrics such as Step Success Rate, action-type accuracy, and click-localization accuracy (Li et al., 22 Sep 2025). UIPro-Qwen2VL, the 7 B model, is reported to outperform equal-size or larger baselines including GPT-4V+OmniParser, SeeClick, OS-ATLAS, and MiniCPM-GUI. Reported Step SR values include 70.4% on AITW Overall versus OS-ATLAS at 63.1% and GPT-4V+OmniParser at 57.7%, 64.0% on AndroidControl High only versus OS-ATLAS at 62.4%, 69.1% on GUIAct-Web versus MiniCPM-GUI at 60.2%, and 48.4% on Mind2Web Cross-Task versus OS-ATLAS at 36.7%. UIPro-SLiME, the smaller 2–3 B model, also attains 68.0% on AITW Overall and 28.7% on Mind2Web Cross-Task.

Grounding evaluation spans ScreenSpot, ScreenSpot-v2, MOTIF, RefExp, VisualWebBench Element/Action grounding, and FuncPred. Selected results reported for UIPro-Qwen2VL are 58.8% on FuncPred versus OS-ATLAS at 52.1%, 82.5% on RefExp versus OS-ATLAS at 52.1%, and 94.9% on VisualWebBench Element Grounding versus OS-ATLAS at 82.6%. The paper interprets these results as evidence that large-scale GUI grounding improves downstream agent behavior.

The ablation analysis identifies three main drivers. First, pre-training scale yields monotonic gains in grounding accuracy and Step SR when moving from 0 M to 5.9 M to 20.6 M grounding tasks. Second, unified action space matters materially: mixing sources without unification reduces Step SR by 5–10 points across benchmarks, while the unified formulation improves both common actions such as click and swipe and rare actions such as wait. Third, denoising is consequential: pre-training on a 6.7 M subset with denoising yields 2–3% absolute grounding improvements across all six evaluated benchmarks.

The reported error patterns are “near-miss” localizations off by a few pixels, underperformance on long-tail actions such as drag and hotkey due to fewer examples, and the effect of strict evaluation protocols that penalize valid alternative actions. The paper argues that UIPro works because massive multi-task grounding pre-training yields robust element-level perception, which correlates strongly with downstream action success at approximately $\frac{522}{600}\times100\% \approx 87.10\%$5, and because unified action semantics enable cross-source knowledge transfer. At the same time, the limitations are explicit: long-horizon planning remains shallow because the policy is one-step greedy; rare gestures such as multi-finger drags and complex hotkeys are underrepresented; and desktop pre-training data is minimal, although fine-tuning on OmniAct-Desktop is said to show promising transfer.

These limitations distinguish the GUI-agent UIPro from the enterprise UIPro. The former is fundamentally an interface-control policy model and therefore constrained by perception, action taxonomy, and planning horizon. The latter is a software-generation and integration pipeline constrained by specification quality, enterprise architecture, and compliance processes. The shared name should not obscure that separation of problem classes.