UI Transition Graph (UTG)

• UI Transition Graph (UTG) is a directed graph abstraction that represents user interface states and transitions induced by user actions or navigation mechanisms.
• UTGs vary in node granularity and edge semantics, enabling diverse applications such as code synthesis, automated testing, and security analysis.
• UTG construction employs static, dynamic, or hybrid methods to achieve high coverage and fidelity, which are critical for reliable GUI modeling.

Searching arXiv for papers on UI Transition Graphs and related GUI modeling. {"query":"\"UI Transition Graph\" mobile GUI agent arXiv", "max_results": 10, "sort_by": "submittedDate"} {"query":"(Zhou et al., 2024) OR (Ma et al., 7 Oct 2025) OR (Chai et al., 28 May 2026) OR (Guan et al., 30 Aug 2025) OR (Zhang et al., 2023) OR (Chen et al., 2022) OR (Wu et al., 2024) OR (Cui et al., 25 Jun 2026)", "max_results": 10, "sort_by": "relevance"} A UI Transition Graph (UTG) is a directed graph abstraction for modeling the reachable user-interface states of an application and the transitions induced by user actions or navigation mechanisms. Across recent work, the abstraction appears under closely related names—such as Page Transition Graph (PTG), Scene Transition Graph (SceneTG), and Activity Transition Graph (ATG)—but the common purpose is to encode UI structure at a level suitable for generation, exploration, planning, retrieval, testing, or security analysis. In the current literature, nodes may denote pages, screens, activities, fragments, scenes, or semantic UI states, while edges may denote unlabeled reachability, action-labeled navigation, or transitions annotated with conditions, triggers, and effects (Zhou et al., 2024, Ma et al., 7 Oct 2025, Zhang et al., 2023, Guan et al., 30 Aug 2025).

1. Formal scope and graph semantics

The literature does not define a single canonical UTG. Instead, several formally specified variants coexist, each tailored to a downstream task.

Formulation	Node semantics	Edge semantics
PTG in DeclarUI	page identifier, name, type, properties	transition identifier, source, target, condition, action
UTG in Agent+P / KG-RAG	UI states or screens	action-labeled directed transitions
UI-KOBE graph	semantic UI states with descriptions, embeddings, schemas, interactable elements	executable transitions, including self-loops
DeUEDroid scene graph	user-visible screens such as Activity or Fragment	possible UI transitions; node labels carry rich UI attributes

DeclarUI formalizes the Page Transition Graph as $PTG=(N,E)$, where each node $n_i=(id_i, name_i, type_i, properties_i)$ and each edge $e_j=(id_j, ns_j, nt_j, condition_j, action_j)$. This formulation explicitly separates page metadata from trigger semantics and framework-level navigation behavior, so that a multimodal model can reason over a structured schema before emitting code (Zhou et al., 2024).

Agent+P defines a UTG as $UTG=(S,E,L)$, where $S$ is the finite set of UI states, $E\subseteq S\times S$ is the set of directed edges, and $L:E\to\mathcal A$ assigns each edge an action $a=(w,e)$ composed of widget and event. The same paper also gives the equivalent edge-bundled form $E\subseteq S\times\mathcal A\times S$, which makes the action label part of the transition relation itself (Ma et al., 7 Oct 2025). KG-RAG adopts a closely related formalism, $G=(S,A,E,s_0)$ with $n_i=(id_i, name_i, type_i, properties_i)$0, together with the partial transition function $n_i=(id_i, name_i, type_i, properties_i)$1 and an adjacency tensor $n_i=(id_i, name_i, type_i, properties_i)$2 (Guan et al., 30 Aug 2025).

UI-KOBE uses $n_i=(id_i, name_i, type_i, properties_i)$3, but enriches both nodes and edges with runtime-oriented attributes. Each node corresponds to a semantic UI state rather than a raw screenshot and stores an identifier, a page description, an embedding, a state schema $n_i=(id_i, name_i, type_i, properties_i)$4, and a list of interactable elements. Each edge stores grounded action JSON, an instruction template, an optional schema delta $n_i=(id_i, name_i, type_i, properties_i)$5, and an optional target observation. Self-loops represent actions that preserve the same UI template while changing state schema, such as typing text or toggling a setting (Chai et al., 28 May 2026).

DeUEDroid defines a UTG, also called a scene graph, as $n_i=(id_i, name_i, type_i, properties_i)$6, where $n_i=(id_i, name_i, type_i, properties_i)$7 is a finite set of user-visible screens, $n_i=(id_i, name_i, type_i, properties_i)$8 is the set of possible UI transitions, and $n_i=(id_i, name_i, type_i, properties_i)$9 maps each node into a rich UI-attribute space containing native widget types and layouts, event listeners, WebView “network imprint” tokens, and XML-versus-dynamic widget statistics. In this formulation, edge labels are not modeled separately; the existence of $e_j=(id_j, ns_j, nt_j, condition_j, action_j)$0 already encodes the presence of some transition mechanism (Chen et al., 2022).

A plausible implication is that “UTG” functions less as a single data structure than as a family of graph abstractions whose exact node and edge semantics are selected according to the intended use: navigation synthesis, design-to-code translation, lightweight on-device inference, pre-training, or static security analysis.

2. Granularity of state representation

A central issue in UTG research is the granularity of the node set. Different systems choose different state abstractions, and this choice materially affects transition coverage, graph size, and the kinds of reasoning the graph can support.

SceneDroid argues that previously extracted transition graphs such as ATG, WTG, and STG have low transition coverage and coarse-grained granularity. Its SceneTG models a scene as any structurally distinct UI page, including a full Activity, a Fragment swap, a Drawer opening, a Dialog, a Menu, a Spinner popup, and even a hierarchy-changing state caused by CheckBox or Switch toggling. A transition is written $e_j=(id_j, ns_j, nt_j, condition_j, action_j)$1, with explicit event and component labels. This is finer grained than ATG nodes as Activities, WTG nodes as Window instances, and STG nodes as pseudo-screens inferred statically (Zhang et al., 2023).

At the opposite end, ATGBuilder keeps the abstraction at the activity level. It defines an Activity Transition Graph as $e_j=(id_j, ns_j, nt_j, condition_j, action_j)$2 with nodes equal to activities and edges equal to directed transitions between distinct activities; self-loops are excluded because they do not represent inter-activity navigation. This reduction supports seed-supervised link prediction over a comparatively compact graph, but omits intra-activity UI changes by construction (Cui et al., 25 Jun 2026).

MobileVLM occupies another point in the design space. For each app in Mobile3M, it defines a UTG $e_j=(id_j, ns_j, nt_j, condition_j, action_j)$3 whose nodes are unique UI pages represented by screenshot-image and parsed XML tuples $e_j=(id_j, ns_j, nt_j, condition_j, action_j)$4, and whose edges carry action labels from the action space $e_j=(id_j, ns_j, nt_j, condition_j, action_j)$5. Here the node is neither a purely semantic page role nor a coarse activity; it is a unique page identified from observed screenshot-plus-XML data (Wu et al., 2024).

DeUEDroid uses nodes corresponding to a single user-visible screen—i.e. an Activity or Fragment—while SceneDroid separates scenes by layout-hash identifiers derived from screen-dump structure (Chen et al., 2022, Zhang et al., 2023). UI-KOBE explicitly rejects raw screenshots as nodes and instead uses semantic UI states such as “Home page” or “Search page,” with cached embeddings and state schemas for retrieval and action selection (Chai et al., 28 May 2026).

This variation clarifies a common misconception: a UTG is not necessarily a graph of screenshots. In several systems, screenshots are evidence for node identification, but the node itself is a semantic or structural state abstraction.

3. Graph construction methodologies

UTGs are constructed through static analysis, dynamic exploration, hybrid procedures, or perception-driven inference, depending on the information source and target application.

DeUEDroid builds UTGs statically. Its Scene-Graph Builder extracts native widgets from layout XML and dynamic widget creations, then derives WebView “network imprint” tokens via bounded backward-only taint analysis over constant-derived Java strings. It constructs the Activity/Fragment Transition Graph by traversing a call graph built with Soot + FlowDroid, resolving explicit transition APIs, system callbacks such as onKeyDown, Fragment callers, and Navigation-Graph XML plus NavController.navigate(...) calls (Chen et al., 2022).

Agent+P uses a hybrid static+dynamic procedure. Off-line, ICCBot and FlowDroid parse bytecode and the AndroidManifest to extract Activity–Activity transitions such as startActivity calls. On-line, the UI Explorer observes transitions $e_j=(id_j, ns_j, nt_j, condition_j, action_j)$6 and updates the current graph by adding nodes, adding edges, or updating edge labels. The paper states that this keeps the UTG both broad via static analysis and precise via actual observed transitions (Ma et al., 7 Oct 2025).

SceneDroid constructs SceneTG through dynamic “smart exploration.” Its SmartDynamicAnalysis pipeline combines state fuzzing, exhaustive exploration, and indirect launching. State fuzzing enumerates values for non-transitive widgets such as EditText, CheckBox, and Switch; if an activity has $e_j=(id_j, ns_j, nt_j, condition_j, action_j)$7 such components, SceneDroid explores up to $e_j=(id_j, ns_j, nt_j, condition_j, action_j)$8 initial states. Exhaustive exploration performs depth-first clicks on every clickable=true view in the current scene and records a new scene whenever the post-click layout-hash—the MD5 of the breadth-first sequence of (resource-id, class, package) triples—differs. Indirect launching uses already discovered ATG predecessors to reach activities that cannot be launched directly (Zhang et al., 2023).

KG-RAG adapts and extends DroidBot into xTester. Starting from the launch state $e_j=(id_j, ns_j, nt_j, condition_j, action_j)$9, the extractor parses the UI tree, enumerates clickable, swipable, and input elements, executes possible actions, detects new states, adds edges $UTG=(S,E,L)$0, and backtracks using hardware back or explicit undo. The reported time budgets are 1 hour for simple English apps in DroidTask and 8 hours for 30 complex Chinese apps in KG-Android-Bench and KG-Harmony-Bench (Guan et al., 30 Aug 2025).

UI-KOBE constructs an app knowledge graph through an iterative Observe–Identify–Plan–Act loop. At each step it takes a screenshot, describes and parses the UI, embeds the result, matches it to an existing node using cosine similarity and screenshot verification, chooses an unexplored action, executes it, and then adds the resulting edge. Post-hoc auditing performs node merging, edge normalization, and coverage-driven re-exploration (Chai et al., 28 May 2026).

DeclarUI derives a PTG from static designs rather than a running app. Grounding DINO zero-shot object detection proposes bounding boxes, Segment Anything Model refines them into pixel-accurate masks, and an LLM such as Claude-3.5 Sonnet receives the formal PTG schema together with page images and component masks to infer component types and likely behaviors, returning structured JSON containing nodes and edges (Zhou et al., 2024).

MobileVLM’s Mobile3M corpus is collected from real-world transition actions by breadth-first random-walk from the home screen using click, scroll, and input. New pages are pruned by retrieving the top-5 most similar XMLs via BM25 and checking whether ElementDiff\<5 and PixelDiff\<30%; if both hold for an existing node, the crawler reuses that node and only adds a new directed edge (Wu et al., 2024).

4. Validation, completeness, and coverage metrics

Because UTGs are used as operational models rather than merely descriptive graphs, the literature places heavy emphasis on completeness, fidelity, and the cost of extraction.

DeclarUI formalizes PTG coverage through the PTG Coverage Rate (PCR),

$UTG=(S,E,L)$1

Here, UTG_edges are transitions discovered by running or statically analyzing the generated app, for example via DroidBot’s UI exploration. In React Native, DeclarUI with Claude-3.5 achieves PCR = 96.8%; in Flutter it reaches 99.4%, and in ArkUI 95.9%. The reported compilation success rate is 98.0% for React Native, 92.0% for Flutter, and 86.0% for ArkUI. The same evaluation reports SSIM = 0.68 and CLIP = 0.88 for React Native, and notes that unassisted baselines achieved as little as 1.4% or 43.3% coverage (Zhou et al., 2024).

SceneDroid evaluates GUI modeling by number of Activities reached, transition pairs extracted, and scenes discovered. On 100 real-world apps, its averages are 29.1 Activities, 30.25 transitions, and 22.93 scenes, compared with 18.4, 12.03, and 13.83 for StoryDistiller. Relative to the best prior tool, SceneDroid reports +168.74 % in transition-pair coverage and +162.42 % in scene extraction. On ten self-developed apps, it recovered all declared Activities, all scenes, and all transition pairs in under 30 minutes each (Zhang et al., 2023).

KG-RAG quantifies extraction cost and graph scale. Across complex Chinese apps, the average UTG size is |S| ≈ 150 distinct screens and |E| ≈ 300 labeled transitions. Its cost ablation on Weibo shows SR (%) rising from 40.0 at 0 hours to 70.0 at 4 hours, DA (%) from 60.0 to 80.0, and average steps AS falling from 6.25 to 4.50; from 4 to 8 hours, SR and DA saturate while AS continues to decline from 4.50 to 3.75. The authors identify ~4 h per complex app as a practical operating point (Guan et al., 30 Aug 2025).

UI-KOBE reports compact on-device graphs with typical values |V|≈54 and |E|≈226 per app, with memory O(|V|(d + |I|) + |E|) and the claim that the full index plus graph metadata fits on-device in a few MBs. Retrieval uses an ANN index with ∼O(log|V|) or sublinear query cost and O(1) verification per candidate with a small constant k (Chai et al., 28 May 2026).

These measurements indicate that UTG quality is evaluated along at least three axes: structural completeness, operational correctness, and extraction or storage cost. A plausible implication is that no single metric is sufficient across tasks: a graph that is adequate for shortest-path planning may still be insufficient for faithful code generation, and a graph suitable for lightweight retrieval may omit semantically rare but behaviorally critical transitions.

5. Algorithmic roles in generation, planning, retrieval, and learning

Once constructed, UTGs are used as algorithmic substrates for several distinct classes of methods.

In DeclarUI, the PTG/UTG mediates between design understanding and code synthesis. After a first-draft declarative codebase is produced, a static analysis tool extracts realized navigation calls and reconstructs a UTG. Navigation consistency validation compares UTG_edges against the original PTG_edges and feeds missing transitions back to the LLM. Compile-time error correction then packages compiler errors and relevant code snippets into prompts for iterative repair. The case study on com.amazon.mShop.android reports that a baseline using Claude-3.5 alone omits the bottom navigation bar and fails to wire the “Orders” button, whereas DeclarUI reproduces all components and its UTG shows full transitions from both Profile and Account pages into Orders (Zhou et al., 2024).

Agent+P turns UI automation into a pathfinding problem on the UTG. The planning translation defines a fluent at(u), uses connected ?from ?to for edges, and maps each transition into a PDDL navigate action. With uniform action cost $UTG=(S,E,L)$2, the objective becomes the unweighted shortest-path problem on a directed graph. Agent+P uses Fast Downward with A* search, and the paper states that the planner returns an optimal sequence of navigate actions. On AndroidWorld, the reported success-rate gains are +12 percentage points for DroidRun, +2 pp for MobileUse, and +14 pp for T3A. On the example task “Go to Privacy Policy,” actions per task decrease from 6.10 to 3.80 and execution time from 19.9 s to 16.9 s (Ma et al., 7 Oct 2025).

KG-RAG uses UTGs as the basis of a retrieval-augmented navigation memory. Offline, an intent-guided LLM search generates intent–trajectory pairs and embeds each pair into a vector store. Online, cosine similarity retrieves the most relevant trajectories for a user instruction. The reported aggregate results are a 75.8% success rate with 8.9% improvement over AutoDroid, 84.6% decision accuracy with 8.1% improvement, and a reduction in average task steps from 4.5 to 4.1. The framework is also reported to transfer to web and desktop, with +40% SR on Weibo-web and +20% on QQ Music-desktop (Guan et al., 30 Aug 2025).

UI-KOBE uses the graph as external guidance for a lightweight GUI agent. At runtime, the agent identifies the current node, retrieves the outgoing edges, and chooses among four classes of options: task completion, self-loop edges with schema deltas, neighboring transitions with instruction templates, or free-action fallback. Because the graph stores app-specific behaviors explicitly, runtime planning is reduced to choosing among a limited local option set rather than solving the task end-to-end from screenshots alone (Chai et al., 28 May 2026).

MobileVLM uses UTG edges as supervision for inter-UI pre-training rather than as an online planner. In Stage 2, positive examples are sampled uniformly from $UTG=(S,E,L)$3, negatives from non-neighbor pages in the same UTG, and the model predicts the action $UTG=(S,E,L)$4 that transforms one page into another. The Mobile3M corpus contains 3,098,786 unique pages and 20,138,332 edges across 49 apps. In the ablation, removing Stage 2 causes Auto-UI overall accuracy to drop from 74.94% to 73.05% and Self-Navigation IoU from 48.49% to 35.89%, implying that exposure to real page-transition edges improves inter-UI reasoning (Wu et al., 2024).

DeUEDroid and ATGBuilder show that UTGs also serve as learned representations and supervision targets. DeUEDroid embeds scene graphs with a self-supervised graph auto-encoder and reports 98.22% detection F1-score and 98.97% classification accuracy for UEware detection, while ATGBuilder treats ATG construction as seed-supervised link prediction with GIN-based node representations, a gravity-based decoder, and auxiliary widget-attribute reconstruction; on a 98-app benchmark, AtgBuilder-FT @t* reaches Prec=0.7148, Rec=0.9363, and F1=0.8101, and its predicted ATGs improve the activity and transition coverage of Monkey, APE, and FastBot2 (Chen et al., 2022, Cui et al., 25 Jun 2026).

6. Limitations, misconceptions, and open directions

The main limitation recurring across the literature is incompleteness. Pure static UTGs may miss reflection-only transitions or dynamically loaded UIs, and WebView imprints are described as an approximation rather than a substitute for full dynamic inspection (Chen et al., 2022). Dynamic approaches have the complementary problem: limited exploration depth can miss rare transitions, and fallback planning or re-exploration is needed to cover these gaps (Chai et al., 28 May 2026, Guan et al., 30 Aug 2025).

Another limitation is version sensitivity. UI-KOBE notes that major UI changes may invalidate nodes and edges, requiring incremental graph repair or partial re-exploration (Chai et al., 28 May 2026). This issue is especially salient for on-device or precomputed graph deployment, where stale graph knowledge can degrade runtime guidance.

A further misconception is that high transition coverage alone implies complete behavioral understanding. DeclarUI couples PTG coverage with visual similarity, compilation success, and iterative compiler-driven correction, while SceneDroid distinguishes among Activities, transitions, and scenes as separate metrics of recovered UI structure (Zhou et al., 2024, Zhang et al., 2023). The literature therefore treats coverage as necessary but not sufficient.

Research directions in the source material remain heterogeneous but consistent with these limitations. SceneDroid identifies better support for React-Native and Flutter apps, richer system-event injection, and stronger EditText fuzzing via symbolic or string-constraint methods (Zhang et al., 2023). UI-KOBE points to fully local embedding solutions and broader GUI domains beyond Android mobile apps (Chai et al., 28 May 2026). ATGBuilder addresses noisy seed supervision through threshold tuning, seed-consistency, and bootstrapping, and reports that even with up to 100% label-flip noise, F1 degrades by less than 10 bps (Cui et al., 25 Jun 2026).

This suggests that the contemporary trajectory of UTG research is not merely toward larger graphs, but toward graphs that are more semantically typed, more auditable, and more tightly integrated with downstream decision procedures. In practice, the UTG has become a unifying intermediate representation for mobile UI systems: a graph that supports code generation, benchmark construction, exploration guidance, symbolic planning, retrieval-augmented action selection, pre-training, and security-oriented app classification.