PG-Agent: Graph-Based GUI Agent Framework

• PG-Agent is a GUI framework that represents user interactions as page graphs, capturing structured page transitions rather than isolated sequential episodes.
• It employs an automated pipeline using MLLM-based summarization to detect page jumps, perform node similarity checks, and construct semantically compact graphs.
• The framework integrates retrieval-augmented generation with a specialized multi-agent execution system to enhance planning, decision-making, and generalization across mobile and web benchmarks.

Searching arXiv for the PG-Agent paper and closely related GUI-agent work to ground the article. First locating the exact PG-Agent paper by arXiv ID and title. PG-Agent is a graphical user interface agent framework that represents prior GUI experience as a page graph rather than as isolated sequential episodes. In this formulation, GUI environments are treated as structured spaces in which pages are connected by actions, and prior trajectories are interpreted as sampled paths through that latent graph. PG-Agent combines an automated pipeline that converts episodes into page graphs with a Retrieval-Augmented Generation mechanism that retrieves perception guidelines from those graphs, and a tailored multi-agent execution framework that injects those guidelines into planning and decision-making so that the agent can generalize to unseen scenarios (Chen et al., 27 Aug 2025).

1. Concept and motivation

PG-Agent is motivated by a limitation of prior GUI agents that use sequential episodes of multi-step operations as their primary prior knowledge. A sequential episode provides only a single path through a GUI and does not explicitly encode that one page may support multiple outgoing actions, that multiple episodes may share intermediate pages, or that new task solutions may require recombining knowledge across trajectories. The framework therefore treats GUIs as environments with an implicit graph structure and reconstructs that structure explicitly as a page graph (Chen et al., 27 Aug 2025).

The central claim is that sequential episodes fail to capture the complex transition relationships between pages, which in turn weakens environmental perception and limits generalization to unseen scenarios. By contrast, a page graph encodes what pages exist, how they are connected, and what action sequences and task contexts are associated with those transitions. This suggests that page graphs serve not merely as memory, but as a structured prior over GUI dynamics (Chen et al., 27 Aug 2025).

The method is designed to operate from visual information only. It does not rely on DOM, HTML, or other structural metadata. This is an important design decision because it makes the framework applicable to both mobile and web interfaces under a unified visual-interaction setting (Chen et al., 27 Aug 2025).

2. Page graph representation and construction pipeline

The page graph is denoted as $\mathcal{G} = (\mathcal{N}, \mathcal{V})$, although the construction procedure later refers to edges as $\mathcal{E}$. Conceptually, nodes represent unique GUI pages and edges represent page transitions caused by actions. Each edge stores an action queue $Q$ and the original task description $T$ from the episode in which the transition occurred (Chen et al., 27 Aug 2025).

A key representational choice is that PG-Agent does not create an edge for every atomic action. Instead, consecutive in-page operations are merged into an action queue and attached to the next page-transition edge. This makes the graph more semantically compact: an edge can encode a meaningful transition recipe rather than a single low-level event (Chen et al., 27 Aug 2025).

For an action tuple in an episode,

$(I_{before}, A, I_{after}),$

$I_{before}$ is the screenshot before the action, $A$ is the action, and $I_{after}$ is the screenshot after the action. Graph construction then proceeds in three stages: page jump determination, node similarity check, and graph update (Chen et al., 27 Aug 2025).

Page jump determination

Because coordinate-based actions are not interpretable without the image, the framework first summarizes an action into natural language:

$$S_{action}=\textrm{MLLM}(I_{before},A,\mathbb{P}_{action}).$$

Here, $S_{action}$ is the action summary and $\mathcal{E}$0 is the prompt template for summarization (Chen et al., 27 Aug 2025).

The system then determines whether the action causes a page transition:

$\mathcal{E}$1

where $\mathcal{E}$2. If the answer is negative, the action is treated as an in-page operation and its summary is appended to the action queue $\mathcal{E}$3 rather than creating a new node or edge (Chen et al., 27 Aug 2025).

Node similarity check

When a page jump is detected, the method summarizes the destination page:

$\mathcal{E}$4

This summary captures the page’s function and salient components (Chen et al., 27 Aug 2025).

The framework then performs a dual-level similarity check. First, it retrieves the top-$\mathcal{E}$5 most similar existing node summaries:

$\mathcal{E}$6

where $\mathcal{E}$7. The paper states that summaries are vectorized using BGE-M3 and retrieved with FAISS (Chen et al., 27 Aug 2025).

Second, an MLLM selects the most similar candidate:

$\mathcal{E}$8

and then checks image-level dissimilarity against the selected node’s original screenshot:

$\mathcal{E}$9

If $Q$0, a new node is created; otherwise the existing node is reused (Chen et al., 27 Aug 2025).

This semantic-retrieval-then-pixel-verification pattern is one of the practical contributions of the framework. It suggests that GUI pages should be canonicalized at the level of functional identity rather than raw screenshot uniqueness, but that pure text similarity alone is insufficient for safe node merging (Chen et al., 27 Aug 2025).

Page graph update

If the page is new, the node is defined as

$Q$1

where $Q$2 is the page summary and $Q$3 is the image location. The image location is retained only for later similarity checking; the graph does not store raw pixel data as part of the node representation (Chen et al., 27 Aug 2025).

After adding the current action summary to the queue, the edge is formed as

$Q$4

The graph update is then

$Q$5

or, if an existing node is reused,

$Q$6

Thus, a node denotes a canonicalized page, while an edge denotes a directed transition labeled by an action queue and a task context (Chen et al., 27 Aug 2025).

3. Retrieval-augmented graph reasoning

The page graph is used online through a tailored RAG procedure that retrieves perception guidelines rather than raw demonstrations. At runtime, for the current screen $Q$7, the framework first generates a summary

$Q$8

then retrieves similar nodes from the graph:

$Q$9

This retrieval again uses BGE-M3 and FAISS (Chen et al., 27 Aug 2025).

For each outgoing edge $T$0 from each retrieved node, the system runs breadth-first search up to depth $T$1:

$T$2

where $T$3 is the set of achievable tasks collected from nearby graph structure (Chen et al., 27 Aug 2025).

The resulting guideline set is

$T$4

where each $T$5 is an action queue on an outgoing edge and each $T$6 is the downstream task set reachable from that edge. In effect, the agent retrieves guidance of the form: from a page like this, a transition realized by $T$7 has been associated with tasks in $T$8 (Chen et al., 27 Aug 2025).

This differs from flat trajectory retrieval. The retrieved object is not a single nearest demonstration but a graph-local view of available transitions and downstream affordances. A plausible implication is that the framework improves compositional generalization because the graph permits recombining transition knowledge across previously separate episodes (Chen et al., 27 Aug 2025).

The reported retrieval hyperparameters are $T$9 retrieved nodes and BFS depth $(I_{before}, A, I_{after}),$0. The maximum number of retrieved guidelines is 20 for AITW and GUI Odyssey, and 10 for Mind2Web (Chen et al., 27 Aug 2025).

4. Multi-agent execution framework

PG-Agent couples graph retrieval with a specialized multi-agent workflow. The paper motivates this design partly by the concern that long contexts can degrade reasoning through lost-in-the-middle effects, and that specialized agents with narrower roles can process relevant information more effectively (Chen et al., 27 Aug 2025).

The execution framework contains four agents: Global Planning Agent, Observation Agent, Sub-Task Planning Agent, and Decision Agent (Chen et al., 27 Aug 2025).

Global Planning Agent

The global planner decomposes the user’s overall task $(I_{before}, A, I_{after}),$1 into high-level sub-tasks:

$(I_{before}, A, I_{after}),$2

Its role is to structure the overall objective into manageable stages conditioned on the current screen (Chen et al., 27 Aug 2025).

Observation Agent

The observation agent produces textual perception from the current screen, the global task, and the interaction history:

$(I_{before}, A, I_{after}),$3

where

$(I_{before}, A, I_{after}),$4

This agent isolates low-level page understanding from later task reasoning (Chen et al., 27 Aug 2025).

Sub-Task Planning Agent

The sub-task planner uses the current screen, observation output, global plan, retrieved guidelines, and history:

$(I_{before}, A, I_{after}),$5

This is the main stage at which graph-derived guidance and task decomposition interact (Chen et al., 27 Aug 2025).

Decision Agent

The decision agent selects the concrete action:

$(I_{before}, A, I_{after}),$6

It integrates perceptual information, the current sub-task, retrieved graph guidance, and history to produce the final operation (Chen et al., 27 Aug 2025).

The full stepwise loop is: retrieve graph guidance, plan globally, observe locally, choose a current sub-task, decide an action, execute, and repeat until the step limit or completion condition is reached. The framework is inference-oriented; the paper does not introduce a dedicated training loss or reinforcement-learning objective for PG-Agent itself (Chen et al., 27 Aug 2025).

5. Experimental evaluation

PG-Agent is evaluated on three GUI benchmarks spanning mobile and web settings: AITW, Mind2Web, and GUI Odyssey (Chen et al., 27 Aug 2025).

AITW

For page graph construction on AITW, the authors randomly sampled one tenth of training episodes. The resulting sampled graph contains 231 episodes, 1983 images, 700 nodes, and 940 edges (Chen et al., 27 Aug 2025).

On AITW, the evaluation metric is action matching score. PG-Agent achieves an overall score of 59.5, exceeding GPT-4V (50.5), Qwen2.5-VL-72B (48.1), and OmniParser (57.5). It is strongest in General (51.9), Install (62.4), GoogleApps (65.0), and WebShopping (53.7), while underperforming OmniParser and GPT-4V on Single (Chen et al., 27 Aug 2025).

The GoogleApps result is especially notable: PG-Agent scores 65.0 versus OmniParser’s 51.6. The paper interprets this as evidence that graph-structured prior knowledge provides action guidance when relevant page-transition experience is available (Chen et al., 27 Aug 2025).

Mind2Web

For Mind2Web graph construction, the sampled training data contain 252 episodes, 1827 images, 416 nodes, and 365 edges, drawn from Entertainment, Travel, and Shopping. The Service and Info scenarios appear only in cross-domain testing and therefore have no corresponding page graphs (Chen et al., 27 Aug 2025).

Mind2Web reports Element Accuracy, Operation F1, and Step Success Rate. PG-Agent achieves the best Element Accuracy and Step Success Rate in all three settings. In Cross-Task it records 59.0 Ele.Acc and 52.9 Step SR; in Cross-Website 57.3 Ele.Acc and 48.7 Step SR; in Cross-Domain 60.2 Ele.Acc and 53.3 Step SR (Chen et al., 27 Aug 2025).

The Cross-Domain result is especially important because the evaluated domains lack matched page graphs. The paper uses this to argue that page-graph retrieval captures transferable navigation knowledge rather than only domain-specific memorization (Chen et al., 27 Aug 2025).

GUI Odyssey

For GUI Odyssey, the sampled graph contains 116 episodes, 1881 images, 615 nodes, and 929 edges across Tool, Information, Shopping, Media, Social, and Multi-Apps scenarios (Chen et al., 27 Aug 2025).

PG-Agent achieves the best overall result, 47.7, exceeding Qwen2.5-VL-72B (42.4) and GPT-4V (19.0). It wins Tool (48.6), Information (61.5), Shopping (47.2), Media (35.5), and Social (46.9), but trails Qwen2.5-VL-72B on Multi-Apps (Chen et al., 27 Aug 2025).

Limited-data graph construction

A practical claim of the paper is that page graphs remain useful even when built from limited demonstrations. On both AITW and Mind2Web, graphs built from random sampling of episodes perform competitively with graphs built from full episodes, and sometimes better. This supports the claim that useful page-transition structure can be recovered from relatively small trajectory sets (Chen et al., 27 Aug 2025).

6. Ablations, interpretation, and limitations

The ablation studies examine how the retrieved guidelines affect different agents in the multi-agent framework. On AITW, removing guidelines reduces overall performance from 59.5 to 58.3. Injecting guidelines only into the Decision Agent gives 59.0, and only into the Sub-Task Planning Agent gives 58.5, whereas full PG-Agent remains strongest overall (Chen et al., 27 Aug 2025).

On Mind2Web, the effect is similarly positive but more split-dependent. Full PG-Agent is strongest overall, though the relative advantage of injecting guidelines into the Decision Agent or the Sub-Task Planning Agent varies across Cross-Task, Cross-Website, and Cross-Domain settings (Chen et al., 27 Aug 2025). This suggests that guideline utility may depend on benchmark-specific interaction structure and on how the backbone model prefers to organize action selection.

A finer-grained Mind2Web analysis shows that guidelines chiefly improve SELECT and TYPE operations. The paper notes that apparent decreases in CLICK performance can reflect dataset-label ambiguity, because some interactions that are effectively selection operations may be labeled as two consecutive clicks (Chen et al., 27 Aug 2025).

The paper also highlights several limitations or borderline cases. In AITW’s Single scenario, some failures arise from ambiguity in completion criteria: PG-Agent may continue an interaction because it interprets the user goal more literally than the benchmark does. In GUI Odyssey’s Multi-Apps scenario, the graph-based prior is less dominant, which suggests that graph retrieval may become less effective when tasks span multiple applications and the reusable local transition structure is weaker (Chen et al., 27 Aug 2025).

The paper does not provide explicit prompt templates, an exact retrieval scoring formula, or a runtime cost analysis. This suggests that part of the framework’s behavior depends on prompt engineering and implementation details that are not fully specified in the main text (Chen et al., 27 Aug 2025).

7. Significance within GUI-agent research

PG-Agent’s main contribution is a shift in prior-knowledge representation: from storing trajectories as isolated chains to storing experience as a graph of reusable page transitions. Its graph construction pipeline, graph-structured RAG, and specialized multi-agent inference architecture are all organized around that change (Chen et al., 27 Aug 2025).

The framework is therefore best understood not merely as a new prompting strategy, but as a structural view of GUI environments. It encodes pages as canonical nodes, transitions as directed edges labeled by action queues and task contexts, and retrieves graph-local transition affordances as perception guidelines for downstream reasoning (Chen et al., 27 Aug 2025).

This suggests a broader methodological point: for GUI agents, prior experience may be more valuable when represented as environment structure rather than as raw demonstrations. PG-Agent operationalizes that claim and reports improved performance across mobile and web benchmarks, including settings with limited graph-construction data and cross-domain evaluation without matched page graphs (Chen et al., 27 Aug 2025).