RAG-GUI: Retrieval-Augmented Guidance

• RAG-GUI is a retrieval-augmented framework that injects external procedural and structural knowledge into GUI agents by condensing web tutorials into task-relevant summaries.
• The system employs a lightweight vision-language model with a guideline-generation adapter and rejection-sampling finetuning to filter and generate actionable guidance.
• Empirical evaluations demonstrate that RAG-GUI outperforms vanilla RAG methods, improving step success rates by up to 13% across multiple GUI benchmarks.

RAG-GUI denotes a retrieval-augmented approach to GUI-centered systems in which external procedural or structural knowledge is injected into generation or decision-making. In the most specific recent usage, “RAG-GUI” names a lightweight vision-LLM that leverages web tutorials at inference time for GUI agents; it is first warm-started via supervised finetuning and further refined through self-guided rejection sampling finetuning, while remaining a model-agnostic plug-in for any VLM-based agent (Xu et al., 29 Sep 2025). The same expression also appears in GUIDE, where a retrieval-augmented generation approach is used to integrate a component library into automated GUI prototyping inside Figma (Kolthoff et al., 28 Feb 2025). This suggests that the term now spans both a particular tutorial-grounded GUI-agent framework and a broader class of GUI-oriented retrieval-augmented workflows.

1. Definition and scope

In “Retrieval-augmented GUI Agents with Generative Guidelines,” RAG-GUI is defined as a lightweight VLM that leverages web tutorials at inference time. Its overall pipeline takes a task description $g$, the current GUI state $s_t$, and the history of past actions $A_t$; it retrieves top-$k$ tutorials $\tau_1 \ldots \tau_k$ from a pre-computed index, passes each tutorial through a guideline-generation adapter $f_\theta$, filters the outputs by a binary relevance label, and then conditions a frozen agent policy $\pi$ on the retained summaries to predict the next action (Xu et al., 29 Sep 2025).

A crucial terminological distinction is that this RAG-GUI is not merely “vanilla RAG.” Retrieval is non-parametric and done at inference-time only, and the adapter $f_\theta$ is inserted between retrieval and the agent. The agent $\pi$ itself is never finetuned; all learned tutorial knowledge lives in $f_\theta$ (Xu et al., 29 Sep 2025). A common misconception is therefore to treat the method as simple prompt stuffing with retrieved tutorials. The paper explicitly separates RAG-GUI from a “vanilla RAG” baseline in its experiments, indicating that the contribution is the generation and filtering of task-conditional guidance rather than retrieval alone (Xu et al., 29 Sep 2025).

The broader literature uses retrieval augmentation in several GUI-related settings. GUIDE decomposes high-level GUI descriptions into fine-granular GUI requirements and retrieves relevant Material Design component types before rendering final JSON prototypes (Kolthoff et al., 28 Feb 2025). GUI-ReRank retrieves and reranks GUI prototypes for inspiration or downstream RAG workflows (Kolthoff et al., 5 Aug 2025). KG-RAG augments GUI agents with a knowledge graph derived from a UI Transition Graph (Guan et al., 30 Aug 2025). InspectorRAGet, RAGExplorer, and XGraphRAG provide visual interfaces for analyzing RAG systems rather than acting within GUIs directly (Fadnis et al., 2024, Tian et al., 19 Jan 2026, Wang et al., 10 Jun 2025).

2. Core architecture of tutorial-grounded RAG-GUI

The operational pipeline begins with a retrieval module built over pre-computed E5 embeddings for approximately $s_t$0 M GUI tutorials, drawn from MINT, OmniCorpus, and WikiHow after FastText filtering (“Computers & Electronics”), deduplication, and LLM-based fine-grained classification with Qwen2.5. At step $s_t$1, the system embeds the task description $s_t$2 and retrieves the top-$s_t$3 nearest tutorials in vector space; Figure 2a reports a typical choice of $s_t$4, balancing coverage versus context length (Xu et al., 29 Sep 2025).

For each retrieved tutorial $s_t$5, the guideline-generation adapter produces a binary relevance label and a concise, task-conditional summary:

$s_t$6

Only summaries with $s_t$7 are retained, yielding the filtered set $s_t$8. The frozen agent policy then predicts the next action from $s_t$9 (Xu et al., 29 Sep 2025).

This decomposition creates a narrow interface between retrieval and action. Because $A_t$0 consumes text inputs and tutorial text, the design is described as model-agnostic and “plug-in”: it requires no modification to the architecture or weights of the downstream policy $A_t$1 (Xu et al., 29 Sep 2025). The architectural role of retrieval is therefore not to replace the policy, but to supply task-aware guidance that is conditioned on the current step and prior actions.

The paper’s example makes this concrete. For the goal “Book a flight from Rotterdam to Puerto Natales on Jan 11,” after the previous action “Clicked on Flights tab,” a retrieved tutorial containing irrelevant material such as reviews and hotels is condensed by RAG-GUI into a summary focused on the immediately useful sequence: fill the form with departure city, arrival city, and date; search; pick a flight; verify price and dates; enter passenger information; accept terms; and confirm (Xu et al., 29 Sep 2025). The mechanism is thus selective and state-conditional rather than purely descriptive.

3. Training objectives and the generative-guidelines mechanism

RAG-GUI trains the adapter $A_t$2 in two stages: supervised finetuning warmup and self-guided rejection-sampling finetuning. In the supervised stage, synthetic $A_t$3 pairs are created by prompting GPT-4.1-mini to produce high-quality guidance $A_t$4 for each tuple $A_t$5. The dataset is denoted $A_t$6, and the optimization objective is

$A_t$7

The result is an adapter that can already produce relevance scores and rough summaries (Xu et al., 29 Sep 2025).

The second stage is motivated by the claim that not all SFT guidance is equally helpful to the final agent $A_t$8. The paper hypothesizes that “good” summaries and labels are those that cause the agent to pick the correct ground-truth action $A_t$9. Let $k$0 be the adapter distribution over guidance outputs for tutorial $k$1, and let the gain be measured through the true next-action probability conditioned on the summary. The marginal likelihood of the correct action is optimized through an evidence lower bound:

$k$2

The optimal posterior is proportional to

$k$3

The rejection-sampling procedure then samples $k$4 candidate guidances, executes the agent with each summary, retains only those for which $k$5 chooses the correct $k$6, forms $k$7, and minimizes

$k$8

The paper describes this as a self-supervision step that “pulls up” the probability of truly helpful guidance (Xu et al., 29 Sep 2025).

Implementation details are explicit: SFT uses $k$9 epoch and learning rate $\tau_1 \ldots \tau_k$0 with a cosine scheduler; RSF adds $\tau_1 \ldots \tau_k$1 epoch with learning rate $\tau_1 \ldots \tau_k$2 and temperature $\tau_1 \ldots \tau_k$3; the backbone is Qwen-2.5-VL-7B (Xu et al., 29 Sep 2025). These choices underscore that the learnable component is deliberately lightweight relative to the downstream agent.

4. Empirical results and comparative behavior

RAG-GUI is evaluated on three benchmarks. AndroidWorld is an online emulator benchmark with $\tau_1 \ldots \tau_k$4 tasks and step success rate as the metric. Multimodal-Mind2Web contains $\tau_1 \ldots \tau_k$5 tasks over $\tau_1 \ldots \tau_k$6 websites and reports Element Accuracy, Operation F1, and Step SR. AndroidControl contains $\tau_1 \ldots \tau_k$7 tasks and reports step accuracy, split into “high-level” and “low-level” instructions (Xu et al., 29 Sep 2025).

The main quantitative result reported in the abstract is that the method consistently outperforms baseline agents and surpasses other inference baselines by $\tau_1 \ldots \tau_k$8 to $\tau_1 \ldots \tau_k$9 across two model sizes (Xu et al., 29 Sep 2025). The table excerpt in the paper makes the comparison more specific. For Qwen2.5-VL-7B, the frozen baseline achieves $f_\theta$0 AW-SR, $f_\theta$1 M2W Element Accuracy, and $f_\theta$2 M2W Step SR; vanilla RAG yields $f_\theta$3, $f_\theta$4, and $f_\theta$5; RAG-GUI-7B yields $f_\theta$6, $f_\theta$7 $f_\theta$8, and $f_\theta$9 $\pi$0 (Xu et al., 29 Sep 2025).

The same pattern appears at the larger scale. Qwen2.5-VL-72B achieves $\pi$1 AW-SR, $\pi$2 M2W Element Accuracy, and $\pi$3 M2W Step SR; vanilla RAG gives $\pi$4, $\pi$5, and $\pi$6; RAG-GUI-72B reaches $\pi$7, $\pi$8 $\pi$9, and $f_\theta$0 $f_\theta$1 (Xu et al., 29 Sep 2025). This directly refutes the idea that retrieved tutorials are uniformly beneficial in raw form. On these results, tutorial retrieval without guideline generation can underperform the frozen agent on some metrics, whereas the adapter-based formulation improves them.

The paper also reports qualitative behavior. Tutorials often contain noise or irrelevant sections, but RAG-GUI’s summaries focus only on the steps needed now. In AndroidControl, the system correctly conditions on prior actions such as “Flights tab already open” and avoids repeating them (Xu et al., 29 Sep 2025). Gains are described as especially large in the online AndroidWorld setting, which features realistic, multi-step scenarios (Xu et al., 29 Sep 2025).

5. Related GUI-oriented retrieval-augmented systems

The most closely related but distinct mechanism appears in GUIDE. GUIDE is an LLM-driven GUI generation decomposition approach integrated into Figma. It first decomposes a high-level GUI description into an array of feature objects, then uses a retrieval-augmented component-selection step over a simplified Material Design component library, and finally generates JSON specifications that instantiate real Figma nodes through the official Material-3 Figma Team Library (Kolthoff et al., 28 Feb 2025). The retrieval stage is formalized with a scoring function

$f_\theta$2

with an example cosine form

$f_\theta$3

followed by top-$f_\theta$4 selection of component types for each decomposed feature (Kolthoff et al., 28 Feb 2025). GUIDE therefore uses RAG to constrain generation with component-library context, not to guide an action policy. Its reported preliminary evaluation gives an average of $f_\theta$5 GUIs per participant in $f_\theta$6 minutes for the control condition and $f_\theta$7 GUIs in $f_\theta$8 minutes for the treatment condition, with crowd-worker ratings significantly favoring GUIDE across all reported quality dimensions (Kolthoff et al., 28 Feb 2025).

GUI-ReRank addresses a different retrieval problem: natural-language search over repositories of GUI prototypes. It defines GUI retrieval as ranking GUI screenshots and annotations against a free-text query through a two-stage architecture: embedding-based constrained retrieval followed by multi-modal LLM reranking (Kolthoff et al., 5 Aug 2025). The framework is explicitly positioned as useful for “rapid discovery of relevant GUIs for inspiration or seamless integration into customized LLM-based RAG workflows” (Kolthoff et al., 5 Aug 2025). On a benchmark of $f_\theta$9 queries, the paper reports GPT-4.1 image-based reranking at AP $\pi$0 versus SOTA BERT-LTR approximately $\pi$1, and MRR $\pi$2 versus approximately $\pi$3; it also reports that text-based reranking is approximately $\pi$4 cheaper and $\pi$5 faster than image-based reranking (Kolthoff et al., 5 Aug 2025). This is retrieval for example selection and inspiration rather than in-situ GUI control.

KG-RAG is closer to agent execution, but its knowledge source is structural rather than tutorial-based. It augments any LLM-based GUI agent with a structured, retrieval-augmented knowledge graph derived from an app’s UI Transition Graph. Offline, xTester explores screens and records transitions; intents and trajectories are generated and embedded into an ANN store; online, a user instruction is embedded and matched to relevant trajectories, which are passed to the agent’s prompt as action-oriented knowledge snippets (Guan et al., 30 Aug 2025). On DroidTask with a GPT-4 backend, KG-RAG reports SR $\pi$6, DA $\pi$7, and AS $\pi$8, compared with AutoDroid at SR $\pi$9, DA $f_\theta$0, and AS $f_\theta$1 (Guan et al., 30 Aug 2025). The contrast with tutorial-grounded RAG-GUI is therefore one of memory substrate: web tutorials versus UTG-derived trajectory memory.

A separate but complementary strand uses GUI systems to inspect RAG itself. InspectorRAGet is a stateless, web-based analytics application that ingests a standardized JSON “experiment results” file and presents coordinated views for aggregate and instance-level inspection of RAG model outputs, human and algorithmic metrics, annotator quality, correlations, and Fisher’s randomization test results (Fadnis et al., 2024). RAGExplorer is a web-based GUI for comparative diagnosis of RAG configurations, with a Configuration Dashboard, Performance Landscape View, Failure-Case Inspector, and Context Manipulation Panel; it supports macro-to-micro analysis over chunk size, embedding model, reranker, and generator choices, and includes a hierarchical failure attribution cascade assigning each incorrect answer to exactly one failure point FP1–FP7 (Tian et al., 19 Jan 2026). XGraphRAG, finally, provides interactive visual analysis for graph-based RAG, exposing extraction, merge, summary, recall, and inference stages through coordinated views such as QA & Inference-Trace, Topic Explore, Entity Explore, and LLM Invocation (Wang et al., 10 Jun 2025). These systems do not implement GUI agents, but they establish a parallel research direction in which graphical interfaces are used to make RAG pipelines diagnosable and traceable.

6. Misconceptions, limitations, and research directions

A recurrent misconception is that GUI-oriented RAG is a single method family with a uniform objective. The literature does not support that simplification. In the cited papers, retrieval augmentation is used to summarize tutorials for action selection, retrieve component types for prototyping, rerank GUI repositories, construct UTG-grounded action memory, and inspect the behavior of RAG systems themselves (Xu et al., 29 Sep 2025, Kolthoff et al., 28 Feb 2025, Kolthoff et al., 5 Aug 2025, Guan et al., 30 Aug 2025, Fadnis et al., 2024, Tian et al., 19 Jan 2026, Wang et al., 10 Jun 2025). The shared mechanism is retrieval-conditioned generation or decision-making; the retrieved object, learning target, and evaluation protocol differ substantially.

For tutorial-grounded RAG-GUI specifically, the paper identifies several limitations. If no relevant tutorial exists in the index, $f_\theta$2 may produce spurious summaries. The evaluation is reported only on Qwen-VL-series backbones, and extending to other VLMs such as LLaVA or GPT-4V is left open. The system also introduces inference overhead: retrieval for $f_\theta$3 tutorials uses E5 embeddings and ANN search, and the adapter requires three forward passes, one per tutorial (Xu et al., 29 Sep 2025). The deployment argument is that this added cost is justified by improvements of up to $f_\theta$4 step success rate (Xu et al., 29 Sep 2025).

The paper’s own future-facing remark is that integrating guideline generation into end-to-end finetuning of $f_\theta$5 could further boost performance (Xu et al., 29 Sep 2025). Parallel work points to adjacent directions rather than direct extensions. GUIDE lists future planned enhancements such as automated pattern discovery, support for hierarchical datasets, and integration of vector-similarity metrics for retrieval evaluation (Fadnis et al., 2024); GUIDE also frames minor-change preservation as a controlled prompting objective in GUI generation (Kolthoff et al., 28 Feb 2025). KG-RAG reports that UTG extraction cost saturates at approximately $f_\theta$6 hours per complex app, guiding practical deployment trade-offs (Guan et al., 30 Aug 2025). RAGExplorer and XGraphRAG emphasize causal verification and end-to-end traceability in diagnosis workflows (Tian et al., 19 Jan 2026, Wang et al., 10 Jun 2025). A plausible implication is that future “RAG-GUI” systems will be evaluated not only by task success, but also by how transparently they expose the retrieved evidence, its transformation, and its causal effect on GUI behavior.