GUI Data Pipeline Overview

• GUI Data Pipeline is a comprehensive framework that collects and harmonizes multimodal UI data (screenshots, hierarchies, OCR, and traces) to facilitate representation learning and task deployment.
• It employs diverse acquisition methods—from static repository mining to live environmental instrumentation—and unifies disparate modalities through structured annotation and preprocessing.
• Quality control and self-evolving training loops refine data fidelity and grounding precision, driving robust navigation, reasoning, and functional deployment in GUI systems.

Searching arXiv for recent and foundational GUI data pipeline papers to ground the article. GUI data pipeline denotes the end-to-end process by which screenshots, UI hierarchies, accessibility trees, OCR text, interaction traces, user queries, and tool-call side effects are collected, normalized, annotated, filtered, and converted into reusable corpora for GUI representation learning, element grounding, navigation, reasoning, and deployment. In the recent literature, the term spans self-supervised embedding pipelines such as Screen2Vec (Li et al., 2021), automatic functionality annotation systems such as AutoGUI (Li et al., 4 Feb 2025), large-scale instruction synthesis pipelines such as UI-E2I-Synth (Liu et al., 15 Apr 2025), self-evolving training engines such as Step-GUI (Yan et al., 17 Dec 2025) and MAI-UI (Zhou et al., 26 Dec 2025), and trajectory-construction frameworks derived from tutorial videos or autonomous exploration such as Video2GUI (Xiong et al., 14 May 2026) and GUI-ReWalk (Lin et al., 19 Sep 2025). Across these systems, the pipeline is not merely data ingestion; it is the mechanism that determines what GUI states are observable, how actions are grounded, which labels are trusted, and which behaviors can later be optimized.

1. Data objects, modalities, and scope

A GUI data pipeline typically operates over multiple synchronized representations of the same interaction state. The most common are screenshot bitmaps, structural trees, textual overlays, action traces, and metadata. Screen2Vec formalized this structure on Android through Rico, where each screen includes a screenshot, a rooted view hierarchy JSON with className, text content, bounding boxes, and properties such as clickable or scrollable, together with interaction traces and Google Play description text (Li et al., 2021). Later systems broadened the same principle to web DOM trees, macOS accessibility trees, Windows UI Automation metadata, and cross-platform action schemas.

The modalities are heterogeneous but complementary. Structural metadata supplies candidate elements and spatial constraints; screenshots encode iconography, styling, and non-textual cues; OCR exposes rendered text not cleanly available in UI trees; interaction traces introduce temporal semantics; and higher-level fields such as app descriptions, task instructions, or user-agent dialogues supply intent. MagicGUI made this multimodal composition explicit by combining screenshots, XML hierarchies, OCR text, curated instructions, and action traces with pre- and post-execution states in a single pipeline designed for perception, grounding, and reasoning (Tang et al., 19 Jul 2025). GUI-Libra extended the canonical step representation further by requiring a > block and a structured JSON action with action_type, action_description, action_target, value, and point_2d, thereby treating reasoning text and executable GUI control as co-labeled outputs rather than separate datasets (Yang et al., 25 Feb 2026).

The scope of a GUI data pipeline is therefore broader than dataset construction in the narrow sense. It may support screen embedding, as in Screen2Vec’s 768-D component vectors and 1536-D screen vectors (Li et al., 2021); grounding-only corpora, as in AutoGUI-704k (Li et al., 4 Feb 2025); trajectory-heavy agent pretraining, as in WildGUI’s 12.7M interaction trajectories (Xiong et al., 14 May 2026); or deployment-facing systems that preserve device-local privacy while selectively escalating difficult states to the cloud, as in MAI-UI’s native device-cloud collaboration (Zhou et al., 26 Dec 2025). This breadth suggests that “GUI data pipeline” is best understood as an architectural layer that mediates between raw interface phenomena and trainable supervision.

2. Data acquisition and environment instrumentation

Data acquisition strategies in this literature range from static repository mining to live environment interaction. Rico-based work mined 66,261 unique Android screens from 9,384 free apps, with interaction traces obtained through hybrid crowdsourcing and automated discovery (Li et al., 2021). At a much larger web-and-desktop scale, Video2GUI started from more than 500M YouTube metadata entries, reduced them to about 20M candidate videos with metadata classification, retained 4.16M high-quality tutorial videos after video scoring, and converted them into WildGUI, a dataset with 12.7M structured trajectories and 124.5M screenshots spanning more than 1,500 applications and websites (Xiong et al., 14 May 2026).

Other pipelines instrument the environment directly rather than mining existing traces. ReCAP built a Flask-based dynamic CAPTCHA environment with seven challenge types—Text, Compact Text, Icon Match, Icon Selection, Paged, Slider, and Image Grid—and used it to generate approximately 150,000 expert solution trajectories, 10,000 self-correction trajectories, and 73,000 general GUI trajectories for capability preservation (Chen et al., 23 Mar 2026). GUIrilla used the native macOS Accessibility API to crawl 1,108 applications, organizing discovered interface states into hierarchical GUI graphs and releasing GUIrilla-Task with 27,171 functionally grounded tasks and full-desktop plus window-level screenshots (Garkot et al., 16 Oct 2025). GUI-ReWalk, by contrast, generated trajectories through stochastic exploration followed by intent-aware completion and retrospective annotation, yielding 50k+ annotated tasks with an average trajectory length of 22.5 steps (Lin et al., 19 Sep 2025).

A recurring pattern is that acquisition is inseparable from instrumentation. Some systems depend on runtime collection pipelines, some on real-device farms, and some on synthetic or cloud-hosted environments. The data source determines what can later be verified: simulator predicates in hybrid flywheel systems can validate subtask completion exactly, whereas video-derived trajectories require multimodal annotation and spatial grounding after the fact (Xu et al., 15 Feb 2026).

Pipeline	Primary acquisition source	Main output
Screen2Vec	Rico Android screens and traces	Screen/component embeddings
AutoGUI	Web crawler and Android emulator interactions	Functionality annotations
Video2GUI	Internet-scale tutorial videos	Grounded trajectories
GUIrilla	macOS AX crawling	Functionally grounded desktop tasks
ReCAP	Dynamic CAPTCHA environment	Reasoning-action and self-correction traces

This diversity of acquisition modes is methodologically important. Pipelines built from public corpora inherit annotation schemas and platform biases; pipelines built from controlled environments gain verifiable transitions; and pipelines built from exploratory agents or tutorial videos can expand coverage far beyond manually annotated datasets, albeit with stronger downstream filtering requirements.

3. Preprocessing, schema unification, and annotation construction

After acquisition, the central problem is converting raw GUI evidence into a unified schema. UI-E2I-Synth addressed this explicitly across web, Windows, and Android by parsing only three attributes per element—element type, element content, and bounding box—and mapping platform-specific metadata into five unified element types: Text, Inputfield, Dropdown, Icon, and Toggle (Liu et al., 15 Apr 2025). The same pipeline then generated explicit and implicit referring expressions with GPT-4o using attribute-enhanced Set-of-Marks prompts, and parameterized them into first-person instructions through actionType, actionContentDescription, and actionContent. The resulting training corpus comprised 1,635,594 screenshots and 9,899,581 instructions (Liu et al., 15 Apr 2025).

Other systems decompose preprocessing by modality. MP-GUI split its corpus into Text-Aware Data, Graphics-Aware Data, and Spatial-Aware Data, using OCR to isolate text boxes, YOLOv8 to detect icons and widgets, and view-hierarchy parsing to derive parent-child and sibling relations for its Spatial Relation Perception task (Wang et al., 18 Mar 2025). GUI-Reflection instead transformed successful trajectories into reflection-oriented supervision by two automated procedures: goal modification that turns a formerly correct step into an error under a modified goal, and ineffective action insertion that creates no visual change but requires reflective correction. Its offline SFT mixture covered 265K steps drawn from AITW, AITZ, AMEX, AndroidControl, and GUI-Odyssey (Wu et al., 9 Jun 2025).

Reasoning annotation introduces another layer of preprocessing. GUI-Critic-R1 built action-level critic data by collecting correct actions from public datasets, generating realistic incorrect alternatives through open-source agents, filtering labels with GPT-4o, and then using progressive CoT bootstrapping so that explanations were accepted only when a strong MLLM independently reproduced the correct label and suggestion (Wanyan et al., 5 Jun 2025). The resulting GUI-Critic-Train contained roughly 11k entries, about 6k with high-quality CoT, while GUI-Critic-Test contained manually validated mobile and web splits (Wanyan et al., 5 Jun 2025). GUI-Libra performed a related but more action-centric construction: it augmented existing trajectories with GPT-4.1-generated reasoning, then filtered them with stochastic re-prediction and bounding-box verification, ultimately releasing 81K reasoning-then-action samples (Yang et al., 25 Feb 2026).

AutoGUI is exemplary in how preprocessing becomes semantics. It does not simply attach captions to elements; it infers functionality from state changes caused by interactions. For each element, it captures pre- and post-action states, computes diffs over AXTree structure or region-level screen descriptions, generates a functionality hypothesis with Llama-3-70B, rejects low-quality samples by rules and an LLM rejector, and keeps only samples that receive maximal scores from two independent LLM verifiers (Li et al., 4 Feb 2025). This procedure produced AutoGUI-704k, a functionality-grounding corpus whose human-evaluated annotation correctness reached 96.7%, compared with 95.5% for a trained human annotator (Li et al., 4 Feb 2025).

4. Quality control, difficulty modeling, and dataset curation

Quality control is the point at which GUI data pipelines diverge most sharply from naïve large-scale scraping. Several systems explicitly treat curation as the primary source of performance gains. GoClick’s Progressive Data Refinement pipeline begins with a 10.8M-sample raw corpus, applies denoising and deduplication, removes reverse-generation tasks and outdated GUI patterns in a coarse stage, adjusts source-task ratios in a fine stage, and ends with a 3.8M-sample core set that improves average grounding accuracy across seven benchmarks despite reducing data volume by 64.8% (Li et al., 27 Apr 2026). Step-GUI’s Calibrated Step Reward System takes a different route: it uses trajectory-level success or failure as a reliable anchor, calibrates dense step signals against that anchor, and reports more than 90% annotation accuracy at 10–100× lower cost than traditional step-level annotation (Yan et al., 17 Dec 2025).

Filtering may be structural, semantic, or difficulty-aware. POINTS-GUI-G uses OmniParser-v2 to validate whether a ground-truth point or box is actually covered by detected UI elements and computes a coverage score

$$S = \frac{\sum_{i=1}^{n}\mathrm{Area}(B_{\mathrm{gt}}\cap B_{\mathrm{det},i})}{\mathrm{Area}(B_{\mathrm{gt}})},$$

retaining samples only when $S \ge \tau$ (Zhao et al., 6 Feb 2026). The same pipeline grades samples by layout entropy and prioritizes higher-resolution, higher-complexity examples, which contributes to its reported gains on high-resolution grounding benchmarks (Zhao et al., 6 Feb 2026).

GUI-D, the curation component used by GUI-C$^2$, formalizes difficulty as a weighted combination of baseline failure rate, mean distance to target box, shortest target side, target area, rollout dispersion, and parsing failure ratio (Li et al., 29 May 2026). Its raw score is

$$\tilde q_i=\operatorname{clip}_{[0,1]}\!\left(0.40g_i^{\mathrm{fail}}+0.20g_i^{\mathrm{dist}}+0.15g_i^{\mathrm{short}}+0.10g_i^{\mathrm{area}}+0.10g_i^{\mathrm{disp}}+0.05g_i^{\mathrm{miss}}\right),$$

followed by platform-wise percentile normalization and conversion into a training weight

$$\alpha_i=\operatorname{clip}_{[\alpha_{\min},\alpha_{\max}]}\!\left(1+\lambda_d(q_i-0.5)\right).$$

The key operational point is that GUI-D does not merely remove bad data; it preserves mixed-outcome samples and attenuates or amplifies their contribution during RL (Li et al., 29 May 2026).

Across these systems, quality control is not ancillary bookkeeping. It directly controls the reliability of grounding labels, the usefulness of failed trajectories, and the distribution of easy versus training-worthy samples. A plausible implication is that, in GUI pipelines, curation policy often matters as much as model architecture because the target space is extremely sensitive to small annotation errors: a single mislocalized click, stale accessibility node, or mislabeled instruction can corrupt both imitation and reward estimation.

5. Supervision objectives and self-evolving training loops

GUI data pipelines are tightly coupled to training objectives. Screen2Vec is an early example of a two-layer self-supervised design: at the component level it predicts a target component from intra-screen context; at the screen level it predicts a target screen from surrounding screens in interaction traces, using app description embeddings only after training to avoid dominance (Li et al., 2021). The supervision signal is entirely endogenous to GUI structure and navigation order rather than manually provided labels.

Later systems extended this into closed-loop post-training and RL. GUI-Libra argues that generic CoT SFT can hurt grounding, and therefore mixes reasoning-then-action data with direct-action data while reweighting action and grounding tokens:

$$\mathcal{L}_{\text{ASFT}(\theta)} = - \mathbb{E}_{(x_t, c_t, a_t, g_t) \sim D_{\rm mix}} \frac{ \log \pi_\theta(c_t \mid x_t) + \alpha_a \log \pi_\theta(a_t \mid x_t, c_t) + \alpha_g \log \pi_\theta(g_t \mid x_t, c_t, a_t) }{ |c_t| + \alpha_a |a_t| + \alpha_g |g_t| }.$$

It then applies KL-regularized GRPO with success-adaptive scaling to handle partial verifiability in GUI navigation (Yang et al., 25 Feb 2026). Step-GUI organizes a different self-evolving loop—Mid-Train, Cold-Start, and RLVR—where successful trajectories yield seven categories of structured training signals and failed ones yield six knowledge-focused categories only, thereby avoiding imitation of wrong actions while still exploiting reasoning traces (Yan et al., 17 Dec 2025).

Hybrid online-offline flywheels push this logic further. GUI-Owl-1.5, described in the Mobile-Agent-v3.5 report, combines simulated environments, cloud-based sandbox rollouts, DAG-based task composition, checkpoint predicates, trajectory truncation to the last verified subtask, and iterative retraining, so that failed long-horizon trajectories can still contribute clean supervision up to the verified prefix (Xu et al., 15 Feb 2026). MAI-UI generalizes the self-evolving loop to include native ask_user and mcp_call actions, fine-grained rejection sampling, and online RL over 32 to 512 parallel environments, reporting that scaling environment count from 32 to 512 adds 5.2 points and increasing the step budget from 15 to 50 adds 4.3 points on its online RL experiments (Zhou et al., 26 Dec 2025). MagicGUI similarly couples large-scale continued pretraining on approximately 7.8M samples with reinforcement fine-tuning under a unified action space and a spatially enhanced composite reward (Tang et al., 19 Jul 2025).

The common structural property is recurrence: data collection informs training, training informs new rollouts, and new rollouts are selectively filtered back into the corpus. In that sense, modern GUI data pipelines are increasingly not static datasets but persistent systems for producing policy-shaped supervision.

6. Applications, evaluation regimes, and unresolved tensions

The outputs of GUI data pipelines support several distinct downstream regimes. Screen2Vec demonstrated screen similarity, composability, and preliminary task embeddings; in a Mechanical Turk nearest-neighbor study over 5,608 pairs, Screen2Vec achieved mean similarity 3.295 versus 3.014 for TextOnly and 2.410 for LayoutOnly (Li et al., 2021). AutoGUI showed strong scaling effects for functionality grounding, with SliME-8B on FuncPred rising from 3.2 to 62.6 as training scale increased from 25k to 702k (Li et al., 4 Feb 2025). Video2GUI showed that pretraining on WildGUI yields consistent 5–20% gains across multiple GUI grounding and action benchmarks and can match or surpass state of the art on several of them (Xiong et al., 14 May 2026).

Pipelines also specialize by control granularity. ShowUI-$\pi$ constructed a continuous-control branch of the field by collecting and synthesizing 20K drag trajectories across five domains and introducing ScreenDrag, a 505-task benchmark for dexterous drag operations (Hu et al., 31 Dec 2025). Its unified discrete-continuous action space represents all interactions as sequences of $(x,y,m)$ triplets, and its 450M-parameter model achieves 26.98 online success on ScreenDrag, exceeding reported proprietary baselines such as Operator at 13.27 and Gemini-2.5-CUA at 22.18 (Hu et al., 31 Dec 2025).

Deployment-oriented pipelines expose another dimension: privacy, latency, and collaboration. MAI-UI’s native device-cloud routing improves on-device performance by 33.4%, reduces cloud model calls by 42.7%, and completes 40.5% of tasks entirely on device while preserving a privacy-based blocking rule for cloud escalation (Zhou et al., 26 Dec 2025). GUI-MCP in Step-GUI serves a related role by separating low-level atomic operations from high-level task delegation to local specialist models in a privacy-preserving hierarchy (Yan et al., 17 Dec 2025). GUIrilla and macapptree highlight a different concern: reproducible desktop-scale collection on macOS, where full-desktop scenes, overlapping windows, and noisy accessibility metadata make dataset construction itself the bottleneck (Garkot et al., 16 Oct 2025).

Several methodological tensions remain unresolved. One is the trade-off between reasoning richness and grounding precision: GUI-Libra presents this as a concrete failure mode of standard CoT SFT (Yang et al., 25 Feb 2026). Another is the trade-off between automation scale and annotation fidelity: AutoGUI, Step-GUI, GoClick, and GUI-D all report that aggressive filtering or calibration is essential because raw scale alone is not sufficient (Li et al., 4 Feb 2025). A third is platform transfer. Screen2Vec was trained and evaluated only on Android and explicitly notes that desktop GUIs may be harder because of denser, more heterogeneous screens (Li et al., 2021). UI-E2I-Synth, GUIrilla, and MAI-UI suggest that cross-platform unification is feasible, but only after substantial schema engineering, platform-specific parsing, and resolution control (Liu et al., 15 Apr 2025).

In aggregate, the literature suggests that a GUI data pipeline is the core systems layer through which GUI agents acquire perceptual grounding, temporal semantics, reasoning traces, and deployment constraints. It is simultaneously a data-engineering problem, a supervision-design problem, and an operational problem. The field’s strongest results increasingly arise not from a single model innovation alone, but from pipelines that can continuously collect, verify, compress, rebalance, and re-inject GUI experience at scale.