Self-Evolving GUI Trajectory Production

• Self-evolving GUI trajectory production is a methodology that autonomously generates, refines, and adapts action sequences using agent-environment interactions and reinforcement mechanisms.
• It leverages techniques such as autonomous exploration, guided replay, and world-model simulation to synthesize high-quality, dynamic GUI trajectories for various automation tasks.
• These methods have demonstrated improved benchmark success rates and cross-platform efficiency, providing robust performance in dynamic, real and simulated digital environments.

Self-evolving GUI trajectory production refers to the family of methodologies, frameworks, and learning paradigms designed to autonomously generate, refine, and adapt sequences of GUI actions (trajectories) for the purposes of automation, agent training, interaction testing, or UI synthesis. Unlike static datasets or rule-based systems, self-evolving approaches dynamically produce and improve trajectory data by leveraging agent-environment interactions, unsupervised or self-supervised learning, iterative feedback, and reinforcement mechanisms. These methods aim to produce high-quality, diverse, and adaptive GUI trajectories that robustly generalize across real and simulated digital environments and can adapt to new tasks, changing interface layouts, and user behaviors.

1. Core Paradigms in Self-Evolving GUI Trajectory Production

Several complementary paradigms underpin self-evolving GUI trajectory production:

• Autonomous Exploration and Reverse Synthesis: Agents actively traverse GUI environments, performing atomic low-level actions (e.g., click, scroll, type), recording state transitions, and subsequently retrofitting these raw interactions into semantically meaningful trajectories using models that synthesize high-level task instructions from the observed effects (Sun et al., 27 Dec 2024).
• Guided Replay and Tutorial Mining: Automated agents replay tasks extracted from web tutorials, articles, or videos. Structured task representations are inferred from unstructured content using LLMs, which are then executed in real or simulated environments to harvest action-sequence data, with trajectory outcomes being automatically validated (Xu et al., 12 Dec 2024, Zhang et al., 17 Apr 2025).
• Self-supervised and Reward-driven Reinforcement Learning: Agents leverage inverse dynamics modeling, reward models, and policy optimization to learn from unlabeled UI transitions or to refine their performance based on diverse, automatically-generated reward signals at both action and trajectory levels (Yuan et al., 18 May 2025, Gao et al., 18 May 2025, Xiao et al., 27 May 2025, Nong et al., 15 Aug 2025).
• World-Model and Simulation-based Planning: A learned or heuristic world model simulates GUI state transitions offline, enabling large-scale, sample-efficient synthesis of diverse and reversible trajectories. Tree-based planning with Monte Carlo Tree Search (MCTS) is employed for goal-directed exploration and error recovery (Gao et al., 6 Jul 2025).
• Curriculum and Preference-based Evolution: The training process is organized as a curriculum of tasks with increasing difficulty, and data is filtered or weighted using quality or preference estimations (e.g., Monte Carlo accuracy, intersection-over-union diversity), to continuously expose agents to challenging or instructive experiences (Wang et al., 4 Sep 2025, Nong et al., 15 Aug 2025, Shi et al., 8 Jul 2025).

These core approaches emphasize the importance of iterative improvement, error-driven adaptation, and automatic synthesis or mining of richly-annotated trajectory datasets.

2. Technical Methodologies and Algorithmic Components

Technical implementations of self-evolving GUI trajectory production are characterized by:

• Interaction Triplet Mining and Reverse Task Synthesis: Recording (pre-state, action, post-state) triplets during agent exploration. An annotation model derives semantic task descriptions:

$$f_{\text{low}}: \langle s_{\text{pre}}, a, s_{\text{post}}\rangle \rightarrow \tau_{\text{low}}$$

$$f_{\text{high}}: \tau_{\text{low}} \rightarrow \tau_{\text{high}}$$

Enabling mapping from low-level atomic manipulations to coherent, high-level tasks (Sun et al., 27 Dec 2024).

• Process and Preference-based Reward Modeling: At each step, a reward model provides scalar feedback $r_t = \mathbb{R}(x, h_t, s_t, a_t)$, where $h_t$ is a historical summary. Mean squared error loss is used for training,

$$\mathcal{L}(\theta) = \frac{1}{\sum_{i=1}^N T^{(i)}} \sum_{i=1}^N \sum_{t=1}^{T^{(i)}} (r_{t,\text{pred}}^{(i)} - r_{t,\text{anno}}^{(i)})^2$$

(Hu et al., 22 Apr 2025, Xiao et al., 27 May 2025). Preference data is constructed using Monte Carlo accuracy and IoU-based diversity:

$$R_\mathrm{acc}(\cdot) = \frac{1}{N} \sum_i \mathbb{I}[a_{\text{click}}^{(i)} = \hat{a}_{\text{click}}]$$

$$R_\mathrm{div}(r_1^1, r_1^2) = \mathrm{IoU}(r_1^1, r_1^2)$$

(Wang et al., 4 Sep 2025).

• RL Policy Optimization with Curriculum and Trajectory-level Signals: Trajectory-aware relative policy optimization (TRPO/GRPO), where the trajectory-level advantage is computed and rescaled:

$$\hat{a}_\tau = \frac{R(\tau) - \bar{R}}{\sigma_R + \epsilon}$$

Each step in the trajectory receives this normalized advantage (Ye et al., 21 Aug 2025, Nong et al., 15 Aug 2025, Shi et al., 8 Jul 2025).

• World-Model-Guided Planning: Trajectories are synthesized by interacting with a learned emulator:

$$o_t \sim \omega_{\theta}(o_t | o_{t-1}, a_t), \quad a_t \sim \pi_{\theta}(a_t | o_{t-1}, q)$$

Tree search (MCTS) is used with UCB criterion:

$U_C = v_C + \epsilon \sqrt{\frac{\ln n_P}{n_C}}$

facilitating efficient exploration and error rollbacks (Gao et al., 6 Jul 2025).

• Data Pipelines and Automated Task/Query Generation: Large-scale virtual environments produce diverse screenshots and metadata; LLMs synthesize realistic queries/instructions; correctness critics filter data at step and trajectory levels, enforcing consistency and rejecting hallucinations (Ye et al., 21 Aug 2025, Sun et al., 27 Dec 2024).

3. Performance Benchmarks and Evaluation Methods

Empirical evaluation of self-evolving approaches utilizes a variety of benchmarks and metrics:

• Major Public Benchmarks: AndroidWorld, AndroidControl, WebArena, OSWorld, ScreenSpot-Pro, and Mind2Web capture grounding, navigation, planning, and procedural knowledge across desktop, web, and mobile GUIs (Ye et al., 21 Aug 2025, Gao et al., 6 Jul 2025, Yuan et al., 18 May 2025).
• Trajectory- and Step-Level Metrics:
  • Success Rate (SR): Fraction of completed tasks.
  • Step Success Rate (SSR):

  $\mathrm{SSR} = \frac{1}{N} \sum_e \frac{C(e)}{T_e}$

  (Cheng et al., 31 Mar 2025). - Weighted Pathway Success Rate (WPSR):

  $$\mathrm{WPSR} = \sum_{j,i} w_{j,i}\, \mathbb{I}\{S_{\text{task}}(j,i)\}$$

  with weight $w_{j,i}$ proportional to task difficulty (Zheng et al., 2 Aug 2025). - Frechet Inception Distance (FID) and 1-Nearest Neighbor Accuracy (1-NNA) for design synthesis (Zhao et al., 2021). - Task and atomic task completion ratios, pass@k, grounding precision, and various fine-grained action and semantic accuracy metrics.

• Empirical Outcomes:

  • RL-based and curriculum strategies consistently outperformed SFT and supervised baselines, e.g., up to 47.3% grounding accuracy on ScreenSpot-Pro for a 7B model with 3k examples—surpassing larger models trained on far more data (Yuan et al., 18 May 2025).
  • World-model-based planning methods (e.g., WebSynthesis) allowed agents trained with only ~4k synthetic samples to match or exceed agents that relied on tens of thousands of real trajectories (Gao et al., 6 Jul 2025).
  • Self-correcting agent capabilities (e.g., via follow-up questions) produced step and task success rates in excess of 99% when ambiguity was encountered (Cheng et al., 31 Mar 2025).
  • Iterative self-improvement and online RL (MobileGUI-RL, Mobile-Agent-v3) achieved state-of-the-art cross-platform scores on AndroidWorld, OSWorld, and AITW, with performance increasing with each self-reinforcing iteration (Ye et al., 21 Aug 2025, Shi et al., 8 Jul 2025).

4. Applications and Agent Designs

Applications and agent architectures for self-evolving GUI trajectory production include:

• General-Purpose GUI Agents: Agent frameworks integrating multimodal perception, hierarchical planning, memory mechanisms, and chain-of-thought reasoning (e.g., GUI-Owl, LightManus, AppAgentX), supporting tasks such as search, navigation, form filling, question answering, and multi-app workflows (Ye et al., 21 Aug 2025, Jiang et al., 4 Mar 2025, Zheng et al., 2 Aug 2025).
• Automated Testing and Cross-Platform Automation: Agents autonomously generate sequences covering diverse execution paths and state transitions, enabling dynamic, scalable testing and robust cross-OS automation (Xie et al., 22 May 2025).
• Self-Improving and Continual Learning Agents: Frameworks such as UI-Genie and GUI-reflection deploy reward-guided exploration in dynamic environments with iterative refinement, reward model updating, and synthetic data augmentation—yielding robust performance on long-horizon and multi-app tasks (Xiao et al., 27 May 2025, Wu et al., 9 Jun 2025).
• Active Perception and Preference Optimization: Models equipped with step-wise region refinement and preference learning (e.g., LASER), enabling superior performance on high-resolution, multi-element GUIs with minimal data (Wang et al., 4 Sep 2025).
• Interactive Agents with Self-Correction: Agents that proactively ask clarifying questions, perform rollback, or replan in cases of ambiguities or errors (see Navi-plus and GUI-reflection), yielding trajectories that adapt in real time to uncertainty or missing information (Cheng et al., 31 Mar 2025, Wu et al., 9 Jun 2025).

5. Challenges, Limitations, and Open Directions

Despite significant advances, several challenges persist:

• Data and Task Diversity: Full generalization requires broad, high-quality trajectory data spanning numerous GUI layouts and task types. Synthetic data pipelines, tutorial mining, and reverse-synthesis are mitigating annotation costs but may still miss rare edge-cases or highly dynamic interfaces (Xu et al., 12 Dec 2024, Sun et al., 27 Dec 2024).
• Error Attribution and Correction: Accurately diagnosing failure modes and efficiently recovering trajectories remains non-trivial, particularly for agents operating over long causal pathways or with limited external feedback (Zheng et al., 2 Aug 2025, Wu et al., 9 Jun 2025).
• Scalability and Efficiency: Real-time RL and online adaptation demand scalable simulation, replay buffers, and distributed agent-environment architectures (Ye et al., 21 Aug 2025, Shi et al., 8 Jul 2025). High API or simulation costs are addressed with world-model-based simulation, though fidelity and coverage depend on model capacity (Gao et al., 6 Jul 2025).
• Evaluation Consistency: Traditional benchmarks often miss long-horizon dependencies or error cascades; emerging standards (WPSR, MATCR, p-ATSR) and public releases of curated datasets with human-verified, causally-grounded trajectories are essential for reproducible progress (Zheng et al., 2 Aug 2025).

A plausible implication is that future agent systems will increasingly integrate reflection, rollback, and continual feedback mechanisms, coupled with scalable, open-source data pipelines and curriculum strategies, moving toward robust, adaptive, and generalizable self-evolving GUI automation.

6. Representative Table of Key Methodologies and Metrics

Framework / Paper	Core Principle	Representative Result/Metric
OS-Genesis (Sun et al., 27 Dec 2024)	Reverse task synthesis	Success rate ↑ from ≈9.8%→17.4% AndroidWorld
AgentTrek (Xu et al., 12 Dec 2024)	Guided replay, tutorial mining	Cost/trajectory ≈ $0.55; SoTA on WebArena
Mobile-Agent-v3 (Ye et al., 21 Aug 2025)	Iterative RL, auto query gen	73.3 on AndroidWorld, SoTA open-source
MobileGUI-RL (Shi et al., 8 Jul 2025)	Online RL, curriculum	SR ↑ to 65.3% AITW-Gen; sample-efficient
UI-Genie (Xiao et al., 27 May 2025)	Reward model & self-improvement	SoTA across Android benchmarks
LASER (Wang et al., 4 Sep 2025)	Active, iterative perception	55.7 on ScreenSpot-Pro, SoTA 7B
GUI-explorer (Xie et al., 22 May 2025)	Autonomous exploration, knowledge mining	47.4% on AndroidWorld

All claims, technical methodologies, and performance figures are grounded explicitly in the referenced literature.

7. Context and Significance for the Field

Self-evolving GUI trajectory production constitutes a paradigm shift from static, manually curated data and rule-driven systems toward dynamic, autonomous, and adaptive agent learning. By leveraging agent-driven exploration, world-model-based simulation, sophisticated reward and preference optimization, and continual data-model co-evolution, this field is rapidly producing agents that exhibit robust generalization, interactive error correction, and scalable automation capabilities across real-world device and interface ecosystems. The release of benchmarks, high-fidelity datasets, and open-source implementations by leading research groups is accelerating progress and providing a foundation for subsequent innovations in human–computer interaction, automated testing, personal digital assistants, and cross-device automation.

As the landscape of GUIs becomes more complex, self-evolving approaches—characterized by exploration-driven data generation, curriculum-guided RL, reflective reasoning, and efficient simulation—are poised to define the operational backbone of next-generation intelligent interface agents.