GUI-ReWalk: Multi-Stage GUI Trajectory Synthesis

• GUI-ReWalk is a reasoning-enhanced, multi-stage framework that generates diverse and intent-aware GUI trajectories by blending random exploration with goal-directed recovery.
• The framework integrates a pipeline of stochastic exploration, reasoning-guided completion, cross-application initiation, and retrospective annotation to produce long-horizon workflows.
• Evaluations on benchmarks like ScreenSpot-Pro and AndroidControl demonstrate superior trajectory diversity, enhanced intent alignment, and improved navigation performance.

Searching arXiv for GUI-ReWalk and closely related GUI-agent papers to ground the article. arXiv search: "GUI-ReWalk" GUI-ReWalk is a reasoning-enhanced, multi-stage framework for synthesizing realistic and diverse GUI trajectories for GUI agents. It is proposed to address a central bottleneck in end-to-end automation over digital environments: the scarcity of scalable, high-quality trajectory data. The framework combines a stochastic exploration phase that emulates human trial-and-error behavior with a reasoning-guided phase in which inferred goals drive coherent and purposeful interactions, and it supports multi-stride task generation for long-horizon workflows across multiple applications. In the reported evaluations, a Qwen2.5-VL-7B model trained on the GUI-ReWalk dataset is assessed on grounding and navigation benchmarks including ScreenSpot-Pro, OSWorld-G, UI-Vision, AndroidControl, and GUI-Odyssey (Lin et al., 19 Sep 2025).

1. Problem setting and design objective

GUI agents, powered by large language and vision-LLMs, are presented as a promising route to end-to-end automation in digital environments, but their progress is described as being fundamentally constrained by the scarcity of scalable, high-quality trajectory data. Existing data collection strategies are characterized as relying either on costly and inconsistent manual annotations or on synthetic generation methods that trade off between diversity and meaningful task coverage (Lin et al., 19 Sep 2025).

GUI-ReWalk is introduced as a response to that data bottleneck. The framework is designed to automate the generation of realistic and diverse GUI interaction trajectories while preserving intent-aware structure. The motivating premise is that natural human-computer interaction exhibits both stochastic exploratory behavior and organized, goal-driven workflow structure. GUI-ReWalk therefore attempts to capture both patterns rather than optimizing only for random coverage or only for task-directed completion (Lin et al., 19 Sep 2025).

The framework is described as supporting both desktop and mobile environments, enabling large-scale data synthesis for multi-application, long-horizon workflows that are intent-aware and robust to errors. This positioning is important because it defines GUI-ReWalk primarily as a data-generation framework rather than as a narrowly scoped grounding module or a platform-specific navigation policy.

2. Multi-stage generation pipeline

GUI-ReWalk is organized as a sequence of stages that move from broad exploration to structured completion and retrospective supervision (Lin et al., 19 Sep 2025).

Stage	Mechanism	Reported role
Stochastic exploration	Random walk over primitive actions	Data diversity
Reasoning-guided completion	LLM-inferred goal and goal-conditioned action policy	Intent-aware trajectories
Cross-application initiation	New semantically related goal after a stride	Long-horizon multi-app workflows
Retrospective annotation	Backward LLM annotation of trajectory	Rich supervision without human labeling
Task recovery	Revised feasible goal when progress stalls	Robustness and human-like correction

The stochastic exploration phase is intended to emulate the way humans initially probe unfamiliar interfaces through random, trial-and-error actions. After this exploratory prefix, the framework transitions into a reasoning-guided phase in which a LLM infers a high-level goal from the end state of the random walk and then drives coherent completion toward that inferred objective. The paper further describes a cross-application task initiation mechanism: after one stride, the LLM proposes a new, semantically related goal, often in a different application or domain, and the cycle restarts (Lin et al., 19 Sep 2025).

Two additional components substantially affect the character of the generated data. First, retrospective annotation is applied at the end of each phase, yielding both step-level instructions and a sequence-level task description. Second, task recovery introduces reflective reasoning when progress stalls, allowing the system to revise the goal and continue from a dead end rather than terminating the trajectory. Together, these stages are intended to make the synthesized trajectories more diverse, more semantically meaningful, and closer to adaptive user behavior than purely scripted or purely random alternatives.

3. Formalization of exploration, goal pursuit, and recovery

The stochastic exploration phase is modeled as a Markov chain,

$\mathcal{M}_r = (\mathcal{S}, \mathcal{A}^r, P^r),$

where $\mathcal{S}$ denotes GUI states, $\mathcal{A}^r$ the available primitive actions, and $P^r$ the state transition probability. Its transition dynamics are written as

$$P^r(s_{t+1} \mid s_t) = \sum_{a_t^r} P^r(s_{t+1} \mid s_t, a_t^r) \cdot P^r(a_t^r \mid s_t),$$

with a uniform policy

$P^r(a_t^r \mid s_t) = \frac{1}{|\mathcal{A}^r|}.$

This phase is explicitly associated with maximum diversity and with visiting a wide variety of states, including rare and long-tail cases (Lin et al., 19 Sep 2025).

Goal-directed behavior is then modeled as a goal-conditioned MDP. A high-level goal is inferred from the terminal state of exploration,

$g = \Phi_{\text{LLM}}(s_{T_r}),$

and the structured phase is represented as

$$\mathcal{M}_g = (\mathcal{S}, \mathcal{A}^g, P^g, \mathcal{R}^g, \pi).$$

The transition dynamics are given as

$$P^t(s_{t+1} \mid s_t) = \sum_{a_t^g} P^t(s_{t+1} \mid s_t, a_t^g) \cdot \pi(a_t^g \mid s_t),$$

with a sparse reward

$$\mathcal{R}^g = \begin{cases} r_{\text{succ}}, & \text{if } s \in \mathcal{S}^g \ 0, & \text{otherwise}. \end{cases}$$

The paper associates this phase with intent-aware trajectory generation: actions are no longer sampled for coverage alone, but are conditioned on an inferred goal and rewarded only when completion is satisfactory (Lin et al., 19 Sep 2025).

For multi-stride generation, the next subgoal is proposed as

$\mathcal{S}$0

Retrospective annotation is formalized as

$\mathcal{S}$1

where each action-state pair receives a fine-grained instruction $\mathcal{S}$2, and the full sequence is summarized by a high-level task $\mathcal{S}$3. Task recovery is triggered by

$\mathcal{S}$4

after which a revised feasible goal is generated as

$\mathcal{S}$5

This suggests a deliberate decomposition: stochasticity is used to expand behavioral support, while goal inference, sparse success signals, and reflective revision are used to preserve coherence.

4. Supervision, dataset characteristics, and diversity claims

A defining feature of GUI-ReWalk is that supervision is synthesized retrospectively rather than supplied by manual trajectory annotation. The backward annotation step assigns a fine-grained instruction to each action-state pair and a high-level task summary to the full sequence, with the stated aim of eliminating costly and inconsistent human labeling (Lin et al., 19 Sep 2025).

The resulting dataset is reported as model-annotated, covering mobile and desktop environments, with 50k+ tasks and an average length of 22.5 steps. In the comparative dataset summary provided in the paper, AndroidControl is listed with 15,283 tasks and 5.5 average steps, GUI-Odyssey with 7,735 tasks and 15.3 average steps, and Mind2Web with 2,350 tasks and 7.3 average steps. GUI-ReWalk is accordingly presented as yielding longer and richer trajectories with broad multi-app coverage and more diverse application domains (Lin et al., 19 Sep 2025).

The paper also attributes several diversity properties to the framework. It reports higher trajectory entropy, broader flow coverage, and more realistic user intent, and it states that task recovery generates patterns of error and correction that further increase realism. Because the framework explicitly chains multiple strides across applications, it is not limited to isolated click-level samples or short single-app episodes. The intended output is long-horizon workflow data in which exploration, goal formation, correction, and cross-application continuation are all represented in the supervision signal.

A common simplification would be to treat GUI-ReWalk as a random-walk generator. The description in the paper does not support that interpretation: stochastic exploration is only the first phase, and it is progressively coupled to inferred goals, reflective recovery, and retrospective semantic annotation.

5. Benchmarks and reported performance

The paper evaluates GUI-ReWalk-7B, a 7B-parameter model built on Qwen2.5-VL-7B, on grounding benchmarks—ScreenSpot-Pro, OSWorld-G, and UI-Vision—and on navigation benchmarks—AndroidControl and GUI-Odyssey. The reported metrics include coverage, entropy, intent alignment, step success rate, and type accuracy (Lin et al., 19 Sep 2025).

On grounding, GUI-ReWalk-7B is reported to achieve 35.1 on ScreenSpot-Pro, 27.5 on OSWorld-G, and 5.9 on UI-Vision. The corresponding Qwen2.5-VL-7B baseline is listed at 20.8, 16.8, and 3.7, yielding improvements of 14.3 points on ScreenSpot-Pro, 10.7 points on OSWorld-G, and 2.2 points on UI-Vision (Lin et al., 19 Sep 2025).

On navigation, the paper reports 91.7 / 96.3 on AndroidControl-Low, 73.1 / 66.2 on AndroidControl-High, and 69.6 / 64.2 on GUI-Odyssey, where the paired values are type accuracy and step success rate. Relative to Qwen2.5-VL-7B, the paper highlights a +11.7 gain in step success rate on AndroidControl-Low, a +2.2 gain in type accuracy on AndroidControl-High, and +10.1 in type accuracy together with +17.9 in step success rate on GUI-Odyssey (Lin et al., 19 Sep 2025).

The empirical interpretation advanced in the paper is that combining randomness for diversity with goal-aware reasoning for structure produces training data with superior coverage of diverse interaction flows, higher trajectory entropy, and more realistic user intent. The results are used to support the claim that the framework is scalable and data-efficient for advancing GUI agent research.

6. Position within GUI-agent research

Within the broader GUI-agent literature represented in the cited papers, GUI-ReWalk occupies the trajectory-synthesis layer of the stack. Earlier work such as SeeClick isolates GUI grounding as a key bottleneck for screenshot-based agents and introduces ScreenSpot as a grounding benchmark (Cheng et al., 2024). Several later systems pursue more accurate grounding directly: ReGUIDE uses self-generated reasoning via online RL, spatial-aware criticism under transformations, and test-time spatial search (Lee et al., 21 May 2025); GUI-Cursor reframes grounding as an interactive search task with cursor feedback and multi-step online reinforcement learning (Zhao et al., 25 Sep 2025); POINTS-GUI-G emphasizes refined data engineering, continuous fine-tuning of the vision encoder, and RL with verifiable rewards (Zhao et al., 6 Feb 2026); and DRS-GUI proposes a training-free dynamic region search framework with Focus, Shift, Scatter, and an MCTS-based Action Planner (Liu et al., 15 May 2026).

A separate line of work addresses knowledge mismatch at inference time. GUIDE resolves domain bias in GUI agents through real-time web video retrieval and plug-and-play annotation, injecting planning and grounding knowledge without modifying model parameters or architecture (Xie et al., 27 Mar 2026). RAG-GUI retrieves web tutorials and generates concise, task-aware guidelines at inference time to augment a base VLM agent (Xu et al., 29 Sep 2025). SpiritSight, by contrast, emphasizes an end-to-end vision-based agent with GUI-Lasagne and Universal Block Parsing for cross-platform navigation from screenshots alone (Huang et al., 5 Mar 2025).

This suggests a useful division of labor. GUI-ReWalk is principally concerned with generating long-horizon, diverse, intent-aware training trajectories; ReGUIDE, GUI-Cursor, POINTS-GUI-G, DRS-GUI, and SeeClick concentrate on grounding precision; GUIDE and RAG-GUI emphasize retrieval-augmented adaptation at inference time; and SpiritSight targets end-to-end navigation with a grounding-oriented visual architecture. In that landscape, GUI-ReWalk is best understood not as a replacement for grounding or retrieval modules, but as a framework for supplying large-scale synthetic interaction data that may complement them.