Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents

Abstract: While GUI agents have advanced rapidly, they often lack the robustness to recover from their own errors, hindering real-world deployment. To bridge this gap at both the evaluation and data levels, we introduce GUI-RobustEval and propose Robustness-driven Trajectory Synthesis. GUI-RobustEval contains $1,216$ executable test cases that systematically measure error recovery capabilities across a broad and realistic spectrum of error modes. At the data level, RoTS is a scalable synthesis framework that creates $800k$ high-quality data via a tree-based pipeline that proactively discovers diverse error modes and synthesizes corresponding recovery steps. Our two models, RoTS-7B and RoTS-32B, fine-tuned on our dataset, both demonstrate significant gains on GUI-RobustEval and traditional GUI benchmarks. Notably, RoTS-32B achieves state-of-the-art performance on OSWorld, with a $47.4\%$ success rate and a $33.8\%$ All-Pass@4 score, suggesting that improved long-horizon error recovery ability contributes to both robustness and overall performance. Our code is available at https://github.com/AlibabaResearch/RoTS.

• The paper introduces GUI-RobustEval and the RoTS pipeline to systematically recover from policy-induced errors in GUI agents.
• The methodology combines Fragility-Driven Exploration and Experience-Informed Recovery for scalable, long-horizon error recovery.
• Empirical results demonstrate that RoTS-32B enhances error awareness and post-error success, outperforming traditional benchmarks.

Robust Error Recovery in GUI Agents: Benchmarking and Synthesis Approaches

Motivation and Problem Formulation

The automation of GUI tasks by agents based on VLMs has led to notable advances in computer-use environments. Despite this progress, GUI agents frequently suffer from policy-induced errors during real-world operation, including perception and planning failures, which compromise robustness and hinder deployment. The inability to detect and recover from such errors remains under-addressed within both evaluation protocols and training data paradigms. Existing benchmarks favor synthetic perturbations or noise injection and emphasize task success or grounding accuracy, but lack coverage and long-horizon recovery metrics for policy-induced mistakes.

Figure 1: Policy-induced errors exhibit diverse types and delayed error detectability, with RoTS achieving lower accuracy drop on All-Pass@4 compared to methods trained on synthetic or human-curated errors.

This work introduces two contributions: GUI-RobustEval, a benchmark focused on measuring agent robustness for policy-induced failures, and RoTS, a scalable data synthesis pipeline generating diverse error-recovery trajectories.

GUI-RobustEval: Benchmark Design and Analysis

GUI-RobustEval is constructed via a systematic process, analyzing failed trajectories from 12 SOTA agents to extract realistic error modes and depths. Human experts annotate root-cause steps and error depths, subsequently normalizing trajectory histories across agent architectures for controlled evaluation. The benchmark comprises 1,216 test cases spanning 11 representative error types and four error depths ($d \in \{0, 1, 3, 5\}$). For each test case, the agent receives an injected erroneous prefix and must recognize and recover from the error, with evaluation conducted via VLM-based judges.

Figure 2: Overview of the GUI-RobustEval benchmark construction and RoTS data synthesis pipeline for error-recovery trajectory generation.

Figure 3: Distribution of policy-induced error types and horizons across existing datasets and GUI-RobustEval; recovery difficulty increases with error depth and certain error types.

Analysis reveals that training datasets concentrate on low-level, human-execution errors and short horizons, whereas real policy-induced failures in deployment are often compositional, high-level, and delayed in detectability. GUI-RobustEval exposes this coverage/horizon mismatch and provides fine-grained metrics: Error-Awareness Rate (agents recognize error at takeover) and Post-Error Success Rate (agents successfully recover and complete the task).

RoTS: Tree-Based Error-Recovery Data Synthesis

RoTS leverages a trajectory tree framework for scalable, online synthesis of failure-recovery data. The pipeline operates in two jointly expanded modes:

• Fragility-Driven Exploration (FDE): Explores fragile states identified via progress critic scoring, branching out from successful rollouts to discover new policy-induced error modes.
• Experience-Informed Recovery (EIR): Localizes failures and synthesizes recovery advice by aggregating neighbor-branch experiences, enabling the recovery actor to learn correction strategies from diverse contexts.

This co-expansion schema efficiently produces rich long-horizon recovery trajectories, better matching failure distributions observed in real execution.

Figure 4: Visualization of the FAR-Tree, where FDE discovers new errors and EIR achieves recovery via advice-conditioned rollouts.

Data cleaning is achieved through VLM-based filtering, critic modules for plan and execution correctness, reflection identification, and rule-based deduplication. The resulting 800k-sample dataset contains both reflection-agnostic and reflection-related trajectories, with mixing controlled by the $\lambda_\text{ref}$ parameter.

Numerical Results and Empirical Evaluation

RoTS models fine-tuned on synthesized datasets (RoTS-7B, RoTS-32B) demonstrate superior performance across GUI-RobustEval and traditional benchmarks (OSWorld, WindowsAgentArena), particularly for long-horizon error recovery. RoTS-32B achieves $33.2\%$ post-error success rate at error depth 5 and $58.8\%$ error-awareness, showing the lowest performance drop compared to competing baselines. On OSWorld (max steps 50), RoTS-32B attains $47.4\%$ success rate and $33.8\%$ All-Pass@4, indicating robust recoverability and stability.

Ablation studies highlight the critical role of both FDE and EIR for error diversity and recovery robustness. Policy-induced reflection data outperform human-execution reflection in improving error recovery, confirming the necessity for realistic synthesis.

Figure 5: Performance sensitivity to ratio of reflection data, with optimal $\lambda_\text{ref}=0.1$ balancing robustness and effectiveness.

Figure 6: RoTS performance scales with increased expansion rounds and dataset size, with gains plateauing at $1000$k samples due to the diversity limitation of current tree expansion settings.

Figure 7: Trajectory comparison between OpenCUA and RoTS models on GUI-RobustEval, demonstrating effective error identification and recovery by RoTS.

Qualitative Error Case Studies

RoTS was assessed in multiple scenarios highlighting complex error modes:

• Incorrect Parameter and Miss Necessary Step: Agents failed to specify correct file paths or omitted necessary dialog actions, resulting in task incompletion.
• Incorrect UI Element Selection and Compositional Errors: Initial perception mistakes led to chains of erroneous actions, revealing that real recovery requires multi-step reasoning and re-planning.

  Figure 8: Examples of Incorrect Parameter and Miss Necessary Step errors illustrating multi-step recovery requirements.

  

  Figure 9: Case studies of Incorrect UI Element and Compositional Errors, highlighting recovery complexity in multi-modal GUI environments.

Error analysis confirms RoTS-32B achieves broad improvement across all error categories, with the most significant gains on high-level planning and ineffective actions. However, bottlenecks remain for fail-to-terminate and task objective misinterpretation, indicating avenues for further architectural and training enhancements.

Figure 10: RoTS model showcases exploration, flexible strategy switching, and multi-step error recovery on OSWorld tasks, with visualization of correct, erroneous, and recovery actions.

Implications and Future Prospects

Practical implications of this research are substantial. Reliable GUI agent deployment in enterprise or consumer environments mandates robustness to internal execution errors, especially those not readily detectable or recoverable by traditional models. RoTS and GUI-RobustEval establish benchmarks and data paradigms for such resilience, enabling safer and more autonomous operation.

Theoretically, the pipeline's tree-based synthesis aligns with POMDP exploration strategies. The combination of critic-driven fragility, neighbor-experience aggregation, and advice-conditioned recovery expands the repertoire of agentic supervision, suggesting methods generalizable to other agent modalities and environments. RoTS’s advances point toward scalable autonomous self-training ("data flywheel") and iterative RL pipelines for continual improvement, essential for evolving AI systems.

Figure 11: Example of over-reflection behavior, demonstrating a challenge in balancing corrective actions with efficient task completion.

Conclusion

This work provides a rigorous benchmarking and synthesis framework addressing robustness deficiencies pervasive in contemporary GUI agents. By targeting policy-induced error coverage and horizon mismatches, and deploying high-throughput, critic-informed trajectory tree synthesis, RoTS models decisively improve error awareness and recovery. The methodology establishes new standards for robust GUI agent evaluation and training, and informs both practical deployment strategies and future research in agent reliability and autonomous learning (2605.29447).