GUI-RobustEval: GUI Error Recovery Benchmark

• GUI-RobustEval is a benchmark that evaluates GUI agents' capability to detect and recover from policy-induced errors through reproducible, executable test cases.
• It employs controlled error depths (0, 1, 3, 5) and measures both Error-Awareness Rate and Post-Error Success Rate to assess long-horizon recovery performance.
• Built from 1,216 real failed trajectories across 11 error types, it advances robustness evaluation beyond traditional clean-start execution methods.

GUI-RobustEval is a benchmark for evaluating whether GUI agents can recognize and recover from policy-induced errors: mistakes generated by the agent’s own actions during execution, such as incorrect grounding, misinterpretation of the screen state, or a wrong subgoal. Introduced together with Robustness-driven Trajectory Synthesis (RoTS), it contains 1,216 executable test cases derived from real failed trajectories in OSWorld-like desktop environments, and it measures both Error-Awareness Rate and Post-Error Success Rate under controlled error depths $d \in \{0,1,3,5\}$ (Bu et al., 28 May 2026).

1. Conceptual basis and problem formulation

GUI-RobustEval adopts the standard POMDP view of GUI agents, with natural-language instructions $\mathcal{U}$, actions $\mathcal{A}$, environment states $\mathcal{S}$, observations $\mathcal{O}$, transition dynamics $\mathcal{T}$, and reward model $\mathcal{R}$. A GUI agent is a policy $\pi_\theta$ that, at step $i$, samples an action from the task instruction $u$, current screenshot $\mathcal{U}$0, and interaction history $\mathcal{U}$1: $\mathcal{U}$2 GUI-RobustEval does not evaluate clean-start execution. Instead, it asks whether an agent can take over from a partially executed, erroneous trajectory with a known root-cause error and still complete the task (Bu et al., 28 May 2026).

Two definitions organize the benchmark. The root-cause action is the first action in a trajectory that commits the error. The error horizon is the minimal number of steps after that root cause required for the error to become identifiable. Recovery therefore has two components: first, detecting that the current state is inconsistent with the task; second, executing a corrective sequence that may require backtracking, strategy revision, or both. This makes GUI-RobustEval a benchmark of long-horizon recovery rather than only local misclick correction (Bu et al., 28 May 2026).

The benchmark is motivated by a gap in existing GUI evaluation. Benchmarks such as SeeClick, ScreenSpot, OSWorld, WindowsAgentArena, AndroidWorld, and WebArena mainly score grounding, single-step accuracy, or end-to-end task success from a clean initial state. Other robustness settings often emphasize external perturbations or synthetic disturbances. GUI-RobustEval instead focuses on realistic failure states produced by deployed agents themselves. This distinction is foundational: the benchmark measures whether an agent can recover from its own trajectory-induced divergence, not merely whether it can withstand exogenous noise (Bu et al., 28 May 2026).

2. Benchmark construction and executable test cases

GUI-RobustEval is built from approximately 1.5k failed trajectories produced by 12 state-of-the-art OSWorld agents, including Jedi-7B, o3, Mobile-Agent-V3, GUI-Owl-7B, UI-TARS, OpenCUA, Kimi-VL-A3B, Doubao-1.5-Thinking, and AutoGLM. Human experts inspect each failed trajectory, identify the root-cause action, and assign one or more error labels. To isolate the intended failure, any unrelated mistakes before the root cause are corrected, yielding an error-free prefix up to the failure point (Bu et al., 28 May 2026).

Each benchmark item is executable in the strong sense that it consists of a replayable system snapshot and a normalized prefix action sequence. Starting from a clean base snapshot, the framework replays verified pre-error steps, injects the root-cause action and its subsequent $\mathcal{U}$3 steps, deterministically reconstructs the erroneous GUI state, and then hands control to the evaluated agent. Formally, evaluation at depth $\mathcal{U}$4 uses a prefix

$\mathcal{U}$5

that includes all corrected pre-error steps, the root-cause step, and $\mathcal{U}$6 post-error steps (Bu et al., 28 May 2026).

The benchmark contains 1,216 test cases spanning 11 error types and four error depths, $\mathcal{U}$7. The environments use the same online desktop infrastructure as OSWorld and WindowsAgentArena, primarily Ubuntu, with realistic applications such as LibreOffice Impress, GIMP, browsers, terminal, and file manager. Reproducibility is enforced through task-specific base snapshots and by disabling updates and notifications that would otherwise break replay fidelity (Bu et al., 28 May 2026).

A critical engineering choice is action normalization. Because the source agents expose heterogeneous action APIs and chain-of-thought formats, the benchmark converts steps into a canonical representation of “action summary + PyAutoGUI” code. At evaluation time, this normalized history is converted back into the tested agent’s native format for prompt injection. This allows a single benchmark to compare agents with different prompting and action interfaces while preserving the same erroneous prefix state (Bu et al., 28 May 2026).

3. Error taxonomy and controlled difficulty

GUI-RobustEval uses a multi-label taxonomy of 11 error types assigned to the root-cause step. These categories were derived from manual analysis of real failed trajectories and are intended to cover both low-level execution failures and high-level planning errors. The taxonomy spans grounding, semantics, strategy, and termination behavior, which is why the benchmark can probe much more than local visual grounding (Bu et al., 28 May 2026).

Error type	Description
Incorrect UI Element	Interact with the wrong UI element even though the correct target is visible
Grounding Failure	Intends the correct action but mis-executes it
Ineffective Action	The action causes no change to the environment state
Typing Error	The text typed is incorrect
Miss Necessary Step	Skips a critical action required for success
Incorrect Tool Usage	Uses an invalid or contextually inappropriate tool
Wrong Target	Operates on the wrong object
Incorrect Parameter	Correct operation and target, but wrong value or option
Misunderstand Task Objective	Global misinterpretation of the user’s goal
Fail to Terminate	Task is already done, or impossible, but the agent continues to act
Lack of Knowledge	Wrong or inefficient high-level strategy due to insufficient domain or app knowledge

Error depth adds a second difficulty axis. For each root cause, the benchmark evaluates takeover at the moment of error ($\mathcal{U}$8) and after the error has propagated for $\mathcal{U}$9, $\mathcal{A}$0, or $\mathcal{A}$1 additional steps. This makes the same underlying failure observable at different stages of compounding. As depth increases, the visible state drifts farther from the intended goal and the injected erroneous history becomes more misleading, so the task shifts from immediate correction toward long-horizon diagnosis and replanning (Bu et al., 28 May 2026).

The benchmark’s hardest classes are not the low-level ones. Per-error-type analysis for RoTS-32B reports 17.4% success on Fail to Terminate and 24.1% success on Misunderstand Task Objective, whereas improvements are larger on categories such as Ineffective Action and Miss Necessary Step. This suggests that progress perception and task-objective understanding remain major unresolved problems even when recovery data are available (Bu et al., 28 May 2026).

4. Evaluation protocol and metrics

For each test case, evaluation begins by resetting the OS to the task’s base snapshot and replaying the verified prefix actions. The tested agent then receives the corresponding injected history in its own action and chain-of-thought format and takes over from the reproduced erroneous state. A total step budget of 50 is enforced, including the erroneous prefix. Each test case is run 3 independent times per agent and metrics are averaged over runs (Bu et al., 28 May 2026).

GUI-RobustEval reports two primary metrics. Error-Awareness Rate measures whether the agent’s first thought after takeover recognizes that a previous action was wrong or that the current state is inconsistent with the goal. Post-Error Success Rate measures whether the agent eventually completes the task from the erroneous prefix under the step budget. The first is judged by Qwen3-VL-Plus, with reported human agreement of at least 96% across agents; the second uses the same reward-model infrastructure as OSWorld, largely WebJudge-like LLM-as-a-judge evaluation (Bu et al., 28 May 2026).

$\mathcal{A}$2

$\mathcal{A}$3

These metrics are correlated but not identical. The benchmark reports cases where models have similar awareness but different post-error success because awareness alone does not guarantee grounding precision, backtracking competence, or replanning quality. This distinction is important because it separates diagnosis from execution: a model may correctly notice that something is wrong and still fail to recover (Bu et al., 28 May 2026).

A common misconception is that final task success from a clean start already captures recovery ability. GUI-RobustEval rejects that premise by construction. Since every trial starts from a replayed erroneous prefix, the benchmark directly measures post-error competence rather than conflating it with initial planning quality. This yields a cleaner estimate of how well agents recover once they have already deviated (Bu et al., 28 May 2026).

5. Coupling with RoTS and empirical findings

GUI-RobustEval is tightly coupled to Robustness-driven Trajectory Synthesis (RoTS), a data-generation framework designed to create failure–recovery trajectories aligned with the benchmark’s error distribution. RoTS builds a trajectory tree $\mathcal{A}$4 over observations, actions, and edges, and expands it using three mechanisms: Parallel Sampling, Fragility-driven Exploration, and Experience-informed Recovery. The resulting training set contains 800k samples, comprising 720k reflection-agnostic samples and 80k reflection-related samples with $\mathcal{A}$5 (Bu et al., 28 May 2026).

RoTS produces two models, RoTS-7B and RoTS-32B, by supervised fine-tuning Qwen2.5-VL-7B and Qwen2.5-VL-32B. On GUI-RobustEval, the benchmark’s main open-source comparison reports the following post-error success rates across depths $\mathcal{A}$6: RoTS-32B scores 49.7, 41.8, 36.5, 33.2, while RoTS-7B scores 43.5, 36.6, 30.1, 26.7. Their awareness rates are 58.8 and 51.9, respectively. Among the listed open-source agents, RoTS-32B shows both the highest success and the lowest relative degradation from $\mathcal{A}$7 to $\mathcal{A}$8, with a 33% drop versus 53% for OpenCUA-32B (Bu et al., 28 May 2026).

The benchmark also supports claims about transfer to clean-start task performance. On OSWorld, RoTS-32B achieves 47.4% success at 50 steps and 33.8% All-Pass@4, exceeding the reported 34.1% success and 15.5% All-Pass@4 of OpenCUA-32B. This suggests that improved long-horizon error recovery contributes not only to benchmarked robustness but also to stronger end-to-end reliability (Bu et al., 28 May 2026).

RoTS ablations further connect training data structure to GUI-RobustEval outcomes. A depth-cap study shows that adding recovery data with depth $\mathcal{A}$9 improves post-error success from 12.1 to 20.7, while gains beyond depth 7 are modest. A plausible implication is that moderate-depth recovery trajectories capture much of the practically useful signal, whereas extremely long-horizon recovery is costlier and yields diminishing returns (Bu et al., 28 May 2026).

6. Position within the broader GUI robustness landscape

GUI-RobustEval occupies one specific part of a broader robustness taxonomy. It targets policy-induced errors in online desktop environments; neighboring benchmarks target other axes. GUI-Perturbed evaluates robustness of single-step GUI grounding by independently perturbing the visual scene and the instruction, reporting 3,120 evaluation samples and showing that relational instructions can cause 27–56 percentage point accuracy drops while 70% zoom also degrades performance (Wang et al., 15 Apr 2026). GUI-Robust evaluates agents on real-world anomalies such as login pages, CAPTCHA pages, ad pop-ups, cookie consent pop-ups, page loading delay, and network disconnection, using 5,318 tasks including 200 abnormal tasks (Yang et al., 17 Jun 2025).

Other work defines robustness through reflection, verification, or temporal understanding. GUI-Reflection introduces a reflection-oriented task suite centered on Action Verification, Action Reversal, and Mistake-Informed Reattempt, together with mobile online reflection tuning (Wu et al., 9 Jun 2025). VeriGUI builds a failure-recognition benchmark on AndroidControl and evaluates Loop Rate and Recovery Success Rate, emphasizing action-effect verification and self-correction under noisy environments (Zhang et al., 7 Apr 2026). GUI-World moves the problem into the video domain, with 12,379 videos across six GUI scenarios and question types designed around dynamic content, prediction, sequential reasoning, and conversation (Chen et al., 2024).

These benchmarks are complementary rather than interchangeable. GUI-RobustEval tests whether an agent can recover from its own already-executed mistakes after a replayed erroneous prefix. GUI-Perturbed isolates perceptual and instruction brittleness at the grounding level. GUI-Robust focuses on exogenous anomalies. GUI-Reflection and VeriGUI emphasize self-checking and correction mechanisms. GUI-World stresses temporal GUI understanding. This suggests that “robust GUI evaluation” is not a single metric family but a collection of orthogonal capability probes spanning grounding, anomaly handling, temporal reasoning, and post-error recovery (Bu et al., 28 May 2026).

7. Limitations and future directions

GUI-RobustEval has several explicit limitations. Its current focus is desktop computer-use tasks, primarily Ubuntu, so mobile and edge-device recovery are not yet covered. The benchmark requires history injection and format conversion for each agent, which introduces engineering complexity and may create mild biases even though conversion is applied consistently across depths for a given model. Its 11 error categories cover many real failures but are not exhaustive, and the benchmark size of 1,216 test cases is substantial yet still limited relative to the diversity of possible GUI failures (Bu et al., 28 May 2026).

Future directions follow naturally from those constraints. The benchmark can be extended to mobile GUIs, edge devices, and cross-device workflows. The associated training framework suggests a data flywheel or RL-driven regime in which the distribution of policy-induced errors and recoveries evolves as agents improve. Another likely direction is tighter coupling between benchmark metrics such as awareness and post-error success and explicit training objectives, including reflection objectives, process supervision, or reward shaping (Bu et al., 28 May 2026).

A broader methodological implication is that robustness should be factorized. GUI-RobustEval already isolates recovery from self-induced trajectory errors; neighboring work indicates additional axes such as visual perturbation, instruction perturbation, anomaly detection, and temporal consistency. A mature robustness suite would therefore combine executable erroneous prefixes, controlled perturbations, anomaly scenarios, and temporal GUI tasks into a single evaluation ecosystem rather than reducing robustness to one clean-start success number (Bu et al., 28 May 2026).