---
title: 'GUI-RobustEval: GUI Error Recovery Benchmark'
url: https://www.emergentmind.com/topics/gui-robusteval
type: topic
---

# GUI-RobustEval: GUI Error Recovery Benchmark

GUI-RobustEval is a benchmark for evaluating whether GUI agents can recognize and recover from **policy-induced errors**: mistakes generated by the agent’s own actions during execution, such as incorrect grounding, misinterpretation of the screen state, or a wrong subgoal. Introduced together with Robustness-driven Trajectory Synthesis (RoTS), it contains **1,216 executable test cases** derived from real failed trajectories in OSWorld-like desktop environments, and it measures both **Error-Awareness Rate** and **Post-Error Success Rate** under controlled error depths \(d \in \{0,1,3,5\}\) [2605.29447].

## 1. Conceptual basis and problem formulation

GUI-RobustEval adopts the standard POMDP view of GUI agents, with natural-language instructions \(\mathcal{U}\), actions \(\mathcal{A}\), environment states \(\mathcal{S}\), observations \(\mathcal{O}\), transition dynamics \(\mathcal{T}\), and reward model \(\mathcal{R}\). A GUI agent is a policy \(\pi_\theta\) that, at step \(i\), samples an action from the task instruction \(u\), current screenshot \(o_i\), and interaction history \(h_{i-1}\):
\[
a_i \sim \pi_\theta(\cdot \mid u, o_i, h_{i-1})
\]
GUI-RobustEval does not evaluate clean-start execution. Instead, it asks whether an agent can take over from a partially executed, erroneous trajectory with a known root-cause error and still complete the task [2605.29447].

Two definitions organize the benchmark. The **root-cause action** is the first action in a trajectory that commits the error. The **error horizon** is the minimal number of steps after that root cause required for the error to become identifiable. Recovery therefore has two components: first, detecting that the current state is inconsistent with the task; second, executing a corrective sequence that may require backtracking, strategy revision, or both. This makes GUI-RobustEval a benchmark of long-horizon recovery rather than only local misclick correction [2605.29447].

The benchmark is motivated by a gap in existing GUI evaluation. Benchmarks such as SeeClick, ScreenSpot, OSWorld, WindowsAgentArena, AndroidWorld, and WebArena mainly score grounding, single-step accuracy, or end-to-end task success from a clean initial state. Other robustness settings often emphasize external perturbations or synthetic disturbances. GUI-RobustEval instead focuses on realistic failure states produced by deployed agents themselves. This distinction is foundational: the benchmark measures whether an agent can recover from its own trajectory-induced divergence, not merely whether it can withstand exogenous noise [2605.29447].

## 2. Benchmark construction and executable test cases

GUI-RobustEval is built from approximately **1.5k failed trajectories** produced by **12 state-of-the-art OSWorld agents**, including Jedi-7B, o3, Mobile-Agent-V3, GUI-Owl-7B, UI-TARS, OpenCUA, Kimi-VL-A3B, Doubao-1.5-Thinking, and AutoGLM. Human experts inspect each failed trajectory, identify the root-cause action, and assign one or more error labels. To isolate the intended failure, any unrelated mistakes before the root cause are corrected, yielding an error-free prefix up to the failure point [2605.29447].

Each benchmark item is **executable** in the strong sense that it consists of a replayable system snapshot and a normalized prefix action sequence. Starting from a clean base snapshot, the framework replays verified pre-error steps, injects the root-cause action and its subsequent \(d\) steps, deterministically reconstructs the erroneous GUI state, and then hands control to the evaluated agent. Formally, evaluation at depth \(d\) uses a prefix
\[
\tau_{\text{prefix}} = (o_1,a_1,o_2,\dots,o_k,a_k,o_{k+1})
\]
that includes all corrected pre-error steps, the root-cause step, and \(d\) post-error steps [2605.29447].

The benchmark contains **1,216 test cases** spanning **11 error types** and **four error depths**, \(d=0,1,3,5\). The environments use the same online desktop infrastructure as OSWorld and WindowsAgentArena, primarily Ubuntu, with realistic applications such as LibreOffice Impress, GIMP, browsers, terminal, and file manager. Reproducibility is enforced through task-specific base snapshots and by disabling updates and notifications that would otherwise break replay fidelity [2605.29447].

A critical engineering choice is **action normalization**. Because the source agents expose heterogeneous action APIs and chain-of-thought formats, the benchmark converts steps into a canonical representation of “action summary + PyAutoGUI” code. At evaluation time, this normalized history is converted back into the tested agent’s native format for prompt injection. This allows a single benchmark to compare agents with different prompting and action interfaces while preserving the same erroneous prefix state [2605.29447].

## 3. Error taxonomy and controlled difficulty

GUI-RobustEval uses a **multi-label taxonomy of 11 error types** assigned to the root-cause step. These categories were derived from manual analysis of real failed trajectories and are intended to cover both low-level execution failures and high-level planning errors. The taxonomy spans grounding, semantics, strategy, and termination behavior, which is why the benchmark can probe much more than local visual grounding [2605.29447].

| Error type | Description |
|---|---|
| Incorrect UI Element | Interact with the wrong UI element even though the correct target is visible |
| Grounding Failure | Intends the correct action but mis-executes it |
| Ineffective Action | The action causes no change to the environment state |
| Typing Error | The text typed is incorrect |
| Miss Necessary Step | Skips a critical action required for success |
| Incorrect Tool Usage | Uses an invalid or contextually inappropriate tool |
| Wrong Target | Operates on the wrong object |
| Incorrect Parameter | Correct operation and target, but wrong value or option |
| Misunderstand Task Objective | Global misinterpretation of the user’s goal |
| Fail to Terminate | Task is already done, or impossible, but the agent continues to act |
| Lack of Knowledge | Wrong or inefficient high-level strategy due to insufficient domain or app knowledge |

Error depth adds a second difficulty axis. For each root cause, the benchmark evaluates takeover at the moment of error (\(d=0\)) and after the error has propagated for \(1\), \(3\), or \(5\) additional steps. This makes the same underlying failure observable at different stages of compounding. As depth increases, the visible state drifts farther from the intended goal and the injected erroneous history becomes more misleading, so the task shifts from immediate correction toward long-horizon diagnosis and replanning [2605.29447].

The benchmark’s hardest classes are not the low-level ones. Per-error-type analysis for RoTS-32B reports **17.4% success** on **Fail to Terminate** and **24.1% success** on **Misunderstand Task Objective**, whereas improvements are larger on categories such as **Ineffective Action** and **Miss Necessary Step**. This suggests that progress perception and task-objective understanding remain major unresolved problems even when recovery data are available [2605.29447].

## 4. Evaluation protocol and metrics

For each test case, evaluation begins by resetting the OS to the task’s base snapshot and replaying the verified prefix actions. The tested agent then receives the corresponding injected history in its own action and chain-of-thought format and takes over from the reproduced erroneous state. A total step budget of **50** is enforced, including the erroneous prefix. Each test case is run **3 independent times** per agent and metrics are averaged over runs [2605.29447].

GUI-RobustEval reports two primary metrics. **Error-Awareness Rate** measures whether the agent’s first thought after takeover recognizes that a previous action was wrong or that the current state is inconsistent with the goal. **Post-Error Success Rate** measures whether the agent eventually completes the task from the erroneous prefix under the step budget. The first is judged by **Qwen3-VL-Plus**, with reported human agreement of **at least 96%** across agents; the second uses the same reward-model infrastructure as OSWorld, largely WebJudge-like LLM-as-a-judge evaluation [2605.29447].

\[
\text{Awareness} = \mathbb{E}[\mathbf{1}\{\text{judge says agent recognized prior error}\}]
\]

\[
\text{Post-Error Success} = \mathbb{E}[\mathbf{1}\{\mathcal{R}(u,\tau)=1\}]
\]

These metrics are correlated but not identical. The benchmark reports cases where models have similar awareness but different post-error success because awareness alone does not guarantee grounding precision, backtracking competence, or replanning quality. This distinction is important because it separates diagnosis from execution: a model may correctly notice that something is wrong and still fail to recover [2605.29447].

A common misconception is that final task success from a clean start already captures recovery ability. GUI-RobustEval rejects that premise by construction. Since every trial starts from a replayed erroneous prefix, the benchmark directly measures post-error competence rather than conflating it with initial planning quality. This yields a cleaner estimate of how well agents recover once they have already deviated [2605.29447].

## 5. Coupling with RoTS and empirical findings

GUI-RobustEval is tightly coupled to **Robustness-driven Trajectory Synthesis (RoTS)**, a data-generation framework designed to create failure–recovery trajectories aligned with the benchmark’s error distribution. RoTS builds a trajectory tree \(T=(O,A,E)\) over observations, actions, and edges, and expands it using three mechanisms: **Parallel Sampling**, **Fragility-driven Exploration**, and **Experience-informed Recovery**. The resulting training set contains **800k** samples, comprising **720k** reflection-agnostic samples and **80k** reflection-related samples with \(\lambda_{\text{ref}}=0.1\) [2605.29447].

RoTS produces two models, **RoTS-7B** and **RoTS-32B**, by supervised fine-tuning Qwen2.5-VL-7B and Qwen2.5-VL-32B. On GUI-RobustEval, the benchmark’s main open-source comparison reports the following post-error success rates across depths \(d=0,1,3,5\): **RoTS-32B** scores **49.7, 41.8, 36.5, 33.2**, while **RoTS-7B** scores **43.5, 36.6, 30.1, 26.7**. Their awareness rates are **58.8** and **51.9**, respectively. Among the listed open-source agents, RoTS-32B shows both the highest success and the lowest relative degradation from \(d=0\) to \(d=5\), with a **33%** drop versus **53%** for OpenCUA-32B [2605.29447].

The benchmark also supports claims about transfer to clean-start task performance. On OSWorld, **RoTS-32B** achieves **47.4%** success at 50 steps and **33.8% All-Pass@4**, exceeding the reported **34.1%** success and **15.5% All-Pass@4** of OpenCUA-32B. This suggests that improved long-horizon error recovery contributes not only to benchmarked robustness but also to stronger end-to-end reliability [2605.29447].

RoTS ablations further connect training data structure to GUI-RobustEval outcomes. A depth-cap study shows that adding recovery data with depth \(\le 5\) improves post-error success from **12.1** to **20.7**, while gains beyond depth 7 are modest. A plausible implication is that moderate-depth recovery trajectories capture much of the practically useful signal, whereas extremely long-horizon recovery is costlier and yields diminishing returns [2605.29447].

## 6. Position within the broader GUI robustness landscape

GUI-RobustEval occupies one specific part of a broader robustness taxonomy. It targets **policy-induced errors** in online desktop environments; neighboring benchmarks target other axes. **GUI-Perturbed** evaluates robustness of single-step GUI grounding by independently perturbing the visual scene and the instruction, reporting **3,120 evaluation samples** and showing that relational instructions can cause **27–56 percentage point** accuracy drops while **70% zoom** also degrades performance [2604.14262]. **GUI-Robust** evaluates agents on **real-world anomalies** such as login pages, CAPTCHA pages, ad pop-ups, cookie consent pop-ups, page loading delay, and network disconnection, using **5,318 tasks** including **200 abnormal tasks** [2506.14477].

Other work defines robustness through reflection, verification, or temporal understanding. **GUI-Reflection** introduces a reflection-oriented task suite centered on **Action Verification**, **Action Reversal**, and **Mistake-Informed Reattempt**, together with mobile online reflection tuning [2506.08012]. **VeriGUI** builds a failure-recognition benchmark on AndroidControl and evaluates **Loop Rate** and **Recovery Success Rate**, emphasizing action-effect verification and self-correction under noisy environments [2604.05477]. **GUI-World** moves the problem into the video domain, with **12,379 videos** across **six GUI scenarios** and question types designed around dynamic content, prediction, sequential reasoning, and conversation [2406.10819].

These benchmarks are complementary rather than interchangeable. GUI-RobustEval tests whether an agent can recover from its own already-executed mistakes after a replayed erroneous prefix. GUI-Perturbed isolates perceptual and instruction brittleness at the grounding level. GUI-Robust focuses on exogenous anomalies. GUI-Reflection and VeriGUI emphasize self-checking and correction mechanisms. GUI-World stresses temporal GUI understanding. This suggests that “robust GUI evaluation” is not a single metric family but a collection of orthogonal capability probes spanning grounding, anomaly handling, temporal reasoning, and post-error recovery [2605.29447].

## 7. Limitations and future directions

GUI-RobustEval has several explicit limitations. Its current focus is **desktop** computer-use tasks, primarily Ubuntu, so mobile and edge-device recovery are not yet covered. The benchmark requires **history injection and format conversion** for each agent, which introduces engineering complexity and may create mild biases even though conversion is applied consistently across depths for a given model. Its **11 error categories** cover many real failures but are not exhaustive, and the benchmark size of **1,216 test cases** is substantial yet still limited relative to the diversity of possible GUI failures [2605.29447].

Future directions follow naturally from those constraints. The benchmark can be extended to **mobile GUIs**, **edge devices**, and **cross-device workflows**. The associated training framework suggests a **data flywheel** or **RL**-driven regime in which the distribution of policy-induced errors and recoveries evolves as agents improve. Another likely direction is tighter coupling between benchmark metrics such as awareness and post-error success and explicit training objectives, including reflection objectives, process supervision, or reward shaping [2605.29447].

A broader methodological implication is that robustness should be factorized. GUI-RobustEval already isolates recovery from self-induced trajectory errors; neighboring work indicates additional axes such as visual perturbation, instruction perturbation, anomaly detection, and temporal consistency. A mature robustness suite would therefore combine executable erroneous prefixes, controlled perturbations, anomaly scenarios, and temporal GUI tasks into a single evaluation ecosystem rather than reducing robustness to one clean-start success number [2605.29447].

Source: https://www.emergentmind.com/topics/gui-robusteval