- The paper introduces HiViG, an 8B multimodal critic that tracks compressed task history and verifies proposed GUI actions against screenshot coordinates before execution.
- The paper reports success gains from 31.0% to 38.3% for Qwen3-VL and from 41.4% to 50.4% for Gemini-3-Flash across evaluated computer-use benchmarks, outperforming scalar reward and verbal-critique baselines.
- The paper shows that grounded state-transition data, coordinate markers, and intent masking reduce hallucinated GUI reasoning, while history tracking and visual verification provide complementary error recovery benefits.
Motivation and problem statement
Computer Use Agents (CUAs) that operate Graphical User Interfaces through raw screenshots inevitably commit errors—misgrounded clicks, hallucinated UI elements, premature termination—and many GUI actions are irreversible, making post-hoc correction unsafe. Test-time intervention via critic models offers a way to intercept flawed actions before execution, but the authors identify two systematic deficiencies in existing critics: they are short-sighted, failing to track what has already been accomplished or failed across a long trajectory, and they lack visual grounding, relying on the policy's verbalized action intent rather than verifying the actual pixel coordinates against the current screenshot. The consequence is that critics can approve logically coherent but spatially misaligned actions, allowing silent failures to compound.
The paper also argues against scalar reward models (PRMs) as the primary intervention mechanism: in continuous parameter spaces such as pixel coordinates, when all candidate actions are flawed, a scalar score provides no gradient toward improvement and can trap the policy in low-reward regions. Verbal feedback, by contrast, can explain why an action fails and how to fix it.
The HiViG framework
HiViG (History-aware Visually Grounded test-time intervention) centers on an 8B multimodal critic, initialized from Qwen3-VL-8B-Thinking, that performs two functions at every timestep:
- History state tracking: before the policy plans its next move, the critic recursively compresses past micro-steps into a compact macro-action history of achieved goals (e.g., "opened the Downloads directory and confirmed the folder is empty"), giving the policy global task progress awareness.
- Visually grounded error analysis: after the policy proposes an action, the critic renders a visual marker at the proposed coordinates on the screenshot, verifies which UI element lies beneath it, predicts the visual state-transition the action would cause, and checks alignment with the instruction. Actions graded "Good" execute; actions graded "Bad" receive an error-dimension classification plus corrective verbal feedback, forcing the policy to refine before execution.
Training data construction
Both SFT datasets derive from open-source multi-domain trajectories in ScaleCUA, with Qwen3-VL-32B-Thinking as annotator, yielding 52k samples (20k history tracking, 32k error analysis split evenly between expert and perturbed actions). Two design choices deserve emphasis because they directly target known critic failure modes:
- Grounded state-transitions: rationales are conditioned on verbalized transitions extracted from actual observed screenshots rather than the annotator's parametric predictions. An ablation removing this grounding drops WebArenaLitev2 success from 25.3% to 20.8%, indicating that ungrounded "world modeling" during data generation distills hallucinated dynamics into the critic.
- Forced visual grounding: the policy's verbal intent is masked in 30% of training samples to break text-reliance, and a Set-of-Mark-style visual marker anchors reasoning to exact coordinates. Removing intent masking degrades AndroidLab from 51.5% to 47.1%; removing the marker degrades WebArenaLitev2 from 25.3% to 20.8%. The paper attributes the marker's importance to MLLMs' difficulty interpreting raw numerical coordinates in dense visual layouts.
Error synthesis follows a 12-dimension taxonomy spanning grounding/spatial errors (e.g., near-miss clicks, visual hallucination, polarity reversal in scrolls) and cognitive execution errors (e.g., procedural prerequisite neglect, termination misjudgment, timing neglect).
Empirical results
Evaluation covers WebArenaLitev2 (web), AndroidLab (mobile), and WindowsAgentArena (desktop), with Qwen3-VL-32B-Thinking and Gemini-3-Flash as policies. Baselines include Best-of-N PRM templates adapted from OpenCUA and SE-WSM, zero-shot CGI critique prompts, and the specialized GUI-Critic-R1.
| Policy |
Base agent |
Best baseline |
HiViG |
| Qwen3-VL-32B-Thinking |
31.0 |
32.5 |
38.3 |
| Gemini-3-Flash |
41.4 |
41.2 |
50.4 |
Several findings stand out. First, most baselines fail to help—or actively hurt—strong policies: for Gemini-3-Flash on WebArenaLitev2, interventions range from a 8.4-point degradation (GUI-Critic-R1, 22.1%) to no gain, whereas HiViG raises success from 30.5% to 45.5%. Scalar PRMs yield essentially no improvement anywhere, consistent with the argument that scores are uninformative when all candidates are poor. Second, gains persist even for a frontier closed-weight policy (+15.0% absolute on WebArenaLitev2), which the authors interpret as evidence that history tracking and error recovery remain unsolved even at the frontier. Third, category-level results are striking: Qwen3-VL-32B improves from 3.9% to 23.1% on WebArenaLitev2 Map tasks, and Gemini-3-Flash from 4.7% to 23.3% on WindowsAgentArena Office tasks.
Component ablations show each capability is independently useful but synergistic: on WebArenaLitev2 with Gemini-3-Flash, visually grounded analysis alone reaches 35.1% and history tracking alone 42.9%, versus 45.5% combined; task-level overlap analysis shows the components solve largely orthogonal failure sets, with the combination uniquely solving 4 tasks neither fixes alone. Notably, a single mixed-trained 8B critic matches the zero-shot history-tracking performance of its own 32B annotator model, indicating effective distillation.
Limitations and open questions
The authors concede several boundaries. The 12-dimension error taxonomy is bounded by current interface designs and policy capabilities; new error classes will require iterative expansion. Failure-case analysis reveals residual weaknesses: the critic misses fine-grained visual distinctions (e.g., "red" vs. "dark red" in Paint) and can record false achievements in macro-action history due to textual bias (mistaking a hidden configuration view for a deleted alarm). The intent-masking ratio is sensitive—a 50% ratio helps web tasks but hurts mobile, plausibly due to train/test distribution mismatch since test-time intents are unmasked. All reported results come from single runs owing to compute cost, so variance is not characterized. Open questions include whether macro-action compression remains reliable over substantially longer horizons than the 15–30 step limits evaluated here, and whether marker-based coordinate verification scales to higher-resolution or dynamically rendered interfaces.
Conclusion
HiViG demonstrates that pre-execution critique for CUAs benefits from two jointly trained capabilities—compressed macro-action history and marker-grounded coordinate verification—and that these address failure modes orthogonal to those targeted by scalar PRMs and prior verbal critics. The consistent cross-platform gains, including over frontier policies, support the claim that history awareness and visual grounding, rather than larger critic backbones, are the binding constraints on current test-time intervention.