VISTA: View-Consistent Self-Verified Training for GUI Grounding

Abstract: When applying Group Relative Policy Optimization (GRPO) for GUI Grounding, rollouts are sampled from a single screenshot view; groups often become either all failures on difficult instances or all successes on easy ones, yielding no useful relative advantage. We propose VISTA (View-Consistent Self-Verified Training), a GRPO-based training framework that constructs each comparison group from multiple target-preserving views of the same GUI instance.Each view is generated by a crop that keeps the target element visible and remaps its box exactly, so model rollouts are compared across semantically equivalent but geometrically different inputs. To stabilize short coordinate generation without turning reinforcement learning into unconditional imitation, VISTA further adds a self-verified cross-view anchor: an oracle answer optimized with an advantage-weighted loss, excluded from the group baseline and activated only when the model has produced a maximum-reward rollout. Across five GUI-grounding benchmarks and multiple Qwen backbones, VISTA consistently improves grounding accuracy.On ScreenSpot-Pro, it raises Qwen3-VL 4B/8B/30B-A3B from 55.5/52.7/53.7 to 63.4/65.8/67.0. Robustness analyses further show higher worst-view accuracy and lower prediction flip rates.

• The paper introduces a novel view-consistent framework that constructs diverse target-preserving views to overcome reward degeneracy in GUI grounding.
• It integrates self-verified cross-view anchoring where successful model outputs trigger supervised learning, ensuring higher stability and effective training.
• Empirical results across five benchmarks demonstrate up to 7.3 point accuracy gains and improved robustness compared to standard GRPO methods.

View-Consistent Self-Verified Training for GUI Grounding: VISTA

Motivation and Problem Formulation

Graphical User Interface (GUI) grounding tasks require mapping an image of a GUI and a natural language instruction to a precise coordinate, typically for autonomous agents acting on screens. Unlike conventional visual grounding, GUI grounding demands pixel-accurate localization due to small, dense, and visually similar interface elements: minor errors often result in incorrect actions. While Group Relative Policy Optimization (GRPO) with verifiable point-in-box rewards has advanced the state-of-the-art for GUI grounding, standard GRPO suffers from reward degeneracy. Specifically, repeated rollouts on a single fixed screenshot view produce groups with uniform (all-success or all-failure) binary rewards, eliminating the relative advantage needed for effective policy learning.

Figure 1: VISTA converts the degenerate, homogeneous rewards of fixed-view GRPO into informative cross-view variation by constructing rollout groups from target-preserving views.

The central issue is the construction of the comparison group in GRPO. When groups are comprised of rollouts from identical screenshot views, reward statistics either collapse to all-zero or all-one, stripping the policy optimization of meaningful gradient signals—particularly acute in coordinate-sensitive GUI grounding tasks.

The VISTA Framework

VISTA (View-Consistent Self-Verified Training) introduces a novel training framework for GUI grounding, built upon two main innovations:

1. View-Consistent Group Rollout: Instead of repeating rollouts on an identical screenshot, VISTA constructs each GRPO group from multiple target-preserving views—i.e., semantically matched but geometrically different crops—of the same GUI instance. Each crop ensures that the target element is always visible and its coordinates precisely remapped in the cropped context. This approach increases the fraction of informative training groups (from <5% to about 20%), mitigating reward degeneracy and enforcing cross-view robustness.
2. Self-Verified Cross-View Anchoring: To counter instability in short coordinate outputs—an artifact of RL in ambiguous or difficult views—VISTA adds a gated oracle anchor. Whenever at least one model-generated rollout achieves a maximum reward, an oracle-format answer (center of the target box) is appended to the group, with supervised learning enforced only in these self-verified, solvable instances. The anchor is strictly excluded from the group reward statistics, preserving purity of the RL learning signal.

   Figure 2: VISTA group construction: model rollouts are compared across multiple target-preserving views, with the self-verified oracle anchor used only when the model demonstrates capability in the group.

Experimental Evaluation and Results

VISTA was evaluated across five GUI grounding benchmarks—ScreenSpot-Pro, ScreenSpot-V2, MMBench-GUI L2, OSWorld-G, and OSWorld-G-R—using the Qwen3-VL backbone at 4B/8B/30B parameter scales and cross-verified on Qwen3.5-based models. Key findings include:

• ScreenSpot-Pro Accuracies: Substantial improvements over the Qwen3-VL baseline: from 55.5/52.7/53.7 to 63.4/65.8/67.0 on the 4B/8B/30B models respectively.
• Average Benchmark Gains: On all evaluated tasks, VISTA consistently outperforms standard GRPO, with average accuracy increases of approximately 4.4–7.3 points depending on model scale.
• Robustness: Beyond raw accuracy, VISTA shows higher worst-view accuracy, increased view consistency rate, and significantly reduced prediction flip rates, affirming enhanced stability under input perturbations.

Figure 3: ScreenSpot-Pro accuracy curves showing performance gains across training with VISTA.

Figure 4: Robustness analysis, including view consistency rate and prediction flip rate, highlighting improved cross-view stability.

Ablation studies demonstrate that neither standard crop augmentation in supervised fine-tuning, nor anchor supervision alone, provide the full benefit. The synergy of view-consistent groups and self-verified anchors is critical. Further ablations reveal that even on non-Qwen3-VL backbones (e.g., Qwen3.5), VISTA confers notable relative improvements, supporting its model-agnostic applicability.

Methodological and Theoretical Implications

VISTA's architectural advancements mark a substantive shift in how RL is applied to GUI grounding:

• Comparison Set Engineering: By recomposing the GRPO group from semantically equivalent, geometrically diverse inputs, VISTA provides a higher-variance, more informative learning signal. This cross-view strategy is aligned with the underlying challenge of GUI grounding, where robustness to superficial layout changes is essential.
• Conditional Oracle Anchoring: Gating oracle updates on observed model success precludes degenerate imitation learning, enhances stability, and prevents reward hacking or update explosion. This design strictly decouples RL from unconditional supervised updates, a prevalent pitfall in other GRPO variants for reasoning and math tasks.

Practical Implications and Future Directions

Practically, VISTA offers a more robust and scalable recipe for RLHF in GUI grounding, especially as real-world interfaces grow in resolution and complexity. With superior generalization to unseen view augmentations, VISTA-trained models are better candidates for deployment in autonomy-critical or user-facing digital agents. The framework is also directly compatible with inference-time multi-view prediction (MVP), with gains being additive, suggesting orthogonality of training and inference augmentations.

Theoretically, VISTA exposes new axes for RL-based interface grounding—most notably, the value of constructing diversified, semantically tied comparison sets for effective off-policy optimization. Future extensions could focus on adaptive view scheduling, dynamically learned augmentation policies, or broader application in grounded reasoning domains where reward signals are equally sparse or degenerate.

Conclusion

VISTA establishes a strong methodological foundation for GUI grounding with RLHF by leveraging view-consistent group construction and conditional oracle anchoring. Its consistent empirical gains in both accuracy and stability, model-agnostic transferability, and principled separation of RL and supervised objectives collectively point towards a general paradigm for robust agent training in interface-driven environments (2606.14579).