Graphical User Interface Visual Grounding

• GUI-VG is a visual grounding task where multimodal language models precisely localize GUI elements based on natural language queries.
• Contemporary methods leverage autoregressive, diffusion-based, and multi-stage frameworks to overcome challenges like complex layouts, tiny elements, and ambiguous instructions.
• Benchmarking with metrics such as point-in-box accuracy and mIoU demonstrates the efficiency and generalization of GUI-VG across diverse platforms.

Graphical User Interface Visual Grounding (GUI-VG) is the core capability by which artificial agents, typically built upon multimodal LLMs (MLLMs), map free-form natural language instructions to precise on-screen regions or coordinates within a graphical user interface (GUI). As GUI automation, digital agency, and virtual assistant technologies mature, robust GUI-VG undergirds a new class of agents that interact entirely through visual observations, eschewing structured representations such as HTML or accessibility trees in favor of direct pixel-level grounding. Modern research frames GUI-VG as either a coordinate or region prediction task, exhaustively benchmarked on diverse platforms (web, desktop, mobile) with fine-grained requirements for accuracy, generalization, and efficiency.

1. Problem Definition, Challenges, and Data Formulation

GUI-VG takes as input a screenshot $I \in \mathbb{R}^{H\times W\times 3}$ and a natural language query $q$ (referred to as an instruction or referring expression), outputting a click coordinate $(x^*, y^*)$ or a bounding box $(x_1^*, y_1^*, x_2^*, y_2^*)$ corresponding to the designated GUI element (Cheng et al., 2024, Gou et al., 2024, Kang et al., 6 Aug 2025). Expressed formally:

$$f_\theta: (I, q) \longrightarrow (x_1^*, y_1^*, x_2^*, y_2^*)$$

or for point-based methods:

$f_\theta: (I, q) \longrightarrow (x^*, y^*)$

The task imposes pronounced technical demands:

• Interface Complexity: GUIs are information-dense, hierarchical, and populated by small, visually similar elements (e.g., buttons, icons, tabs) interlaced with complex, unpredictable layouts and window states (Li et al., 1 Dec 2025, Kwak et al., 17 Nov 2025).
• Linguistic Ambiguity: User instructions are often underspecified, requiring disambiguation through spatial or contextual clues.
• Scale and Resolution: Elements can be minuscule (few pixels), necessitating robust localization even at high screen resolutions (Jiang et al., 5 Dec 2025).
• Semantic Diversity: Labels/text may be occluded or absent; iconography varies by platform and application domain (Li et al., 15 Sep 2025, Zhao et al., 25 Sep 2025).
• Generalization: GUIs are domain-diverse and evolve rapidly, motivating solutions that transfer across unseen platforms or layouts (Gou et al., 2024).

Datasets reflect this diversity. For example, the UGround corpus enumerates 10 million elements over 1.3 million screenshots, spanning synthetic web, desktop, and mobile scenarios with a mix of visual, positional, and functional referring expressions (Gou et al., 2024).

2. Architectural Approaches and Modeling Paradigms

2.1 Autoregressive and Diffusion Vision-LLMs

Most prevailing solutions are constructed by augmenting an existing MLLM (e.g., Vicuna, Qwen, LLaVA architectures), introducing task-specific output heads to decode region coordinates as structured text (e.g., "(x, y)" or "[x1, y1, x2, y2]") (Cheng et al., 2024, Gou et al., 2024, Jiang et al., 5 Dec 2025). Key ingredients include:

• AnyRes Tokenization: Partitioning high-res screenshots into patches, each encoded independently (e.g., CLIP ViT, (Gou et al., 2024)).
• Cross-modal Fusion: Visual patch tokens projected and concatenated with textual encodings, followed by joint cross-attention stacks.
• Output Decoding: Autoregressive generation of coordinates as language tokens; some models formulate region selection as token classification.

An emerging alternative employs discrete diffusion vision-LLMs (DVLMs) (e.g., LLaDA-V (Kumbhar et al., 27 Mar 2026)), leveraging bidirectional attention and a masked token diffusion process. By mixing linear and deterministic masking schedules, spatial consistency of bounding-box outputs is improved, and parallel token generation accelerates decoding.

2.2 Modular and Multi-Stage Frameworks

Recent advances demonstrate substantial benefits from decomposing GUI-VG into hierarchically staged processes (Kwak et al., 17 Nov 2025, Tang et al., 9 Mar 2025, Pei et al., 18 Mar 2026):

• Coarse-to-Fine Localization: Initial coarse Region-of-Interest (ROI) selection is followed by fine-grained grounding, with iterative zooming or cropping to minimize spatial dilution and maximally exploit MLLM spatial priors (MEGA-GUI (Kwak et al., 17 Nov 2025)).
• Instruction Refinement: Dedicated LLM modules rewrite ambiguous instructions into explicit, resolution-enhanced queries, improving downstream grounding accuracy, especially for small or ambiguous targets (AdaZoom-GUI (Pei et al., 18 Mar 2026)).
• Dual-Process Reasoning: Inspired by human cognition, systems dynamically switch between fast, direct prediction for simple cases and slower, analytical chains (e.g., interface summarization and region-proposal) for complex queries (Tang et al., 9 Mar 2025).

2.3 Iterative Reasoning and Feedback Loops

Training-free, iterative refinement frameworks (e.g., Chain-of-Ground (Li et al., 1 Dec 2025), ZoomUI (Liu et al., 15 Mar 2026), ZoomClick (Jiang et al., 5 Dec 2025)) apply reference feedback (visual or textual markers at intermediate predictions) or adapt multi-step zoom-based focusing strategies:

• Each iteration either overlays a visual marker at the prior prediction or augments the input instruction with coordinate feedback, guiding the next step to attend more locally.
• Zoom-based techniques systematically crop around the predicted location with adaptive window sizes, balancing localization precision and preservation of global context.

These iterative methods can leverage either heterogeneous MLLM pipelines or repeated passes through a single model. Pre-zoom selection, depth, shrink ratio, and minimal crop size are critical hyperparameters governing localization success (Jiang et al., 5 Dec 2025).

2.4 Reinforcement and Rule-Based Fine-Tuning

Rule-based RL frameworks such as Group Relative Policy Optimization (GRPO) (Kang et al., 6 Aug 2025), and its stabilized variants (VISTA (Qiu et al., 12 Jun 2026), GUI-Spotlight (Lei et al., 5 Oct 2025)), optimize grounding with group-normalized rewards and robust KL-divergence regularization, often using sparse point-in-box or IoU-based reward functions. Notable methodologies enforce stabilization through dynamically scaled KL penalties (Adversarial KL Factor (Kang et al., 6 Aug 2025)), multi-view group rollout construction (VISTA (Qiu et al., 12 Jun 2026)), and self-verified oracle anchors.

Co-evolutionary schemes such as COPC (Propose-then-Critic with maturity-aware RL) (Wang et al., 23 Apr 2026) introduce a learnable critic that ranks a set of proposer-generated candidate regions, significantly outperforming geometric clustering or spatial consensus baselines.

3. Data Curation, Benchmarking, and Evaluation Protocols

Benchmarks such as ScreenSpot, ScreenSpot-Pro, OSWorld-G, GUIZoom-Bench, and WinSpot are the primary empirical arbiter of GUI-VG methods (Cheng et al., 2024, Gou et al., 2024, Zhang et al., 9 Dec 2025, Hui et al., 27 Jan 2025). Key metrics include:

• Point-in-Box Accuracy: Fraction of predicted points within the annotated bounding box (canonical for ScreenSpot, WinSpot).
• Mean Intersection-Over-Union (mIoU): Area overlap between prediction and ground truth.
• Step Success Rate (SSR): For bounding-box outputs, whether the predicted center lands in or near the annotated box (Kumbhar et al., 27 Mar 2026).
• Oracle@K / Pass@K: Whether any of K sampled predictions hits the target (used for multi-candidate approaches).

Data construction strategies pair large-scale synthetic web/mobile element extraction with functional, positional, and visual referring expressions, often augmented by LLM-generated or human-verified instructions. Advanced pipelines (e.g., WinClick (Hui et al., 27 Jan 2025)) use web-scale scraping, proprietary icon detectors, and GPT-4o for region–description matching.

Sample efficiency is a major modern emphasis: state-of-the-art models often outperform prior approaches using just a few thousand aligned examples for RL-based fine-tuning (e.g., GuirlVG (Kang et al., 6 Aug 2025): 5.2K samples, <0.1% prior SFT data).

4. Empirical Advances, Generalization, and Robustness

The confluence of MLLM advances and RL-posttrained, modular, or trainless (inference-time only) workflows has yielded sharp improvements:

• GuirlVG achieves 88.7% on ScreenSpot (5.2K samples), +7.7% over best SFT, and 36.1% on ScreenSpot-Pro, +17.2% over SFT (Kang et al., 6 Aug 2025).
• VISTA: multi-view, self-verified training raises Qwen3-VL-8B accuracy from 52.7%→65.8% on ScreenSpot-Pro, with sharply reduced robustness failures and prediction flips (Qiu et al., 12 Jun 2026).
• MVP: by aggregating predictions across multiple instruction-attended views, Qwen3VL-32B-Instruct jumps from 55.3%→74.0% (+18.7%) on ScreenSpot-Pro (Zhang et al., 9 Dec 2025).
• AdaZoom-GUI: conditional zooming lifts icon-level grounding from 40.0%→47.6% (ScreenSpot-Pro); LLM-based instruction refinement delivers 76.8% overall (Pei et al., 18 Mar 2026).
• Chain-of-Ground: multi-step, marker-feedback cycles yield up to 68.4% accuracy on ScreenSpot-Pro, +4.8 points over prior SOTA (Li et al., 1 Dec 2025).
• MEGA-GUI: bidirectional ROI zoom and context-aware rewriting reach 73.18% on ScreenSpot-Pro and 68.63% on OSWorld-G (Kwak et al., 17 Nov 2025).

Generalization: RL and iterative methods consistently outperform SFT in out-of-domain and high-ambiguity settings, supporting that SFT “memorizes, RL generalizes” (Kang et al., 6 Aug 2025). Training-free approaches (ZoomClick (Jiang et al., 5 Dec 2025), MVP (Zhang et al., 9 Dec 2025), ZoomUI (Liu et al., 15 Mar 2026)) enable deployment on novel applications with no task-specific supervision, and scaling model capacity (Qwen3-VL-4/8/30B, UI-TARS-72B) steadily raises ceiling performance.

Table: Representative Benchmark Results

Model	Training Samples	ScreenSpot-Pro (%)	Approach Type
V2P-7B	9.6M	50.6	SFT RL
GuirlVG (Qwen2.5)	5.2K	36.1	RL (Adversarial KL)
GUI-Spotlight	18.5K	52.8	Iterative RL Tools
VISTA (Qwen3-VL-8B)	120K	65.8	RL Multi-View
MVP (Qwen3-32B)	0 (test-time)	74.0	Multi-View Cluster
AdaZoom-GUI	–	76.8	Zoom-in + Refinement

5. Auxiliary Cues, Zero-Shot Grounding, and Interpretability

Auxiliary reasoning, overlay scaffolding, and iterative prompt engineering have been found to unlock substantial latent VLM grounding potential without additional training (Li et al., 15 Sep 2025). Three dominant cues:

1. Coordinate Scaffold: Overlaying anchors (printed pixel coordinates) for indirect copying.
2. Axis-Grid Scaffold: Drawing labeled axes and grids, enabling grid-index reasoning.
3. Mark-Grid Scaffold: Superimposing cell IDs and using discrete cell labels as output, followed by coarse-to-fine refinement.

On upstream GUI-VG benchmarks, mark-grid scaffolds improve click accuracy by as much as 60–66 points for proprietary VLMs; even open-source models like Qwen2-VL-7B see latent pointing accuracy >58% via attention extraction, despite <5% for direct coordinate prediction (Li et al., 15 Sep 2025).

Iterative pipelines (e.g., Chain-of-Ground (Li et al., 1 Dec 2025)), modular architectures (MEGA-GUI (Kwak et al., 17 Nov 2025)), and cursor-based interactive search (GUI-Cursor (Zhao et al., 25 Sep 2025)) all favor interpretability: per-step outputs, visual feedback overlays, and explicit chain-of-thought traces clarify grounding trajectories and expose common error types.

6. Future Directions and Research Challenges

Key open questions and research directions identified across recent literature:

• Multi-Resolution and Context-Aware Training: Designing zoom-aware, multi-scale loss objectives that prevent contextual drift and maximize local discrimination (Jiang et al., 5 Dec 2025, Pei et al., 18 Mar 2026).
• Flexible Fine-Tuning: Hybrid test-time adaptation (thought vectors (Liu et al., 15 Mar 2026), anchor supervision (Qiu et al., 12 Jun 2026)) and lightweight LoRA for data-efficient transfer.
• Generalization to Dynamic and Multi-Action GUIs: Extending schemes to sequential task settings, drag-and-drop, and dynamically shifting interfaces (Lei et al., 5 Oct 2025).
• Diffusion-Based Models for GUI-Grounding: Bidirectional, parallel token-generation DVLMs demonstrate competitive SSR with autoregressive models but with distinct trade-offs in speed and training schedules (Kumbhar et al., 27 Mar 2026).
• Interpretability and Robustness: Quantitative robustness measures (worst-view accuracy, flip rate, group informativeness in RL) becoming standard (Qiu et al., 12 Jun 2026, Zhang et al., 9 Dec 2025).
• Scalability and Practical Deployment: Algorithmic efficiency (conditional zoom, modular routing, reduced data budgets) brings SOTA grounding within reach for real-world agent applications (Kang et al., 6 Aug 2025, Kwak et al., 17 Nov 2025).

Taken together, GUI-VG has evolved from heavily supervised, monolithic pipelines toward dynamically routed, modular, and feedback-rich frameworks that leverage rule-based RL, trainless inference, and auxiliary reasoning—enabling robust, generalizable, and efficient grounding for next-generation GUI agents.