GUI-AIMA: Coordinate-Free GUI Grounding

• GUI-AIMA is an attention-based framework that maps natural-language instructions to specific visual patches using latent multimodal attention.
• It aligns intrinsic attention signals via a context anchor to bypass coordinate generation, yielding state-of-the-art performance on benchmark datasets.
• A plug-and-play zoom-in refinement stage enhances localization precision on high-resolution screenshots by iteratively refining the patch selection.

GUI-AIMA is an attention-based, coordinate-free supervised fine-tuning framework for GUI grounding, the task of mapping natural-language instructions to actionable screen regions for computer-use agents. It is built on the observation that pretrained multimodal LLMs already contain a latent grounding capability in their multimodal self-attention patterns, and it seeks to align that intrinsic attention with patch-wise grounding supervision rather than forcing the model to autoregressively generate coordinates as text. Using Qwen2.5-VL-3B-Instruct as backbone, GUI-AIMA-3B was trained with approximately 85k screenshots and reported state-of-the-art performance among 3B models, including 58.6% average accuracy on ScreenSpot-Pro and 62.2% on OSWorld-G when combined with its plug-and-play zoom-in refinement stage (Zhou et al., 2 Nov 2025).

1. Definition and research setting

GUI-AIMA belongs to the GUI-agent literature concerned with screenshot-based computer use. In this setting, a model receives a screenshot and an instruction such as “click Save,” and must identify the correct target region before any downstream action can be executed. The GUI-AIMA paper treats this grounding step as a core function of computer-use agents and argues that the dominant alternative—direct text-based coordinate generation—is both precision-demanding and computationally indirect, especially on dense or high-resolution interfaces (Zhou et al., 2 Nov 2025).

This problem sits inside a broader shift from parser-dependent or API-dependent automation toward screenshot-native control. Auto-GUI, for example, argued for direct screen interaction and direct production of screen-executable actions without OCR/HTML parsing or application-specific APIs, showing that a screenshot-native agent can be practical for mobile device control (Zhang et al., 2023). At the evaluation level, MMBench-GUI later identified accurate visual grounding as a critical determinant of overall GUI task success and reported large gains from modular systems that combine strong planners with specialized grounders (Wang et al., 25 Jul 2025). Within that landscape, GUI-AIMA is specifically a grounding framework rather than a full end-to-end action policy.

A common misconception in this area is that GUI grounding is simply a small coordinate-regression problem. GUI-AIMA rejects that framing. Its central claim is that grounding is more naturally expressed as instruction-conditioned selection of relevant visual patches, with precise clicking treated as a later refinement step rather than the primary learning objective (Zhou et al., 2 Nov 2025).

2. Attention-based formulation

GUI-AIMA uses the standard multimodal transformer internals of its backbone rather than adding a separate grounding head. For an MLLM with $L$ layers and $H$ attention heads, and previous-layer hidden states $\mathbf{H}^{l-1}\in\mathbb{R}^{n\times d}$, head-specific queries and keys are defined as

$$\mathbf{Q}^{l,h}=\mathbf{H}^{l-1}\mathbf{W}^{l,h}_{Q}, \qquad \mathbf{K}^{l,h}=\mathbf{H}^{l-1}\mathbf{W}^{l,h}_{K},$$

with causal self-attention

$$\mathbf{A}^{l,h}=\operatorname{softmax}\left(\mathbf{Q}^{l,h}\mathbf{K}^{l,h^\top}/\sqrt{d_h}+\mathbf{M}\right),$$

where $\mathbf{M}_{ij}=-\infty$ for $i<j$ and $0$ otherwise. GUI-AIMA interprets the query-to-visual slices of these matrices as latent grounding signals and supervises them directly (Zhou et al., 2 Nov 2025).

The model input is organized conceptually as visual tokens, query tokens, and a learned context anchor:

$$[\mathcal{V}, \mathcal{Q}, \langle \text{ANCHOR} \rangle].$$

The paper notes that the implemented form is abbreviated from

$$[\mathcal{V}, \mathcal{Q}, \langle \text{ANCHOR\_START} \rangle, \langle \text{ANCHOR} \rangle, \langle \text{ANCHOR\_END} \rangle].$$

The grounding prediction is not a coordinate pair. Instead, it is a distribution over visual patches. If $H$0 is the anchor-token index, GUI-AIMA aggregates anchor-to-visual attention as

$H$1

This is the essence of its coordinate-free design: the model learns which patches matter, not which numeric tokens should spell out a point (Zhou et al., 2 Nov 2025).

Ground-truth boxes are converted into soft patch-wise labels rather than hard binary masks. For a box $H$2, each patch $H$3 receives

$H$4

where $H$5 is the patch center, $H$6 is the box center, and

$H$7

with $H$8. After normalization, this yields the target patch distribution $H$9. Training then minimizes

$\mathbf{H}^{l-1}\in\mathbb{R}^{n\times d}$0

The paper also states that next-token prediction loss is retained, following GUI-Actor’s special-token setting, although the exact combined weighting is not specified (Zhou et al., 2 Nov 2025).

3. Context anchor, visual-sink query tokens, and head aggregation

The context anchor is the framework’s defining mechanism. Rather than supervising the pretrained instruction tokens directly, GUI-AIMA appends a learned <[ANCHOR](https://www.emergentmind.com/topics/anchor-4b25332b-d1d6-4f8e-ad12-92dd1bad4c4d)> token whose attention to visual patches is trained to summarize the full instruction-conditioned grounding signal. This avoids manually aggregating every instruction token and reduces the risk of destabilizing the pretrained language representations. The anchor therefore functions as a compact grounding surrogate rather than as a conventional decoder token (Zhou et al., 2 Nov 2025).

Head weighting is learned adaptively, because not every attention head is equally useful for grounding. To identify informative heads, the method first computes hidden-state-based visual affinity for each instruction token. For layer $\mathbf{H}^{l-1}\in\mathbb{R}^{n\times d}$1,

$\mathbf{H}^{l-1}\in\mathbb{R}^{n\times d}$2

where $\mathbf{H}^{l-1}\in\mathbb{R}^{n\times d}$3 is cosine similarity. The paper reports that selecting visual-sink query tokens from hidden-state similarity is more reliable than summing attention mass directly. In the preferred setting, a global top-1 query token is chosen:

$\mathbf{H}^{l-1}\in\mathbb{R}^{n\times d}$4

Given these visual-sink tokens, the head weight is

$\mathbf{H}^{l-1}\in\mathbb{R}^{n\times d}$5

This weighting scheme lets GUI-AIMA emphasize layer-head pairs whose query–visual behavior best matches the instruction for the current example (Zhou et al., 2 Nov 2025).

The paper also evaluates a softer alternative. It forms a global query–visual distribution $\mathbf{H}^{l-1}\in\mathbb{R}^{n\times d}$6, compares it with each head’s own query–visual distribution $\mathbf{H}^{l-1}\in\mathbb{R}^{n\times d}$7, and sets

$\mathbf{H}^{l-1}\in\mathbb{R}^{n\times d}$8

This “soft” variant slightly improves ScreenSpot-Pro in one setting but is worse on ScreenSpot-v2 and OSWorld-G than the main design. The preferred configuration remains global top-1 visual-sink query selection with anchor-based aggregation (Zhou et al., 2 Nov 2025).

4. Supervision, training corpus, and zoom-in inference

GUI-AIMA-3B uses Qwen2.5-VL-3B-Instruct and is trained on the full training sets of GUIAct, AndroidControl, and Wave-UI, plus 60k randomly sampled samples from UGround and 60k from GTA1. The resulting corpus contains 259k instructions and approximately 85k screenshots. Training uses 8 NVIDIA A100 80GB GPUs, effective global batch size 64, learning rate $\mathbf{H}^{l-1}\in\mathbb{R}^{n\times d}$9, one epoch, and all parameters unfrozen. The paper presents this as evidence of strong data efficiency: light supervised alignment is sufficient to trigger the backbone’s native grounding capability rather than learning grounding from scratch (Zhou et al., 2 Nov 2025).

The most practical downstream consequence of the coordinate-free formulation is its plug-and-play zoom-in stage. The first pass predicts an approximate target region from the compressed full screenshot. The center of that predicted location is then used to define a crop, and the same model is rerun on the crop. The reported configuration uses crop size 448 pixels and a $$\mathbf{Q}^{l,h}=\mathbf{H}^{l-1}\mathbf{W}^{l,h}_{Q}, \qquad \mathbf{K}^{l,h}=\mathbf{H}^{l-1}\mathbf{W}^{l,h}_{K},$$0 zoom ratio. Because the first-stage output is a patch distribution rather than a coordinate token sequence, the second pass does not require retraining or an auxiliary model; it simply reapplies the same grounding mechanism at higher effective spatial resolution (Zhou et al., 2 Nov 2025).

This refinement targets the main error mode identified by the paper: offset grounding on dense, high-resolution screenshots. GUI-AIMA attributes these errors to the loss of fine spatial detail when large screenshots must be downsampled to fit GPU memory. The zoom-in stage is therefore not a separate planner or detector. It is a geometric refinement that exploits the framework’s coordinate-free patch selection. A plausible implication is that GUI-AIMA is particularly well suited as the grounding component in a larger GUI-agent stack, where coarse localization and precise clicking can be decoupled cleanly (Zhou et al., 2 Nov 2025).

5. Empirical results and ablation findings

GUI-AIMA is evaluated on ScreenSpot-v2, ScreenSpot-Pro, and OSWorld-G using center-point accuracy. ScreenSpot-v2 is a broad GUI grounding benchmark across mobile, desktop, and web; ScreenSpot-Pro is a harder high-resolution benchmark built from professional software; and OSWorld-G reports accuracy across text matching, element recognition, layout understanding, and fine-grained manipulation. The main reported averages are summarized below (Zhou et al., 2 Nov 2025).

Benchmark	GUI-AIMA-3B	GUI-AIMA-3B + zoom-in
ScreenSpot-v2	90.8	—
ScreenSpot-Pro	45.2	58.6
OSWorld-G	56.9	62.2

On ScreenSpot-Pro, GUI-AIMA-3B reaches Text 58.1, Icon 24.2, Avg 45.2; the soft variant reaches Avg 47.1; and the zoom-in version reaches Text 72.1, Icon 36.8, Avg 58.6. On OSWorld-G, GUI-AIMA-3B reaches 62.8 / 64.9 / 68.4 / 32.9 across the four categories, for Avg 56.9, while the zoom-in version reaches 69.4 / 67.0 / 74.3 / 45.4, for Avg 62.2. These numbers place GUI-AIMA-3B above GUI-Actor-3B, Jedi-3B, and several 7B baselines on ScreenSpot-Pro, while on OSWorld-G it surpasses same-scale models and approaches the strongest larger specialized model in the reported table (Zhou et al., 2 Nov 2025).

The ablations are methodologically revealing. Under the 45k-sample training setting, GUI-Actor-3B achieves ScreenSpot-v2 average 87.42 and ScreenSpot-Pro average 35.99, whereas vanilla attention grounding reaches 86.87 and 37.51, and simplified anchor-based attention grounding with uniform head weighting reaches 88.92 and 40.23. Adding global top-1 visual-sink query selection improves ScreenSpot-Pro to 42.13, and weighted patch labeling raises it to 43.39. These results support three specific conclusions: intrinsic attention is already a competitive grounding substrate; the context anchor is better than direct query-token supervision in this setting; and overlap-aware, center-encouraging soft patch labels materially improve localization on hard screenshots (Zhou et al., 2 Nov 2025).

The zoom-in analysis shows the largest single practical gain. On ScreenSpot-Pro offset correction, the reported progression is 47.06 for one-step prediction, 50.16 for crop-only, 57.18 for crop plus $$\mathbf{Q}^{l,h}=\mathbf{H}^{l-1}\mathbf{W}^{l,h}_{Q}, \qquad \mathbf{K}^{l,h}=\mathbf{H}^{l-1}\mathbf{W}^{l,h}_{K},$$1 zoom, 58.57 for $$\mathbf{Q}^{l,h}=\mathbf{H}^{l-1}\mathbf{W}^{l,h}_{Q}, \qquad \mathbf{K}^{l,h}=\mathbf{H}^{l-1}\mathbf{W}^{l,h}_{K},$$2, 58.25 for $$\mathbf{Q}^{l,h}=\mathbf{H}^{l-1}\mathbf{W}^{l,h}_{Q}, \qquad \mathbf{K}^{l,h}=\mathbf{H}^{l-1}\mathbf{W}^{l,h}_{K},$$3, and 58.19 for $$\mathbf{Q}^{l,h}=\mathbf{H}^{l-1}\mathbf{W}^{l,h}_{Q}, \qquad \mathbf{K}^{l,h}=\mathbf{H}^{l-1}\mathbf{W}^{l,h}_{K},$$4. The best point is therefore $$\mathbf{Q}^{l,h}=\mathbf{H}^{l-1}\mathbf{W}^{l,h}_{Q}, \qquad \mathbf{K}^{l,h}=\mathbf{H}^{l-1}\mathbf{W}^{l,h}_{K},$$5, indicating that added resolution helps until further zoom ceases to improve the patch-selection-to-click-center conversion (Zhou et al., 2 Nov 2025).

6. Position within GUI-agent research, significance, and limits

GUI-AIMA is best understood as a grounding-centric contribution rather than a full automation architecture. Auto-GUI directly consumes screenshots and emits executable actions for multi-step control; D-Artemis adds app-specific retrieval, pre-execution alignment, and post-execution reflection for dynamic mobile execution; UI-Copilot addresses long-horizon memory degradation, progress confusion, and numerical computation with selective tool use (Zhang et al., 2023, Mi et al., 26 Sep 2025, Lu et al., 15 Apr 2026). By contrast, GUI-AIMA focuses on the narrower but foundational question of how a multimodal model should localize the target region itself. This division of labor is consistent with MMBench-GUI’s finding that accurate visual grounding is a critical determinant of overall task success and that modular systems with specialized grounders can outperform monolithic alternatives (Wang et al., 25 Jul 2025).

Its methodological significance lies in treating attention not merely as an explanatory artifact but as an optimizable grounding interface. That differs from coordinate-generation formulations, and it also differs from representation-centric high-resolution approaches such as AFRAgent, which improves mobile GUI control by injecting low-resolution and high-resolution visual detail into a compact query-token bottleneck through adaptive feature renormalization (Anand et al., 30 Nov 2025). A plausible implication is that GUI-AIMA and AFRAgent are complementary: one aligns intrinsic attention to patch-wise grounding targets, while the other strengthens the spatial fidelity of the visual representations those attentions operate over.

The main limitation explicitly discussed by GUI-AIMA is offset grounding on high-resolution screenshots, caused by compression and coarse patch granularity. The zoom-in stage alleviates but does not conceptually remove that dependence on a first-pass coarse region proposal. More broadly, the framework does not specify a full action policy, a planner, or a recovery mechanism after grounding errors. This suggests that GUI-AIMA is most naturally deployed as a specialized grounding module inside a larger GUI-agent system. That reading is reinforced by adjacent efficiency work such as AQuaUI, which treats GUI screenshots as spatially non-uniform and reduces visual tokens at inference time while preserving geometric positions, and by collaboration-oriented benchmarks such as GUIDE, which move beyond task execution toward user-state understanding and assistance (Li et al., 19 May 2026, Yang et al., 26 Mar 2026).

Taken together, these results make GUI-AIMA a notable turning point in GUI grounding research. It shows that relatively light supervised alignment can unlock a pretrained MLLM’s latent grounding behavior, that patch-wise attention supervision can outperform more indirect coordinate-centric formulations under constrained data, and that a coordinate-free output space provides a practical path to high-resolution refinement without redesigning the base model (Zhou et al., 2 Nov 2025).