GUI-ARP: Adaptive GUI Grounding

• GUI-ARP is a framework for GUI grounding that employs adaptive multi-stage inference, using ARP to crop regions based on attention maps and ASC to decide the processing stage.
• It leverages a two-phase training pipeline—supervised fine-tuning and reinforcement fine-tuning with GRPO—to optimize both localization precision and decision control.
• GUI-ARP achieves state-of-the-art results on challenging benchmarks like ScreenSpot-Pro and UI-Vision by selectively invoking additional processing on hard cases.

Searching arXiv for the primary paper and closely related named systems to provide accurate citations. GUI-ARP is a framework for GUI grounding that targets fine-grained localization in high-resolution screenshots by introducing adaptive multi-stage inference into a Vision-LLM (VLM) with a Vision Transformer visual encoder and an LLM-style text decoder. It augments the “coordinate-free” grounding paradigm of GUI-Actor with two modules—Adaptive Region Perception (ARP) and Adaptive Stage Controlling (ASC)—so that the model can exploit its own visual attention to crop task-relevant regions and decide whether a screenshot should be handled in a single stage or through a second, finer pass (Ye et al., 19 Sep 2025). The framework is trained with a two-phase pipeline combining supervised fine-tuning and reinforcement fine-tuning based on Group Relative Policy Optimization (GRPO), and it is reported to achieve state-of-the-art performance on challenging GUI grounding benchmarks, including 60.8% accuracy on ScreenSpot-Pro and 30.9% on UI-Vision for the 7B model (Ye et al., 19 Sep 2025).

1. Conceptual basis and problem setting

Existing GUI grounding methods are described as often struggling with fine-grained localization in high-resolution screenshots, particularly when small targets or dense interface regions make single-pass reasoning insufficient (Ye et al., 19 Sep 2025). GUI-ARP is designed to address this limitation through adaptive multi-stage inference rather than a fixed inference schedule. This design couples localization with a runtime decision about whether additional visual refinement is necessary.

At a high level, the framework operates on top of a standard VLM and introduces two new mechanisms. ARP uses the model’s own attention map over image patches to determine how to crop a subregion for a second, finer pass. ASC produces a simple yes/no “tool-call” token that determines whether the screenshot is treated as “easy,” in which case single-stage inference is used, or “hard,” in which case multi-stage analysis with ARP is invoked (Ye et al., 19 Sep 2025).

This organization implies a division of labor between coarse global reasoning and localized re-analysis. A plausible implication is that GUI-ARP treats GUI grounding not only as point prediction but also as an adaptive control problem in which the model must decide when additional perceptual resolution is warranted.

2. Inference architecture and execution flow

The end-to-end inference process begins with Stage 1, where the full GUI screenshot is encoded through the ViT and the instruction is processed through the text encoder. A special anchor token, <ACTOR>, is appended, and its final hidden state is used to compute an attention map $A \in R^{H \times W}$ over visual patches. The decoder also generates a structured control token of the form <tool_call> yes/no </tool_call> (Ye et al., 19 Sep 2025).

ASC then converts this output into a binary decision. If the model outputs no, cropping is skipped and the final $(x,y)$ point is decoded directly from the Stage 1 hidden states. If the model outputs yes, ARP is applied to the attention map $A$ to extract a bounding box $R$; the image is cropped to that region, re-encoded, and Stage 2 attends over the smaller crop before emitting the final coordinate (Ye et al., 19 Sep 2025).

The paper specifies the inference logic in pseudocode:

$H$1

In this formulation, VLM_forward returns visual–textual hidden states $H$, the patch-level attention map $A$ given by attention from <ACTOR> onto each patch, and the probability that the model issues <tool_call> = yes (Ye et al., 19 Sep 2025).

The architectural significance of this design lies in its conditional reallocation of computation. Rather than applying a uniform second-stage zoom to every input, the framework reserves multi-stage analysis for cases that the model itself identifies as difficult.

3. Adaptive Region Perception

ARP takes the Stage 1 attention map $A \in R^{H \times W}$ and produces a crop region $R$ by identifying the $k$ most focused connected regions in attention space. The method is defined procedurally.

First, thresholding is applied to define a seed set:

$S=\{(i,j)\mid A_{ij}\ge\tau\max_{u,v}A_{u,v}\},$

where $(x,y)$0 is a fraction of the maximum attention and is given as, for example, $(x,y)$1 (Ye et al., 19 Sep 2025).

Second, the seed set is divided into connected components $(x,y)$2. Third, each component is scored by the sum of its attention weights:

$(x,y)$3

The top $(x,y)$4 components are then selected (Ye et al., 19 Sep 2025).

Fourth, for each selected component $(x,y)$5, an attention-weighted center is computed:

$(x,y)$6

Finally, the crop region is defined from the extrema of these weighted centers:

$(x,y)$7

with

$(x,y)$8

The paper also provides this as Algorithm 1 (Ye et al., 19 Sep 2025).

This mechanism differs from fixed-zoom heuristics by using the model’s own attention to determine the crop geometry. The stated interpretation is that ARP adapts to large or small localization errors automatically, rather than applying a fixed zoom factor. This suggests that the crop is intended to remain aligned with the model’s internal uncertainty structure rather than with an externally specified resizing rule.

4. Adaptive Stage Controlling

ASC is the decision mechanism that determines whether ARP should be invoked. During decoding, the model emits <tool_call> yes / no </tool_call>, and the decision variable is defined as

$(x,y)$9

If $A$0, the model outputs the location from Stage 1; if $A$1, it invokes ARP and reruns the VLM on the cropped region (Ye et al., 19 Sep 2025).

The framework’s interpretation of ASC is that the network learns to set $A$2 only on “hard” screenshots where Stage 1 attention is too diffuse or ambiguous, and to leave $A$3 on “easy” cases. During supervised fine-tuning, this binary decision is supervised with chain-of-thought annotations that explicitly label each example as “easy” or “challenging,” thereby teaching the model a decision boundary for single-stage versus multi-stage inference (Ye et al., 19 Sep 2025).

A common misconception would be to regard ARP as always active once incorporated into the model. The paper’s design explicitly rejects that view: the full system is conditional, and its effectiveness depends on ASC learning when not to crop as much as when to crop. The reported ablation behavior on simple and difficult benchmarks is presented as evidence for this selectivity (Ye et al., 19 Sep 2025).

5. Two-phase training pipeline

The training procedure has two phases: supervised fine-tuning (SFT) and reinforcement fine-tuning with GRPO (Ye et al., 19 Sep 2025).

For SFT, the paper states that approximately 2.4K examples are collected from several open-source GUI grounding datasets. Each example is run through ARP once. If Stage 1 alone suffices to predict the correct element, the example is labeled “easy”; otherwise it is labeled “challenging.” A large VLM, Qwen-VL-Max, is then prompted to generate a chain-of-thought that includes the correct <tool_call> decision (Ye et al., 19 Sep 2025).

All model parameters are unfrozen, and training runs for 4 epochs with a composite loss

$A$4

The next-token prediction loss is

$A$5

and the attention supervision term is a KL divergence:

$A$6

where $A$7 is 1 on the ground-truth patch and 0 elsewhere, and $A$8 is the model’s predicted attention (Ye et al., 19 Sep 2025).

The reinforcement fine-tuning phase builds on GRPO and introduces three reward components. The format reward $A$9 enforces the required output format. The tool-call reward $R$0 encourages correct single-vs-multi-stage decisions:

$R$1

The Gaussian point reward $R$2 penalizes distance from the ground-truth center:

$R$3

where $R$4 and $R$5 scale with the element’s size, with $R$6. The total reward is

$R$7

For optimization, the VLM-R1 codebase is adapted, all weights are unfrozen, the KL-penalty coefficient is set to $R$8, 8 rollouts are generated per instruction, and training runs for 2 epochs (Ye et al., 19 Sep 2025).

The paper provides the reinforcement loop in algorithmic form:

$H$2

This training design indicates that GUI-ARP is optimized jointly for output correctness, adaptive control behavior, and attention alignment. A plausible implication is that the framework treats attention not merely as an interpretability artifact but as a supervised intermediate variable with direct operational consequences for cropping.

6. Benchmark performance and ablation evidence

GUI-ARP is evaluated on four benchmarks: ScreenSpot-v1 (SS-v1), ScreenSpot-v2 (SS-v2), ScreenSpot-Pro (SS-Pro), and UI-Vision. SS-v1 and SS-v2 are described as general GUI screenshots across desktop, mobile, and web; SS-Pro as high-resolution professional software UIs; and UI-Vision as desktop-centric high-resolution GUI tasks. Accuracy is measured as the fraction of tasks where the predicted point lies within the ground-truth bounding box (Ye et al., 19 Sep 2025).

The main reported results are summarized below.

Benchmark	GUI-ARP-7B	Comparison noted in the paper
ScreenSpot-Pro	60.8%	UI-TARS-72B at 38.1%
UI-Vision	30.9%	UI-TARS-72B at 25.2%
SS-v1 and SS-v2	$R$9–$H$0% average	Competitive versus much larger or proprietary models

The paper further reports that GUI-ARP-7B demonstrates strong competitiveness against open-source 72B models and proprietary models, with ScreenSpot-Pro highlighted as a particularly challenging benchmark (Ye et al., 19 Sep 2025).

Ablation studies isolate the effects of SFT, GRPO, and adaptive staging. Baseline GUI-Actor without ARP or ASC attains 44.6% on SS-Pro with tool-call rate 0%. Adding SFT only yields 54.3% on SS-Pro with tool-call rate 29.7%, and 91.6% on SS-v2 with rate 11.5%. The full system with GRPO reaches 60.8% on SS-Pro with tool-call 92.4%, and 91.8% on SS-v2 with rate 33.4% (Ye et al., 19 Sep 2025).

The paper also notes that single-stage runs, in which no multi-stage inference is permitted even at test time, consistently underperform the full pipeline on SS-Pro but slightly improve on very simple SS-v2. This is presented as confirmation that ASC learns to switch off ARP for easy layouts (Ye et al., 19 Sep 2025). In qualitative examples, ARP is described as zooming into small button clusters that the single-stage model misses or localizes poorly, while ASC preserves full-screen reasoning when the target is already salient (Ye et al., 19 Sep 2025).

7. Relation to prior paradigms and stated contributions

GUI-ARP is positioned as an extension of the “coordinate-free” grounding paradigm of GUI-Actor, with the additions of ARP and ASC as its distinctive modules (Ye et al., 19 Sep 2025). It remains within the VLM paradigm—ViT visual encoder plus LLM-style decoder—but augments that backbone with an explicit mechanism for self-directed cropping and an explicit mechanism for deciding whether cropping should occur.

The paper’s summary of contributions identifies three technical claims. First, by inspecting its own attention, the ARP module yields tighter, model-aligned crops than fixed-zoom heuristics. Second, the ASC policy gives VLMs an on-demand “glance-and-focus” ability, improving accuracy on hard cases and efficiency on easy ones. Third, the two-phase training pipeline allows the model to learn both precise grounding and adaptive invocation of multi-stage inference (Ye et al., 19 Sep 2025).

These claims define GUI-ARP less as a wholly new grounding backbone than as an adaptive inference framework layered onto a standard VLM. This suggests a broader interpretation in which GUI grounding performance on high-resolution interfaces may depend as much on runtime control and attention-conditioned resampling as on raw model scale alone. Within the evidence presented, that interpretation is supported by the comparison between a 7B model and larger 72B systems on the most difficult reported benchmarks (Ye et al., 19 Sep 2025).