---
title: 'GUI-ARP: Adaptive GUI Grounding'
url: https://www.emergentmind.com/topics/gui-arp
type: topic
---

# GUI-ARP: Adaptive GUI Grounding

Searching arXiv for the primary paper and closely related named systems to provide accurate citations.
GUI-ARP is a framework for GUI grounding that targets fine-grained localization in high-resolution screenshots by introducing adaptive multi-stage inference into a Vision-Language Model (VLM) with a Vision Transformer visual encoder and an LLM-style text decoder. It augments the “coordinate-free” grounding paradigm of GUI-Actor with two modules—Adaptive Region Perception (ARP) and Adaptive Stage Controlling (ASC)—so that the model can exploit its own visual attention to crop task-relevant regions and decide whether a screenshot should be handled in a single stage or through a second, finer pass [2509.15532]. The framework is trained with a two-phase pipeline combining supervised fine-tuning and reinforcement fine-tuning based on Group Relative Policy Optimization (GRPO), and it is reported to achieve state-of-the-art performance on challenging GUI grounding benchmarks, including 60.8% accuracy on ScreenSpot-Pro and 30.9% on UI-Vision for the 7B model [2509.15532].

## 1. Conceptual basis and problem setting

Existing GUI grounding methods are described as often struggling with fine-grained localization in high-resolution screenshots, particularly when small targets or dense interface regions make single-pass reasoning insufficient [2509.15532]. GUI-ARP is designed to address this limitation through adaptive multi-stage inference rather than a fixed inference schedule. This design couples localization with a runtime decision about whether additional visual refinement is necessary.

At a high level, the framework operates on top of a standard VLM and introduces two new mechanisms. ARP uses the model’s own attention map over image patches to determine how to crop a subregion for a second, finer pass. ASC produces a simple yes/no “tool-call” token that determines whether the screenshot is treated as “easy,” in which case single-stage inference is used, or “hard,” in which case multi-stage analysis with ARP is invoked [2509.15532].

This organization implies a division of labor between coarse global reasoning and localized re-analysis. A plausible implication is that GUI-ARP treats GUI grounding not only as point prediction but also as an adaptive control problem in which the model must decide when additional perceptual resolution is warranted.

## 2. Inference architecture and execution flow

The end-to-end inference process begins with Stage 1, where the full GUI screenshot is encoded through the ViT and the instruction is processed through the text encoder. A special anchor token, `<ACTOR>`, is appended, and its final hidden state is used to compute an attention map $A \in R^{H \times W}$ over visual patches. The decoder also generates a structured control token of the form `<tool_call> yes/no </tool_call>` [2509.15532].

ASC then converts this output into a binary decision. If the model outputs `no`, cropping is skipped and the final $(x,y)$ point is decoded directly from the Stage 1 hidden states. If the model outputs `yes`, ARP is applied to the attention map $A$ to extract a bounding box $R$; the image is cropped to that region, re-encoded, and Stage 2 attends over the smaller crop before emitting the final coordinate [2509.15532].

The paper specifies the inference logic in pseudocode:

```text
function GUI-ARP_Inference(screenshot I, instruction T):
  # Stage 1
  H_full, A_full, p_tool = VLM_forward(I, T)               # p_tool = P(tool_call=yes)
  if p_tool < 0.5:       # single-stage
    return decode_point(H_full)
  else:       # multi-stage
    R = ARP_cropping(A_full)
    H_crop, A_crop, _ = VLM_forward(I[R], T)
    return decode_point(H_crop)
```

In this formulation, `VLM_forward` returns visual–textual hidden states $H$, the patch-level attention map $A$ given by attention from `<ACTOR>` onto each patch, and the probability that the model issues `<tool_call> = yes` [2509.15532].

The architectural significance of this design lies in its conditional reallocation of computation. Rather than applying a uniform second-stage zoom to every input, the framework reserves multi-stage analysis for cases that the model itself identifies as difficult.

## 3. Adaptive Region Perception

ARP takes the Stage 1 attention map $A \in R^{H \times W}$ and produces a crop region $R$ by identifying the $k$ most focused connected regions in attention space. The method is defined procedurally.

First, thresholding is applied to define a seed set:
$$
S=\{(i,j)\mid A_{ij}\ge\tau\max_{u,v}A_{u,v}\},
$$
where $\tau$ is a fraction of the maximum attention and is given as, for example, $0.3$ [2509.15532].

Second, the seed set is divided into connected components $\{G_1,G_2,\dots\}$. Third, each component is scored by the sum of its attention weights:
$$
\mathrm{score}(G)=\sum_{(i,j)\in G}A_{ij}.
$$
The top $k$ components are then selected [2509.15532].

Fourth, for each selected component $G$, an attention-weighted center is computed:
$$
c_x(G)=\frac{\sum_{(i,j)\in G} i\,A_{ij}}{\sum_{(i,j)\in G}A_{ij}},\quad
c_y(G)=\frac{\sum_{(i,j)\in G} j\,A_{ij}}{\sum_{(i,j)\in G}A_{ij}}.
$$
Finally, the crop region is defined from the extrema of these weighted centers:
$$
x_{\min}=\min_m c_x(G_{(m)}),\;
y_{\min}=\min_m c_y(G_{(m)}),\;
x_{\max}=\max_m c_x(G_{(m)}),\;
y_{\max}=\max_m c_y(G_{(m)}),
$$
with
$$
R=[x_{\min},y_{\min},x_{\max},y_{\max}].
$$
The paper also provides this as Algorithm 1 [2509.15532].

This mechanism differs from fixed-zoom heuristics by using the model’s own attention to determine the crop geometry. The stated interpretation is that ARP adapts to large or small localization errors automatically, rather than applying a fixed zoom factor. This suggests that the crop is intended to remain aligned with the model’s internal uncertainty structure rather than with an externally specified resizing rule.

## 4. Adaptive Stage Controlling

ASC is the decision mechanism that determines whether ARP should be invoked. During decoding, the model emits `<tool_call> yes / no </tool_call>`, and the decision variable is defined as
$$
d =
\begin{cases}
1 & \text{if }P(\text{tool\_call}=\text{yes}\mid I,T) \ge 0.5,\\
0 & \text{otherwise.}
\end{cases}
$$
If $d=0$, the model outputs the location from Stage 1; if $d=1$, it invokes ARP and reruns the VLM on the cropped region [2509.15532].

The framework’s interpretation of ASC is that the network learns to set $d=1$ only on “hard” screenshots where Stage 1 attention is too diffuse or ambiguous, and to leave $d=0$ on “easy” cases. During supervised fine-tuning, this binary decision is supervised with chain-of-thought annotations that explicitly label each example as “easy” or “challenging,” thereby teaching the model a decision boundary for single-stage versus multi-stage inference [2509.15532].

A common misconception would be to regard ARP as always active once incorporated into the model. The paper’s design explicitly rejects that view: the full system is conditional, and its effectiveness depends on ASC learning when not to crop as much as when to crop. The reported ablation behavior on simple and difficult benchmarks is presented as evidence for this selectivity [2509.15532].

## 5. Two-phase training pipeline

The training procedure has two phases: supervised fine-tuning (SFT) and reinforcement fine-tuning with GRPO [2509.15532].

For SFT, the paper states that approximately 2.4K examples are collected from several open-source GUI grounding datasets. Each example is run through ARP once. If Stage 1 alone suffices to predict the correct element, the example is labeled “easy”; otherwise it is labeled “challenging.” A large VLM, Qwen-VL-Max, is then prompted to generate a chain-of-thought that includes the correct `<tool_call>` decision [2509.15532].

All model parameters are unfrozen, and training runs for 4 epochs with a composite loss
$$
\mathcal{L}=\mathcal{L}_{\mathrm{NTP}}+\mathcal{L}_{\mathrm{Attn}}.
$$
The next-token prediction loss is
$$
\mathcal{L}_{\mathrm{NTP}}(\theta)
= -\frac{1}{T}\sum_{t=1}^T \log P\bigl(x_t\mid x_{<t};\theta\bigr),
$$
and the attention supervision term is a KL divergence:
$$
\mathcal{L}_{\mathrm{Attn}}
= \sum_{i=1}^M p_i\log\frac{p_i}{a_i},
\quad
p_i=\frac{y_i}{\sum_j y_j+\varepsilon},
$$
where $y_i$ is 1 on the ground-truth patch and 0 elsewhere, and $a_i$ is the model’s predicted attention [2509.15532].

The reinforcement fine-tuning phase builds on GRPO and introduces three reward components. The format reward $\mathcal{R}_{\mathrm{format}}\in\{0,1\}$ enforces the required output format. The tool-call reward $\mathcal{R}_{\mathrm{tool}}\in\{0,1\}$ encourages correct single-vs-multi-stage decisions:
$$
\mathcal{R}_{\mathrm{tool}}=
\begin{cases}
1,&
\bigl(\text{tool=no}\wedge \text{point inside gt\_bbox}\bigr)\;\lor\;
\bigl(\text{tool=yes}\wedge \text{point outside gt\_bbox}\bigr),\\
0,&\text{otherwise.}
\end{cases}
$$
The Gaussian point reward $\mathcal{R}_{\mathrm{point}}\in(0,1]$ penalizes distance from the ground-truth center:
$$
\mathcal{R}_{\mathrm{point}}
= \exp\!\Bigl(
-\tfrac12
\Bigl[\tfrac{(c_x^p - c_x^{gt})^2}{\sigma_x^{2}}
+\tfrac{(c_y^p - c_y^{gt})^2}{\sigma_y^{2}}
\Bigr]
\Bigr),
$$
where $\sigma_x=\alpha(x_2-x_1)$ and $\sigma_y=\alpha(y_2-y_1)$ scale with the element’s size, with $\alpha=0.5$. The total reward is
$$
\mathcal{R}=\mathcal{R}_{\mathrm{format}}+\mathcal{R}_{\mathrm{tool}}+\mathcal{R}_{\mathrm{point}}.
$$
For optimization, the VLM-R1 codebase is adapted, all weights are unfrozen, the KL-penalty coefficient is set to $\beta=0.04$, 8 rollouts are generated per instruction, and training runs for 2 epochs [2509.15532].

The paper provides the reinforcement loop in algorithmic form:

```text
Initialize π_θ from SFT
for epoch in {1…2}:
  for each (I,T) in training data:
    Sample {y_k}ₖ₌₁⁸ ~ π_θ(·|I,T)
    For each y_k compute Rₖ = R_format + R_tool + R_point
    Compute advantages Aₖ = Rₖ − baseline
    Compute policy gradient with KL penalty β:
      ∇θ L = − Σₖ Aₖ ∇θ log π_θ(yₖ|I,T)
             + β ∇θ KL(π_θ || π_old)
    θ ← θ − Adam(∇θ L)
    θ_old ← θ
```

This training design indicates that GUI-ARP is optimized jointly for output correctness, adaptive control behavior, and attention alignment. A plausible implication is that the framework treats attention not merely as an interpretability artifact but as a supervised intermediate variable with direct operational consequences for cropping.

## 6. Benchmark performance and ablation evidence

GUI-ARP is evaluated on four benchmarks: ScreenSpot-v1 (SS-v1), ScreenSpot-v2 (SS-v2), ScreenSpot-Pro (SS-Pro), and UI-Vision. SS-v1 and SS-v2 are described as general GUI screenshots across desktop, mobile, and web; SS-Pro as high-resolution professional software UIs; and UI-Vision as desktop-centric high-resolution GUI tasks. Accuracy is measured as the fraction of tasks where the predicted point lies within the ground-truth bounding box [2509.15532].

The main reported results are summarized below.

| Benchmark | GUI-ARP-7B | Comparison noted in the paper |
|---|---:|---|
| ScreenSpot-Pro | 60.8% | UI-TARS-72B at 38.1% |
| UI-Vision | 30.9% | UI-TARS-72B at 25.2% |
| SS-v1 and SS-v2 | $\sim 89$–$92$% average | Competitive versus much larger or proprietary models |

The paper further reports that GUI-ARP-7B demonstrates strong competitiveness against open-source 72B models and proprietary models, with ScreenSpot-Pro highlighted as a particularly challenging benchmark [2509.15532].

Ablation studies isolate the effects of SFT, GRPO, and adaptive staging. Baseline GUI-Actor without ARP or ASC attains 44.6% on SS-Pro with tool-call rate 0%. Adding SFT only yields 54.3% on SS-Pro with tool-call rate 29.7%, and 91.6% on SS-v2 with rate 11.5%. The full system with GRPO reaches 60.8% on SS-Pro with tool-call 92.4%, and 91.8% on SS-v2 with rate 33.4% [2509.15532].

The paper also notes that single-stage runs, in which no multi-stage inference is permitted even at test time, consistently underperform the full pipeline on SS-Pro but slightly improve on very simple SS-v2. This is presented as confirmation that ASC learns to switch off ARP for easy layouts [2509.15532]. In qualitative examples, ARP is described as zooming into small button clusters that the single-stage model misses or localizes poorly, while ASC preserves full-screen reasoning when the target is already salient [2509.15532].

## 7. Relation to prior paradigms and stated contributions

GUI-ARP is positioned as an extension of the “coordinate-free” grounding paradigm of GUI-Actor, with the additions of ARP and ASC as its distinctive modules [2509.15532]. It remains within the VLM paradigm—ViT visual encoder plus LLM-style decoder—but augments that backbone with an explicit mechanism for self-directed cropping and an explicit mechanism for deciding whether cropping should occur.

The paper’s summary of contributions identifies three technical claims. First, by inspecting its own attention, the ARP module yields tighter, model-aligned crops than fixed-zoom heuristics. Second, the ASC policy gives VLMs an on-demand “glance-and-focus” ability, improving accuracy on hard cases and efficiency on easy ones. Third, the two-phase training pipeline allows the model to learn both precise grounding and adaptive invocation of multi-stage inference [2509.15532].

These claims define GUI-ARP less as a wholly new grounding backbone than as an adaptive inference framework layered onto a standard VLM. This suggests a broader interpretation in which GUI grounding performance on high-resolution interfaces may depend as much on runtime control and attention-conditioned resampling as on raw model scale alone. Within the evidence presented, that interpretation is supported by the comparison between a 7B model and larger 72B systems on the most difficult reported benchmarks [2509.15532].

Source: https://www.emergentmind.com/topics/gui-arp