GUI-Spotlight: Adaptive GUI Grounding

• GUI-Spotlight is a visual grounding model that defines focus refinement as an iterative tool-using process for accurate GUI element localization.
• It employs specialized tools such as 'extract', 'find_color', and 'crop' to progressively narrow the search area and manage spatial offsets in complex interfaces.
• Performance benchmarks indicate competitive accuracy improvements through sample-efficient training and a robust multi-turn reasoning loop.

Searching arXiv for the core paper and closely related GUI grounding work. GUI-Spotlight is a GUI visual grounding model that treats grounding as an image-grounded reasoning problem with adaptive tool use rather than as a one-shot coordinate prediction task. It is designed to map textual references to exact on-screen elements in dense, high-resolution interfaces where small localization errors can invalidate pointer-level actions such as clicking or dragging. The method maintains a multi-turn interaction loop in which the model repeatedly narrows its visual focus by invoking specialized tools and only then emits a final coordinate. In the reported ScreenSpot-Pro evaluation, the variant initialized from UI-TARS-1.5-7B achieves 52.8% accuracy with 18.5K training samples (Lei et al., 5 Oct 2025).

1. Problem setting and historical context

GUI visual grounding is the task of localizing the GUI element referred to by a natural-language instruction so that an agent can execute an action at the correct screen position. In practical systems, this capability is a prerequisite for reliable computer-use automation because textual task understanding is insufficient if the model cannot map that understanding to exact pixels. The difficulty becomes acute in professional and OS-level interfaces, where screens are high-resolution, targets are small, and many controls are visually repetitive or densely packed (Lei et al., 5 Oct 2025).

The broader literature had already identified “focus” as a useful abstraction for UI understanding. “Spotlight: Mobile UI Understanding using Vision-LLMs with a Focus” introduced a vision-only mobile UI framework that takes a screenshot and a region of interest as input, replacing direct reliance on view hierarchies with an explicit focus mechanism (Li et al., 2022). Later GUI grounding research shifted from static focused regions to search-oriented and multi-stage localization. ScreenSpot-Pro crystallized the professional high-resolution setting as a dedicated benchmark, reporting that targets cover only 0.07% of screenshot area on average, compared with 2.01% in ScreenSpot, and showing that direct full-screen grounding is inadequate for realistic professional computer use (Li et al., 4 Apr 2025).

Within that trajectory, GUI-Spotlight formalizes focus refinement as an iterative tool-using process. Its central premise is that the model should “think with the image”: instead of producing a final click point immediately, it should progressively isolate the relevant region until the target can be localized precisely (Lei et al., 5 Oct 2025).

2. Core architecture and inference loop

GUI-Spotlight is implemented as an agentic interaction framework around an MLLM backbone. The inference state contains an image registry

$R=\{\, i \mapsto (I_i,\boldsymbol{\delta}_i)\,\}$

where $\boldsymbol{\delta}_i=(\delta_i^x,\delta_i^y)$ is the top-left offset of image $I_i$ with respect to the original screenshot $I_0$, and a message history

$\mathcal{H} \gets \{(d, I_0)\},$

initialized with the user description $d$ and the original image (Lei et al., 5 Oct 2025).

At each round, the model consumes the history $\mathcal{H}$ and outputs either a tool invocation or a termination action. The two forms are:

• Action(i, Tool, args)
• Stop(i, (x_{\mathrm{rel}}, y_{\mathrm{rel}}))

If the output is a tool call, the selected tool operates on image $I_i$, produces a new cropped image $I_{i+1}$, an informational message, and a new offset $\boldsymbol{\delta}_{i+1}$. These are appended to the registry and dialogue history. If the output is Stop, the absolute click coordinate is recovered by adding the local coordinate to the stored offset:

$\boldsymbol{\delta}_i=(\delta_i^x,\delta_i^y)$0

This design makes the model inherently multi-turn. The paper gives the inference algorithm as a bounded loop over rounds $\boldsymbol{\delta}_i=(\delta_i^x,\delta_i^y)$1 to $\boldsymbol{\delta}_i=(\delta_i^x,\delta_i^y)$2: generate an action, terminate if it is Stop, otherwise execute the tool and update state (Lei et al., 5 Oct 2025).

A key consequence is that grounding is not represented as a single monolithic pass over the full screenshot. Instead, the model alternates between reasoning and state-transforming visual operations. This suggests that GUI-Spotlight belongs to the same broad family as active-perception methods, but with its own explicit image registry, offset bookkeeping, and dialogue-style reasoning loop (Lei et al., 5 Oct 2025).

3. Specialized tools and adaptive focus refinement

GUI-Spotlight uses three specialized visual tools: extract, find_color, and crop. All three return a cropped image, an informational message, and the top-left offset relative to the original screenshot, so every intermediate coordinate can be mapped back to the full screen (Lei et al., 5 Oct 2025).

The extract tool is a coarse quadrant-based crop operator. It selects one of the 1/4 regions specified by horizontal position left / center / right and vertical position top / center / bottom, and crops a $\boldsymbol{\delta}_i=(\delta_i^x,\delta_i^y)$3 region. Its role is broad focus narrowing when the model only knows the rough part of the screen where the target lies (Lei et al., 5 Oct 2025).

The find_color tool is a color-guided search operator. It scans the image with $\boldsymbol{\delta}_i=(\delta_i^x,\delta_i^y)$4 patches and stride 10, computes perceptual color difference in CIE Lab space using $\boldsymbol{\delta}_i=(\delta_i^x,\delta_i^y)$5, identifies the patch with minimal $\boldsymbol{\delta}_i=(\delta_i^x,\delta_i^y)$6 to a target RGB value, and centers a window around that patch. This makes the tool particularly suited to targets that are distinguishable by color, such as highlighted controls or colored buttons (Lei et al., 5 Oct 2025).

The crop tool is a fine-grained rectangular crop specified by opposite corners. It is used after coarse narrowing, when the model has enough evidence to perform precise zooming around a candidate area (Lei et al., 5 Oct 2025).

The iterative use of these tools distinguishes GUI-Spotlight from earlier screenshot-only grounding systems that predicted coordinates directly. SeeClick, for example, formulates grounding from screenshot and text as point or box prediction through natural-language output, and its main contribution is grounding pre-training rather than explicit multi-turn visual refinement (Cheng et al., 2024). GUI-Eyes later proposed active perception with crop and zoom tools inside a two-stage RL framework, where the agent learns whether and how to invoke tools, reinforcing the broader importance of adaptive visual focus in GUI grounding (Chen et al., 14 Jan 2026). MEGA-GUI similarly decomposes grounding into coarse ROI selection and fine-grained localization through a bidirectional ROI zoom policy, though its operational structure is a staged modular pipeline rather than GUI-Spotlight’s conversational image-tool loop (Kwak et al., 17 Nov 2025).

4. Training curriculum, data curation, and reinforcement learning objective

GUI-Spotlight is trained in three stages. Stage 1 is supervised warm-up with tool-use trajectories. The authors first run the same inference pipeline with Qwen2.5-VL-72B on filtered UGround data, collect 2561 multi-turn dialogue trajectories, and then perform supervised imitation learning from UI-TARS-1.5-7B and Qwen2.5-VL-7B-Instruct initializations (Lei et al., 5 Oct 2025).

Stage 2 performs reinforcement learning on 12K samples from filtered UGround using a modified GSPO objective:

$\boldsymbol{\delta}_i=(\delta_i^x,\delta_i^y)$7

with normalized advantage

$\boldsymbol{\delta}_i=(\delta_i^x,\delta_i^y)$8

and sequence ratio

$\boldsymbol{\delta}_i=(\delta_i^x,\delta_i^y)$9

The auxiliary term $I_i$0 is a cross-entropy stabilizer applied only to format-valid and result-correct samples, with $I_i$1 in Stage 2 (Lei et al., 5 Oct 2025).

Stage 3 refines the model with 4000 high-resolution samples collected by the authors. In this stage, $I_i$2 is reduced to 0.01, and the sample-selection mechanism changes to bucketed uniform sampling across tool types:

$I_i$3

$I_i$4

$I_i$5

This balancing scheme is intended to encourage exploration across tools instead of concentrating updates on the most common tool categories (Lei et al., 5 Oct 2025).

The data curation pipeline is correspondingly strict. The authors collect 15K high-resolution samples using a Selenium-based headless browser that batch-loads webpages at fixed resolution, detects common interactive elements, extracts readable text, crops element-level images, and stores text and bounding-box metadata; after filtering and additional language/interface checks, this becomes an 11.6K refined high-resolution subset. For UGround filtering, samples are discarded if image Laplacian variance is below 100.0 or if boxes cover less than 1% of image area. Qwen2.5-VL-72B then audits the remaining samples using three criteria: Instruction Quality with threshold $I_i$6, Bounding Box Accuracy with threshold $I_i$7, and Consistency requiring IoU $I_i$8. The reported outcome is that about 50% of UGround is retained (Lei et al., 5 Oct 2025).

The paper’s RL reward is a weighted sum of five components,

$I_i$9

comprising answer reward, crop reward, extract reward, find_color reward, and format reward. The answer reward is sparse and depends on whether the final predicted point lies inside the ground-truth box. Crop reward uses IoU between crop and ground truth. Extract reward gives credit when the extracted quadrant fully contains the target. Find_Color reward checks whether the 200×200 color-match window covers the target. Format reward checks syntactic validity of the tool call (Lei et al., 5 Oct 2025).

The ablations reported in the paper are diagnostic. A dense center-shaped answer reward using normalized Chebyshev distance is compared against the binary sparse reward, and the sparse answer reward is reported as slightly better after convergence. The Crop/Extract reward ratio is also varied, with larger weight on Extract yielding better results because Extract is easier to use and requires less precise coordinates. In RL algorithm comparisons, two alternatives are discarded because they hurt performance: selecting only the most uncertain prompts, and continuously updating the reference policy. The best stabilization variant is tool-filtered positives with extra cross-entropy loss (Lei et al., 5 Oct 2025).

5. Benchmarks and reported performance

GUI-Spotlight is evaluated on ScreenSpot-Pro, UI-Vision, and OSWorld-G. ScreenSpot-Pro is the principal benchmark in the paper because it stresses grounding on 4K-resolution professional interfaces; UI-Vision emphasizes desktop screenshots from 83 applications across six domains; and OSWorld-G contains 564 screenshots covering file operations, application launching, text editing, system configuration, and related OS-level tasks (Lei et al., 5 Oct 2025).

The reported results for the UI-TARS-1.5-7B–initialized variant are as follows.

Benchmark	GUI-Spotlight	Reported comparison
ScreenSpot-Pro	52.8%	V2P-7B 50.6%, GTA-1-7B 50.1%
UI-Vision	23.4	UI-TARS-1.5-7B baseline 18.1
OSWorld-G	62.7	UI-TARS-1.5-7B baseline 61.9

On ScreenSpot-Pro, the same paper reports two initializations. The Qwen2.5-VL-7B-Instruct–initialized model reaches 38.7%, which is described as a gain of 11.9 points over the raw Qwen2.5-VL-7B-Instruct baseline of 26.8%. The UI-TARS-1.5-7B–initialized model reaches the headline 52.8%, and the paper explicitly states that this surpasses V2P-7B at 50.6% and GTA-1-7B at 50.1% while using only 18.5K training samples, compared with 9.6M and 1.56M respectively (Lei et al., 5 Oct 2025).

The ScreenSpot-Pro domain breakdown for the UI-TARS-1.5-7B–initialized model is Development 53.3, Creative 44.4, CAD 51.0, Scientific 52.4, Office 71.3, and Operating System 46.9, with 52.8 overall. The paper states that GUI-Spotlight improves over its backbone in all six domains (Lei et al., 5 Oct 2025).

On UI-Vision, GUI-Spotlight reports Basic 32.1, Functional 30.2, Spatial 9.1, and average 23.4, compared with 18.1 for the UI-TARS-1.5-7B baseline. On OSWorld-G, it reports Text Matching 68.2, Element Recognition 60.6, Layout Understanding 63.2, Fine-grained Manipulation 45.6, and average 62.7, compared with 61.9 for the baseline (Lei et al., 5 Oct 2025).

The paper also evaluates whether iteration alone explains the gains. Two training-free iterative baselines are considered: multi-turn conversational inference with appended tool outputs, and repeated single-turn inference based on a crop centered at the previously predicted click. GUI-Spotlight’s trained multi-step reasoning outperforms both, indicating that the gain is attributed to learned tool use rather than to simple repetition (Lei et al., 5 Oct 2025).

6. Interpretation, research position, and limitations

GUI-Spotlight occupies a specific position within GUI grounding research. Earlier screenshot-only agents such as SeeClick argued that GUI grounding is the key bottleneck for visual GUI agents and showed that improvements in grounding correlate with better downstream agent performance (Cheng et al., 2024). GUI-Actor later reformulated grounding as attention over visual patches with a dedicated <ACTOR> token and verifier, emphasizing coordinate-free region grounding rather than explicit coordinate generation (Wu et al., 3 Jun 2025). V2P calibrated attention through Suppression Attention and Fitts-Gaussian Peak Modeling, arguing that grounding should suppress background regions and peak at the actionable center of the target (Chen et al., 11 Jan 2026). GUI-Spotlight differs from all three by externalizing focus refinement into a multi-turn tool-augmented loop (Lei et al., 5 Oct 2025).

A plausible implication is that GUI-Spotlight’s main contribution is not merely higher benchmark accuracy, but a change in abstraction: full-screen GUI grounding is treated as a sequence of visual state transformations, each with explicit spatial bookkeeping and reinforcement feedback. This places it close to search-oriented frameworks such as ScreenSeekeR, which uses cascaded search to reduce the search area before grounding (Li et al., 4 Apr 2025), and to active-perception systems such as GUI-Eyes, which learns crop and zoom tool invocation under a two-stage RL policy (Chen et al., 14 Jan 2026). Later work such as MEGA-GUI extends the modular principle further by separating coarse ROI deduction and fine-grained grounding into different agents and scales (Kwak et al., 17 Nov 2025).

The limitations stated or implied in the GUI-Spotlight paper are equally specific. Performance remains well below ceiling on difficult high-resolution screens. Tool use depends on syntactically valid commands, making format robustness operationally important. The find_color mechanism is only useful when color cues are genuinely discriminative. The multi-turn loop is slower than one-shot prediction because inference includes repeated tool execution and dialogue expansion. Most importantly, the RL stage is unstable without the auxiliary positive-example cross-entropy term: the authors explicitly report that naïve RL can yield invalid tool syntax, sparse rewards, oscillation, and training collapse (Lei et al., 5 Oct 2025).

Taken together, GUI-Spotlight is best understood as a focus-refinement framework for GUI grounding. It extends the older “focus” idea from static region conditioning to adaptive iterative localization, combines specialized image tools with offset-aware multi-turn reasoning, and shows that curated data plus stabilized RL can make such a strategy competitive on high-resolution GUI grounding benchmarks (Lei et al., 5 Oct 2025).