GUI-Cursor: Interactive Grounding in GUI Design

• GUI-Cursor is a grounding approach that uses iterative cursor movement, visual feedback, and spatial reasoning to localize targets on screenshots.
• It reformulates one-shot coordinate regression into an interactive search process that improves accuracy on benchmarks like ScreenSpot-v2 and ScreenSpot-Pro.
• It leverages reinforcement learning with a detailed reward structure and cursor-centric focusing to optimize cursor trajectories and mitigate prediction brittleness.

GUI-Cursor denotes a cursor-grounded approach to graphical user interface interaction in which a model uses visible cursor movement, visual feedback, and spatial reasoning to localize the intended target on a screenshot. In its specific formulation as a GUI grounding framework, GUI-Cursor replaces one-shot coordinate regression with an interactive search process: a vision-LLM moves a virtual cursor, inspects the rendered result, and decides whether to move again or stop. This reframing was introduced to address the brittleness of direct coordinate prediction on high-resolution, densely structured GUIs, and it improves grounding accuracy on both ScreenSpot-v2 and ScreenSpot-Pro (Zhao et al., 25 Sep 2025).

1. Historical antecedents of cursor-based GUI control

Long before GUI-Cursor was framed as an interactive grounding policy, cursor control had already been treated as a hands-free and perception-driven problem. A survey of eye-cursor control systems describes the central motivation succinctly: conventional devices such as mice, trackballs, and light pens require the user’s hands, which is inconvenient or impossible in hands-busy, hands-free, or physically challenged scenarios. The same survey emphasizes three recurrent technical difficulties in eye-based GUI control: uncomfortable or undesirable head attachments, reduced robustness or constrained head motion in non-attached systems, and inaccurate selection on small or crowded GUIs. It then describes two patented responses to these difficulties. One, “Assisting selection of GUI elements,” predicts the user’s intended element from candidate GUI characteristics, moves the pointer to the predicted item, and, if the pointer lies in a border region, “pulls it on to the nearest selectable item.” The other, “System and methods for controlling automatic scrolling of information on a display or screen,” uses eyeball/head tracking, gaze-direction estimation, and screen gaze coordinates to drive hands-free scrolling, with eye scroll control regions segmented into concentric circles (Mishra, 2014).

A different line of work replaced specialized trackers with commodity vision. “Mouse Simulation Using Two Coloured Tapes” uses a webcam, background subtraction, skin segmentation, HSV colour detection, and noise reduction to emulate pointer movement and mouse actions. In that system, a yellow tape on the index finger controls cursor movement, while a red tape on the thumb supports click recognition through distance thresholds and waiting-time logic. The paper discusses absolute position mapping and weighted speed cursor control, notes an expected jump of up to 4 pixels under resolution mismatch, and reports that the velocity-based alternative can be sluggish on machines that cannot process more than about 15 fps. Its final design chooses absolute fingertip positioning, with left click, right click, and double click determined by the relations among $D$, $D'$, and $d$, plus a waiting time of about 7 seconds for right click and double click (Kumar et al., 2012).

Wearable cursor controllers pursued yet another route. “Head-mouse” proposes a wearable wireless cursor controller that uses head tilt, two IR LED / photodetector pairs, a TI CC2541 MCU, BLE, and a host-side GUI / HID interface. The system samples at 20 Hz with a 5% duty cycle, smooths the raw signal with a moving average window of 15 samples, and supports both joystick mode and direct-mapped mode. Joystick mode is threshold-based and relative; direct mapping is absolute and calibrated over 4 seconds of maximal motion. In the reported evaluation, joystick mode outperformed direct mapping, with higher path efficiency and higher throughput, and the paper explicitly states that joystick mode had better controllability. At the same time, the evaluation involved only 2 users, and clicking was not yet implemented (HeydariGorji et al., 2020).

Taken together, these antecedents establish the persistent design constraints that later reappear in GUI grounding: comfort, calibration, robustness to head or hand motion, high-resolution small-target localization, and the need for feedback rather than raw coordinate emission. GUI-Cursor inherits that problem structure, but relocates it into a modern VLM-and-RL setting.

2. Reformulating GUI grounding as interactive cursor search

GUI-Cursor starts from a critique of standard GUI grounding. In the conventional formulation, the model sees a screenshot $O$ of size $W \times H$ and an instruction $I$, and must directly predict a coordinate $(x,y)$, with correctness defined by whether $(x,y)\in B$ for the target bounding box $B$. The paper argues that this one-shot setup is brittle because spatial semantic alignment is hard: the model must identify the intended button, text, or icon, map that latent understanding into discrete coordinate tokens, and do so without any visual confirmation of where its own prediction lands (Zhao et al., 25 Sep 2025).

GUI-Cursor therefore reframes GUI grounding as an interactive cursor-based search problem. The cursor starts at the center of the screenshot. At each step, the model receives the current screenshot with the cursor rendered at its latest position. It then identifies the intended target element from the instruction, reasons about where the cursor currently is relative to that target, and decides either to move the cursor to a new coordinate or to stop if the cursor is already on the target. The generated response has an explicit two-part structure: a “think” portion that analyzes target, cursor location, and spatial relation, and an “answer” portion that outputs either a new coordinate $(x_t,y_t)$ or the special action STOP (Zhao et al., 25 Sep 2025).

This interaction loop makes the cursor a visible spatial anchor. Instead of learning only from static supervision, the model learns to inspect its own predicted location in context. A common misconception is that GUI grounding is merely a coordinate-regression problem; GUI-Cursor treats it as a search-and-verification problem, in which the rendered cursor closes the loop between prediction and spatial verification.

3. Reward design, reinforcement learning, and cursor-centric focusing

Training in GUI-Cursor uses multi-step online reinforcement learning with GRPO. For each instruction, the model samples a batch of trajectories, and the trajectory-level reward is defined as

$D'$0

where $D'$1 if the output obeys the required think/answer format and $D'$2 otherwise. The core search reward combines a dense geometric final-position reward with four penalties designed to suppress degenerate cursor behavior: false stop, false move, false direction, and repeated position. These are aggregated as

$D'$3

with $D'$4 (Zhao et al., 25 Sep 2025).

The final-position term $D'$5 is dense rather than binary. If the final cursor lies inside the target box, the reward increases as the cursor approaches the center of the box. If it lies outside, the reward decreases with the Euclidean distance to the nearest edge of the target box, after normalization by image width and height. This reward structure encourages not only success but center-seeking precision. The four penalties regulate search discipline: false stop penalizes stopping off target; false move penalizes leaving the target after having reached it; false direction penalizes ending farther from the target than the first predicted location; repeated position penalizes predicting the same coordinate multiple times (Zhao et al., 25 Sep 2025).

A second practical mechanism is cursor-centric focusing, or ccf. During training, very large screenshots are downscaled to a manageable maximum resolution $D'$6, reported as $D'$7. During inference on larger images, the model first makes a coarse prediction on the full screen, then crops a region of size $D'$8 around that prediction and continues interactive refinement within the focused crop. This reduces the need to process the full high-resolution screen at every step while preserving high-resolution local reasoning where needed. The paper explicitly notes, however, that ccf can fail if the first prediction is too far off, because the target may fall outside the crop (Zhao et al., 25 Sep 2025).

The resulting training procedure is not merely “RL on clicks.” It is PPO-like clipped policy optimization applied to entire cursor trajectories rather than single-step actions. This suggests that GUI-Cursor’s main novelty lies in treating the search path itself as an object of optimization.

4. Empirical performance, movement behavior, and failure modes

GUI-Cursor reports state-of-the-art results on two GUI grounding benchmarks. Gui-Cursor-7B achieves 93.9% on ScreenSpot-v2 and 56.5% on ScreenSpot-Pro. Relative to the base Qwen2.5-VL-7B setup, the improvement is reported as $D'$9 on ScreenSpot-v2 and $d$0 on ScreenSpot-Pro. On ScreenSpot-Pro, the paper further highlights that Gui-Cursor surpasses GTA1 by 6.4 points (Zhao et al., 25 Sep 2025).

The behavior analysis is as important as the headline scores. GUI-Cursor learns to solve the problem within two steps for 95% of instances, while still adapting to use more steps on more difficult examples. With ccf enabled, more than one movement step occurs on only 0.5% of ScreenSpot-v2 examples, but on 9.4% of ScreenSpot-Pro examples, and those multi-step cases tend to involve smaller target objects. The training dynamics are described as a three-phase pattern: a cold-start phase in which the model learns the output format, a single-step grounding phase in which one-step accuracy improves, and a later multi-step phase in which trajectory reward, response length, and movement count all increase (Zhao et al., 25 Sep 2025).

The paper also studies whether off-the-shelf VLMs can perform the iterative cursor task without specialized training. GPT-4o improves from 17.5% in the standard one-step setup to 21.7% with direct move and 25.5% with relative move over 10 steps. By contrast, Qwen2.5-VL-7B performs poorly in prompt-only iterative mode: direct move reaches 36.3%, while relative move collapses to 1.3%. The authors use this to argue that good one-step grounding does not necessarily imply robust cursor-based spatial reasoning (Zhao et al., 25 Sep 2025).

The ablations reinforce the same interpretation. Removing any of the trajectory penalties hurts performance; the false-stop penalty is particularly important because it prevents collapse into a single-step policy that ignores visual feedback. Removing the thinking process also substantially reduces accuracy. One explicit limitation remains ccf sensitivity: the crop excludes the true element in about 10.3% of ScreenSpot-Pro examples when the initial guess is too far from the target (Zhao et al., 25 Sep 2025).

5. Relation to contemporary GUI grounding and agent systems

GUI-Cursor belongs to a larger shift away from brittle, purely symbolic GUI automation. Auto-GUI, for example, argues that GUI agents should stop converting interfaces into text and acting through application-dependent APIs. Its multimodal encoder-decoder instead reads screenshots directly, fuses visual features with the natural-language goal and action history, and predicts a full action command including action type, touch point, lift point, and typed text. On AITW, Auto-GUI$d$1 reaches 74.27% overall action matching accuracy, with action type prediction around 90%, while the remaining errors are concentrated in low-level spatial decisions such as click region and scroll direction (Zhang et al., 2023).

Other systems attack the same localization problem with different decomposition strategies. Explorer specializes to a target application, trains an FCOS interactable detector per GUI, reuses detector features in a screen-similarity model, and maps user traces into GUI state graphs for replay and voice-based navigation. Its scratch-trained interactable detector reaches mAP@0.5 values such as 0.9710 on KhanAcademy and 0.9356 on Spotify, and cross-platform trace replay from MacOS recording to Windows 11 replication yields 55.6% overall accuracy (Chaimalas et al., 12 Apr 2025). AdaZoom-GUI instead combines an instruction refinement module with a conditional zoom-in strategy and GRPO-trained point-and-box prediction; on ScreenSpot-Pro, the reported average rises from 61.6 for AdaZoom-GUI-4B to 70.6 with conditional zoom-in and 76.8 with Qwen3.5-397B-A17B refinement (Pei et al., 18 Mar 2026). DRS-GUI takes a training-free route, adding Focus, Shift, and Scatter actions scheduled by MCTS; on ScreenSpot-Pro it improves Qwen2.5-VL-7B from 26.8 to 40.9 and UGround-V1-7B from 31.4 to 45.7 (Liu et al., 15 May 2026).

A related but distinct trend emphasizes structured observation and memory. MGA models each step as an independent environment state $d$2, where the current screenshot, task-agnostic spatial information, and structured memory are separated explicitly. It reports 54.6% overall accuracy on OSWorld among pure pyautogui-style methods and frames this as an “observe first, then decide” alternative to long raw trajectory replay (Cheng et al., 28 Oct 2025).

Within this landscape, GUI-Cursor is distinguished by making the cursor itself part of the observation and part of the optimization target. It does not only refine a region, refine an instruction, or compress history; it trains the model to inspect the spatial consequences of its own cursor placement.

6. Extensions of the cursor paradigm beyond one-shot clicking

Recent work extends the cursor from a click locator into a general interaction primitive. ShowUI-$d$3 argues that existing GUI agents rely on discrete click predictions $d$4, which prohibit free-form, closed-loop trajectories such as dragging a progress bar. It therefore unifies clicks and drags in a sequence of triplets $d$5, treats clicks as degenerate drags, and uses a flow-based generative model to predict incremental cursor adjustments from continuous visual observations. The associated ScreenDrag benchmark contains 505 real-world drag tasks across five domains and is paired with 20K drag trajectories. ShowUI-$d$6 reports 78.55% overall endpoint accuracy, 159.05 px overall trajectory error, and 26.98% online success rate, outperforming the listed proprietary and open-source GUI agents on that benchmark (Hu et al., 31 Dec 2025).

The cursor metaphor has also been extended into XR. World Mouse reinterprets the familiar 2D desktop mouse for complex 3D scenes through within-object interaction, which uses surface normals for precise cursor placement, and between-object navigation, which leverages interpolation to traverse empty space. By combining semantic segmentation and mesh reconstruction, the system treats physical objects as interactive surfaces and supports screen-to-world transitions, object manipulation, and continuous cursor movement across real and virtual objects. The paper explicitly argues that cursor-based interaction should not be seen as obsolete in spatial computing (Tütüncü et al., 11 Mar 2026).

A further extension treats the cursor as an explanatory cue rather than an actuator. GUI Narrator addresses GUI video captioning by using the cursor as an inherent visual prompt for both spatial and temporal grounding. Its Act2Cap benchmark contains 4,189 video-captioning samples, and the framework combines a cursor detector, keyframe selection, and high-resolution crops around the cursor. In the reported results, the prompting pipeline raises GPT-4o from 19.5 to 31.8 average score and raises fine-tuned QwenVL-7B from 9.73 to 23.5 (Wu et al., 2024).

These developments suggest a broader interpretation of GUI-Cursor. The cursor is no longer only a final output coordinate; it functions as a search trace, a control trajectory, a cross-reality pointer, and a visual prompt for action understanding. In that sense, GUI-Cursor marks a transition from coordinate prediction toward cursor-mediated spatial reasoning as a general organizing principle for GUI interaction.