See, Point, Refine: Multi-Turn Approach to GUI Grounding with Visual Feedback

Abstract: Computer Use Agents (CUAs) fundamentally rely on graphical user interface (GUI) grounding to translate language instructions into executable screen actions, but editing-level grounding in dense coding interfaces, where sub-pixel accuracy is required to interact with dense IDE elements, remains underexplored. Existing approaches typically rely on single-shot coordinate prediction, which lacks a mechanism for error correction and often fails in high-density interfaces. In this technical report, we conduct an empirical study of pixel-precise cursor localization in coding environments. Instead of a single-step execution, our agent engages in an iterative refinement process, utilizing visual feedback from previous attempts to reach the target element. This closed-loop grounding mechanism allows the agent to self-correct displacement errors and adapt to dynamic UI changes. We evaluate our approach across GPT-5.4, Claude, and Qwen on a suite of complex coding benchmarks, demonstrating that multi-turn refinement significantly outperforms state-of-the-art single-shot models in both click precision and overall task success rate. Our results suggest that iterative visual reasoning is a critical component for the next generation of reliable software engineering agents. Code: https://github.com/microsoft/precision-cua-bench.

• The paper demonstrates that iterative visual feedback improves pixel-level localization, boosting GPT-5.4 accuracy from 20.6% to 38.1% and reducing error from 80.4 to 57.3 pixels.
• The study introduces a robust VS Code instrumentation pipeline that aligns symbolic cursor positions with pixel measurements to provide high-fidelity, dense GUI data.
• The paper highlights the critical impact of prompt specialization and feedback-aware finetuning, showing model-dependent correction patterns for precise GUI grounding.

Multi-Turn Iterative GUI Grounding with Visual Feedback

Motivation and Problem Setting

Graphical user interface (GUI) grounding is an indispensable module for Computer Use Agents (CUAs) operating in automation, developer tooling, and productivity agents, serving as the explicit mapping between natural-language instructions and actionable screen regions. While state-of-the-art multimodal agents achieve reliable localization on coarse elements (buttons, panels), the high-density, editing-level regime characterized by pixel-precise cursor placement in code editors such as VS Code remains unsolved. Minor localization error can lead to semantic failures (e.g., incorrect cursor insertion point between characters), highlighting the precision gap unaddressed by previous benchmarks and methods.

The paper “See, Point, Refine: Multi-Turn Approach to GUI Grounding with Visual Feedback” (2604.13019) empirically investigates the hypothesis that iterative refinement, inspired by human pixel-level visual correction, can consistently reduce displacement errors in dense coding environments. The method introduces a closed-loop, multi-turn grounding system where the agent consumes visual feedback—specifically, a marked screenshot of previous prediction error (red cross)—to iteratively adjust and improve coordinate predictions.

Data Collection and System Architecture

Robust, high-fidelity data is critical for benchmarking pixel-level GUI grounding. The authors construct a VS Code instrumentation pipeline that precisely aligns symbolic cursor positions ($(file, line, col, character)$) with measured screen-absolute coordinates, leveraging a process-separated architecture bridged via localhost WebSocket communication.

Figure 1: The collector synchronizes symbolic cursor state from the VS Code extension host with DOM-side pixel measurements, capturing per-character GUI metadata for supervision and evaluation.

This system captures dense, per-character records, including exact screen coordinates, typography settings, window geometry, and device scaling factors. The resulting dataset (257 samples; line/word/character targets) supports deterministic replay and cross-platform reproducibility, and is particularly adapted for evaluating cursor placement tasks in text-dense IDE environments.

Multi-Turn Refinement Protocol

Grounding is formalized as an iterative correction process with explicit visual feedback. The initial turn predicts coordinates from raw screen and instruction. On subsequent attempts, the agent receives the original screenshot overlayed with a red cross at the last prediction as well as the numerical $(x, y)$, and is prompted to self-correct. Feedback is multimodal: both the image and a textual template indicating failure and previous coordinates.

Each iteration leverages a family of system prompts (baseline, chain-of-thought, cursor-aware, stepwise, minimal, visual-anchoring) and feedback templates (weak “previous incorrect” versus strong “reason about cross-relative location”). These are designed to control the agent’s spatial reasoning granularity, task priors, and explicit anchoring behavior.

Experimental Evaluation

The experiments evaluate iterative, training-free correction (i.e., no model reweighting or RL finetuning) across leading LMMs: OpenAI GPT-5.4, Anthropic Claude 3.5 Sonnet/Opus, and Alibaba Qwen-3.5-9B. Core metrics include turn-indexed accuracy, localization distance (center and bounding box), and correction rate, stratified by prompt/feedback variant and GUI element granularity.

Key empirical findings:

• Iterative visual feedback robustly improves accuracy and reduces localization error for all models. For example, GPT-5.4 accuracy rises from 20.6% (turn 1) to 38.1% (turn 2) under cursor-aware prompting, with average distance decreasing from 80.4 to 57.3 pixels.
• Prompt specialization is critical in maximizing correction rate and geometric precision. Cursor-aware prompts yield highest post-refinement accuracy, while visual-anchor prompts minimize point-to-center distances.
• Model-level correction patterns are non-uniform: Claude achieves the lowest mean distance (31.3 pixels) but lower final accuracy, suggesting sharper but less robust spatial selectivity; Qwen-3.5-9B shows measurable but modest iterative improvement, with absolute performance trailing GPT-5.4 and Claude.
• Feedback-aware finetuning closes the performance gap. When Qwen-3.5-9B is finetuned with visual feedback, its accuracy surpasses GPT-5.4 (41.6% vs. 38.1%), with average distance as low as 24.8 pixels, and exceeds baseline coordinate prediction models such as GUI-G2 and GUI-Actor.

Comparative Analysis and Practical Implications

The paper’s closed-loop, feedback-guided protocol sets a new standard in evaluation for pixel-level GUI grounding in dense text/code environments. By directly exposing LMMs to their prediction errors in visual space and iteratively querying for corrections, the method reveals both the current limitations and the potential of prompt engineering and visual feedback as effective augmentation strategies even without retraining.

• Practical implications: Iterative refinement—especially when combined with spatially specialized prompting—substantially boosts reliability for developer-assistive agents working in complex IDEs. The protocol is training-agnostic, providing immediate gains atop API-frontier models, and is extensible to more sophisticated feedback (e.g., richer overlays, contrastive demonstrations, or highlight propagation).
• Theoretical implications: Correction dynamics vary not just by model scaling, but by their internal representation of geometric error and their ability to parse sparse, spatially explicit feedback. This reveals key axes for future research: spatial generalization, visual anchoring, human-in-the-loop correction, and prompt-conditioned agent introspection.
• Limitations: The benchmark size is limited and constrained to single-editor (VS Code) setups, which may not capture the distribution shifts in other applications or less-regularized GUIs. The method depends on the model's ability to effectively parse and reason about explicit spatial overlays and coordinates, which may not generalize to more ambiguous or lower-resolution settings.

Future Directions

Future research should expand the scale and diversity of dense cursor-level datasets, explore multimodal feedback signals (e.g., interactive tooltips, animated markers), and combine closed-loop correction with post-hoc reward modeling or imitation finetuning for sample-efficient adaptation. The refinement protocol is amenable to hierarchical agent architectures wherein prediction, error diagnosis, and correction can be modularized and independently optimized. Scaling up such human-inspired correction strategies may be critical for achieving robust general-purpose agentic interaction in unconstrained, professional UI environments.

Conclusion

“See, Point, Refine” (2604.13019) demonstrates that iterative, visual-feedback-driven GUI grounding reliably corrects pixel-level localization errors in dense coding interfaces, achieving substantial gains with only prompt and protocol modifications. Model-agnostic improvement is observed across LMMs, with prompt specialization further closing the gap to supervised finetuned baselines. The proposed approach underscores the value of closed-loop correction and tailored prompting for next-generation multimodal agents, motivating future work toward scalable, feedback-intensive, high-precision GUI interaction systems.