UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding

Abstract: GUI grounding, which localizes interface elements from screenshots given natural language queries, remains challenging for small icons and dense layouts. Test-time zoom-in methods improve localization by cropping and re-running inference at higher resolution, but apply cropping uniformly across all instances with fixed crop sizes, ignoring whether the model is actually uncertain on each case. We propose \textbf{UI-Zoomer}, a training-free adaptive zoom-in framework that treats both the trigger and scale of zoom-in as a prediction uncertainty quantification problem. A confidence-aware gate fuses spatial consensus among stochastic candidates with token-level generation confidence to selectively trigger zoom-in only when localization is uncertain. When triggered, an uncertainty-driven crop sizing module decomposes prediction variance into inter-sample positional spread and intra-sample box extent, deriving a per-instance crop radius via the law of total variance. Extensive experiments on ScreenSpot-Pro, UI-Vision, and ScreenSpot-v2 demonstrate consistent improvements over strong baselines across multiple model architectures, achieving gains of up to +13.4\%, +10.3\%, and +4.2\% respectively, with no additional training required.

• The paper introduces UI-Zoomer, a training-free adaptive zoom-in method that quantifies uncertainty to refine GUI grounding predictions.
• It employs global multi-sampling and reliability gating to decide when adaptive cropping is needed, ensuring context preservation and improved precision.
• Empirical results show up to +13.4% accuracy gains over fixed cropping, validating its effectiveness in localizing small, ambiguous UI elements.

Uncertainty-Driven Adaptive Zoom-In for GUI Grounding with UI-Zoomer

Problem Formulation and Prior Paradigms

Graphical User Interface (GUI) grounding, the task of localizing interface elements in screens based on natural language descriptions, is a crucial capability for autonomous agents in human-computer interaction. The challenge is particularly acute for small, densely packed elements in modern high-resolution GUIs, where state-of-the-art vision LLMs (VLMs) frequently falter. Prevailing approaches using direct inference and iterative test-time cropping are hampered by two core deficiencies: indiscriminate application of zoom-in (i.e., cropping for all instances regardless of model uncertainty) and static crop ratios (fixed, unadaptive window sizes irrespective of spatial ambiguity). Uniform cropping degrades performance and incurs unnecessary computational overhead, especially harming easy cases by removing essential global context.

Figure 1: Schematic comparison of GUI grounding paradigms—direct grounding, conventional iterative cropping, and UI-Zoomer's adaptive zoom-in strategy.

UI-Zoomer: Adaptive Test-Time Scaling via Uncertainty Quantification

UI-Zoomer introduces a training-free, uncertainty-driven adaptive zoom-in framework, fundamentally reframing test-time cropping as a problem of quantifying model uncertainty at both spatial and token levels. The method operates as follows:

1. Global Multi-Sampling: For every input $(I, q)$, the model stochastically samples $N=8$ bounding box predictions at temperature $T=0.9$, capturing candidate localizations and their token-level confidences.
2. Reliability Gating: A gating mechanism fuses spatial consensus (mean pairwise IoU among candidates) and averaged scalar token-level confidence into a unified score. If this reliability score exceeds a tunable threshold $\tau$, the candidate with maximum peer support and confidence is chosen; otherwise, the instance is deemed uncertain, requiring zoom-in refinement.
3. Uncertainty-Driven Adaptive Cropping: For uncertain cases, the crop window is analytically determined via variance decomposition of candidate boxes—combining inter-sample positional spread and intra-sample box extent—yielding a Gaussian-modeled, per-instance crop radius that contracts for confident cases and expands for ambiguous ones. Outlier filtering retains the top-75% spatially consistent candidates before window fitting. If the window extends beyond the image, an inward shift preserves spatial context.
4. Zoom-In Re-inference and Remapping: A single deterministic re-inference on the cropped region produces the final localization, which is then mapped back to global image coordinates.

   Figure 2: Overview of UI-Zoomer's workflow, showing candidate sampling, reliability gating, choice of consensus voting or adaptive cropping, and uncertainty-driven crop sizing.

Empirical Results and Numerical Insights

Extensive evaluation on three benchmarks—ScreenSpot-Pro, UI-Vision, and ScreenSpot-v2—demonstrates UI-Zoomer's efficacy across both general-purpose and GUI-specialized vision LLMs. Key results:

• ScreenSpot-Pro: Gains up to +13.4% accuracy over baseline, outperforming both naive sampling and prior fixed-ratio cropping methods (e.g., DiMo-GUI, RegionFocus), with pronounced improvements for icon targets in dense interfaces.
• UI-Vision: Improvements up to +10.3% across multiple functional regimes.
• ScreenSpot-v2: Gains up to +4.2%, with larger benefits for desktop/web layouts than mobile UIs.

Adaptive cropping consistently outperforms fixed-ratio alternatives, with accuracy peaking at a Gaussian scale parameter $\gamma=2.5$, gating threshold $\tau=1.0$, and crop minimum size $m=512$. Comparisons show UI-Zoomer surpasses both SFT-trained and RL-trained VLMs on high-resolution benchmarks.

Figure 3: Ablation of sampling temperature $T$ and number of candidates $N$ reveals optimal settings ($T=0.9$, $N=8$0) for crop estimation and accuracy.

Figure 4: Ablation of gating threshold $N=8$1 and Gaussian spread $N=8$2, showing optimal accuracy with moderate thresholds and adaptive crop size.

Technical Analysis and Ablations

UI-Zoomer's design is validated through systematic ablations:

• Gating Score: Combining spatial consensus ($N=8$3) and average token confidence ($N=8$4) produces a more discriminative reliability measure than either signal alone: accuracy increases from 60.81% ($N=8$5 only) to 61.80% (combined).
• Variance Decomposition: Including both intra-sample (predicted box extent) and inter-sample (predicted position spread) variances yields best crop sizing and accuracy.
• Boundary Handling: Inward shifting for out-of-bound crops outperforms shrinkage or clipping.
• Outlier Filtering: Top-75% spatial candidate retention optimizes crop estimation.
• Aspect Ratio: Enforcing a square crop yields +1.24% improvement, mitigating elongated crop artifacts.

Further, gating thresholds show strong correlation between routing decisions and actual localization accuracy (Table~\ref{tab:gating_rationality}).

Figure 5: Distributions of spatial consensus and token confidence signals, illustrating their complementary characteristics for gating.

Representative Case Visualizations

Case studies highlight UI-Zoomer's robustness: successful examples show the transition from scattered initial predictions to accurate localization post-zoom, while failure cases elucidate scenarios (dense, ambiguous layouts) where prediction ambiguity remains unresolved.

Figure 6: Representative successful and failed GUI grounding cases, showing candidate dispersion, adaptive cropping, and final prediction.

Prompt Engineering and Evaluation

A unified prompt template is used across all experiments to ensure consistency and isolate the effects of UI-Zoomer's adaptive logic.

Figure 7: Full prompt template for model inference during empirical evaluation.

Implications, Future Directions, and Conclusion

UI-Zoomer establishes that selective, uncertainty-driven test-time scaling is a robust principle for GUI grounding with VLMs. Its training-free paradigm enables consistent performance boosts across model families without the need for task-specific fine-tuning or architectural modifications. Practically, this approach can be integrated into any existing interface agent pipeline as an inference-time wrapper, yielding substantial gains on critical cases where model uncertainty is high (small, dense, or ambiguous elements).

Theoretically, UI-Zoomer’s reliance on spatial consensus and token confidence for gating and adaptive cropping generalizes to other vision-language tasks involving spatial localization. Future work may extend its logic to hierarchical zoom-in, multi-stage refinement, or broader interactive multimodal agents, and adapt gating criteria to richer uncertainty models such as Bayesian neural networks or ensemble-based spatial reasoning.

Conclusion

UI-Zoomer provides a rigorous, adaptive solution for GUI grounding by quantifying model uncertainty to drive selective zoom-in refinement. Its principled variance decomposition, reliability gating, and per-instance crop sizing lead to consistent numerical improvements across challenging benchmarks and architectures, particularly benefiting compact and ambiguous UI elements. This framework opens the door to further research into uncertainty-aware inference and adaptive scaling for complex multimodal interaction tasks.