GUI Gaussian Grounding Rewards (GUI-G²)

• GUI Gaussian Grounding Rewards are a probabilistic framework that replaces sparse binary rewards with continuous, Gaussian-based signals for precise GUI grounding.
• It integrates Gaussian point and coverage rewards to provide dense learning gradients aligned with human clicking patterns, enhancing spatial accuracy.
• Experimental results show that GUI-G² improves accuracy by up to 9.4% on benchmarks, offering greater sample efficiency and robustness in GUI automation.

GUI Gaussian Grounding Rewards (GUI-G²) refer to a principled reward modeling framework for graphical user interface (GUI) grounding tasks that replaces sparse binary rewards with continuous, Gaussian-based spatial reward signals. This approach directly addresses the limitations of traditional hit-or-miss supervision by leveraging dense, behaviorally-motivated gradients based on human clicking distributions, thereby advancing the state of the art in reinforcement learning-driven grounding and spatial reasoning for autonomous GUI agents (Tang et al., 21 Jul 2025).

1. Motivation and Theoretical Foundations

Conventional reinforcement learning for GUI grounding typically deploys binary rewards: a prediction falling strictly within the target bounding box receives a reward of 1, whereas predictions outside the region receive 0. This “hard boundary” scheme leads to extremely sparse learning signals, with models receiving no gradient information for near-miss predictions, impeding both convergence and generalization. Moreover, this formalism ignores empirical findings from human-computer interaction research: studies such as those on the AITW dataset show that actual human click locations form 2D Gaussian distributions centered on the intended target, with dispersion determined by Fitts’ law.

GUI-G² reconceptualizes the reward mechanism by treating each GUI target as a spatial probability distribution, specifically a two-dimensional Gaussian, and assigning exponentially decaying rewards as a function of distance from the correct center. This continuous relaxation transforms sparse classification (hit/miss) into a dense, differentiable optimization problem, enabling more effective reinforcement learning.

2. Gaussian Point and Coverage Reward Mechanisms

GUI-G² incorporates two synergistic reward mechanisms:

2.1 Gaussian Point Rewards

Each GUI element is modeled as a 2D Gaussian distribution centered at its centroid $\mu_{gt} = (c_x^{gt}, c_y^{gt})$ with an adaptive covariance matrix $\Sigma_{gt}$, which scales according to element size. The point reward for a predicted coordinate $\mu_p = (c_x^p, c_y^p)$ is defined as:

$$R_{\text{point}} = \exp\left[ -\frac{1}{2} \left( \frac{(c_x^p - c_x^{gt})^2}{\sigma_x^2} + \frac{(c_y^p - c_y^{gt})^2}{\sigma_y^2} \right) \right]$$

where $\sigma_x$ and $\sigma_y$ denote the standard deviations along x and y axes, scaled adaptively.

This reward is maximized only at the centroid, providing a strong signal for precise localization, but decreases smoothly with increased spatial error, offering a rich gradient even for off-center predictions.

2.2 Coverage Rewards

To capture the spatial extent of GUI elements and not just the centroid alignment, GUI-G² introduces a coverage reward. This measures the overlap between the predicted and ground-truth Gaussian distributions using the Bhattacharyya coefficient:

$$R_{\text{coverage}} = \exp\left[ - \frac{1}{8} (\mu_p - \mu_{gt})^\mathrm{T} \Sigma^{-1} (\mu_p - \mu_{gt}) - \frac{1}{2} \ln \left( \frac{\det \Sigma}{\sqrt{\det \Sigma_p \det \Sigma_{gt}}} \right) \right]$$

Here, $\Sigma$ is the average covariance, and $\Sigma_p$, $\Sigma_{gt}$ are the predicted and true covariances. This reward encourages both correct center alignment and scale matching, ensuring that predictions match the size and shape of the true element.

The aggregate reward is given by:

$$R_{\text{total}} = \nu \cdot R_{\text{point}} + \gamma \cdot R_{\text{coverage}}$$

with tunable weights $\nu$ and $\gamma$.

3. Adaptive Variance for Scale Robustness

GUI-G² addresses the inherent variability of element sizes in GUIs (ranging from small icons to large components) via an adaptive variance mechanism. The Gaussian’s standard deviations are scaled as:

$$\sigma_x = \alpha \cdot (x_2 - x_1), \qquad \sigma_y = \alpha \cdot (y_2 - y_1)$$

where $(x_1, y_1, x_2, y_2)$ defines the bounding box of the element, and $\alpha$ is a tuning parameter. This adaptivity ensures that larger GUI elements permit greater spatial tolerance, while smaller elements demand sharper precision. Consequently, the reward surface is properly calibrated across heterogeneous element scales, which is crucial for real-world robustness (Tang et al., 21 Jul 2025).

4. Experimental Evidence and Quantitative Impact

Experimental results substantiate GUI-G²’s advantages over prior reward regimes:

• Benchmarks: On ScreenSpot, GUI-G²-7B attains 92.0% accuracy; on ScreenSpot-v2, 93.3%; and on the challenging ScreenSpot-Pro, 47.5%, exceeding the previous state-of-the-art by up to 9.4% (Tang et al., 21 Jul 2025).
• Sample Efficiency: The approach demonstrates superior robustness and generalization, especially in data-scarce or high-variation layouts, due to its rich gradient information.
• Learning Dynamics: Ablation studies reveal that continuous Gaussian rewards enable smoother convergence and more stable learning dynamics than both strict binary rewards and simpler distance-based schemes.

Empirical analysis also shows that the continuous reward surface helps models recover from poor initial predictions by steadily guiding outputs toward the true target, in contrast to binary rewards which can stagnate once predictions fall outside the bounding box.

5. Connections to Broader GUI Grounding Research

The GUI-G² framework builds on and extends several strands of recent GUI grounding research:

• Dense Reward Design: Efforts such as pointwise dense reward functions in RL-based GUI grounding systems (Yuan et al., 18 May 2025) set the stage for probabilistic, spatially dense feedback.
• Region-Aware and Two-Stage Methods: Approaches incorporating region proposals and zoom-in refinement (Park et al., 8 Jul 2025) benefit from Gaussian-like decay in their reward surfaces, as continuous feedback better aligns with incremental localization refinement.
• Uncertainty and Calibration: Methods advocating for reward shaping using Gaussian distributions for uncertainty modeling and output calibration (Yang et al., 20 Dec 2024, Liu et al., 15 Apr 2025, Lee et al., 21 May 2025) find direct realization in GUI-G²’s probabilistic rewards.
• Test-Time Aggregation: Techniques such as kernel density estimation (KDE) for coordinate aggregation at inference (Lee et al., 21 May 2025) reflect the “Gaussian-weighted” confidence perspective formalized in GUI-G².

A plausible implication is that the adoption of Gaussian reward modeling will catalyze further innovations in multimodal spatial reasoning, particularly in interactive and open-ended GUIs exhibiting substantial visual and contextual variability.

6. Mathematical Formalism

The key mathematical ingredients of GUI-G² are as follows:

• Gaussian representation for a GUI element:

$$N(x; \mu, \Sigma) = \frac{1}{2\pi \sqrt{|\Sigma|}} \exp\left(-\frac{1}{2} (x - \mu)^\mathrm{T} \Sigma^{-1} (x - \mu)\right)$$

• Gaussian point reward: as described above.
• Coverage reward via Bhattacharyya coefficient.
• Adaptive variance mechanism.
• Total reward as a linear combination:

$$R_{\text{total}} = \nu \cdot R_{\text{point}} + \gamma \cdot R_{\text{coverage}}$$

These formulas underlie the policy objectives in reinforcement learning-based GUI grounding agents, enabling the transformation of GUI grounding from sparse label classification to gradient-rich continuous optimization.

7. Practical Implications and Future Directions

GUI-G² delivers practical advances for real-world GUI automation, with high accuracy on diverse benchmarks and robustness to out-of-distribution layouts. Its dense spatial gradients facilitate both sample-efficient learning and transferability to unseen interfaces. By aligning reward modeling with human behavior-derived distributions, GUI-G² enhances generalization and reliability in automated GUI interaction domains.

Future work is likely to target computational optimization (e.g., model compression or real-time deployment in high-resolution environments) and further advances in multimodal reasoning—potentially integrating richer semantic understanding and attention-guided refinement, especially for complex icons and dense UIs.

GUI Gaussian Grounding Rewards (GUI-G²) thus represent a paradigm shift in GUI grounding research, unifying behavioral, statistical, and reinforcement learning principles into a mathematically principled framework for precise, robust, and generalizable spatial reasoning in interactive interfaces (Tang et al., 21 Jul 2025).