BAMI: Training-Free Bias Mitigation in GUI Grounding

Abstract: GUI grounding is a critical capability for enabling GUI agents to execute tasks such as clicking and dragging. However, in complex scenarios like the ScreenSpot-Pro benchmark, existing models often suffer from suboptimal performance. Utilizing the proposed \textbf{Masked Prediction Distribution (MPD)} attribution method, we identify that the primary sources of errors are twofold: high image resolution (leading to precision bias) and intricate interface elements (resulting in ambiguity bias). To address these challenges, we introduce \textbf{Bias-Aware Manipulation Inference (BAMI)}, which incorporates two key manipulations, coarse-to-fine focus and candidate selection, to effectively mitigate these biases. Our extensive experimental results demonstrate that BAMI significantly enhances the accuracy of various GUI grounding models in a training-free setting. For instance, applying our method to the TianXi-Action-7B model boosts its accuracy on the ScreenSpot-Pro benchmark from 51.9\% to 57.8\%. Furthermore, ablation studies confirm the robustness of the BAMI approach across diverse parameter configurations, highlighting its stability and effectiveness. Code is available at https://github.com/Neur-IO/BAMI.

• The paper introduces Bias-Aware Manipulation Inference (BAMI), a training-free method to counteract precision and ambiguity biases in GUI grounding.
• BAMI employs a coarse-to-fine localization strategy and structured candidate selection to enhance spatial resolution and candidate diversity.
• Empirical results on ScreenSpot-Pro show improvements from 51.9% to 57.8%, demonstrating BAMI's effectiveness across various model backbones.

Training-Free Inductive Bias Mitigation for GUI Grounding: An Expert Overview of BAMI

Introduction and Motivation

Graphical User Interface (GUI) grounding—localizing GUI elements via language instructions on high-resolution screen images—remains a fundamental challenge for the development of intelligent interface agents. Despite the rapid advances in large vision-LLMs (MLLMs), their performance on benchmarks such as ScreenSpot-Pro is suboptimal, with complex, high-resolution, and element-dense interfaces exposing critical limitations. The work "BAMI: Training-Free Bias Mitigation in GUI Grounding" (2605.06664) identifies the predominant causes of grounding errors as stemming from inductive biases: precision bias (systematic coordinate localization error in high-resolution, small-object scenarios) and ambiguity bias (selection errors among semantically similar candidates). To address these, the authors introduce Bias-Aware Manipulation Inference (BAMI), a training-free inference methodology that systematically reduces these error modes through two manipulations: coarse-to-fine focus and structured candidate selection.

Figure 1: BAMI achieves accurate localization in complex GUIs without additional training, using structured inference with bias-aware manipulations.

Analysis of Grounding Errors and Inductive Bias

The paper begins with an error decomposition using the Masked Prediction Distribution (MPD) method, which involves perturbing screenshots via masking and observing the distribution of prediction outputs. A rigorous analysis on ScreenSpot-Pro reveals that only 14% of errors derive from true knowledge deficiencies, while 74% are attributable to inductive biases—20% from precision bias and 54% from ambiguity bias. Traditional inference enhancements (e.g., language-driven reasoning or chain-of-thought prompting) do not remedy these strongly spatial or representation-driven error sources.

Figure 2: BAMI mitigates accuracy and ambiguity biases using coarse-to-fine focus and candidate selection, respectively.

Methodology: Bias-Aware Manipulation Inference (BAMI)

Precision Bias Mitigation: Coarse-to-Fine Focus

MLLMs predict GUI coordinates as discretized text sequences, leading to quantization errors, especially problematic for small objects in high-resolution GUIs. BAMI counters this with a coarse-to-fine localization approach: iteratively cropping the region around the initial prediction and redrawing model attention into a narrower window, enhancing spatial granularity. Hyperparameters (crop ratio and iteration count) are tuned to maximize resolution without depriving the model of contextual cues.

Ambiguity Bias Mitigation: Structured Candidate Selection

Ambiguity bias arises from the misalignment between cross-entropy training objectives (minimizing token edit distance) and real-space coordination (requiring low Euclidean error). To overcome this, BAMI generates multiple diversified candidate boxes by masking previously predicted areas in each iteration. An auxiliary correction module—either a prompt-engineered LLM (e.g., GPT-5, Qwen3-VL-8B) or a local model—applies domain-specific selection criteria to identify the most plausible grounding among the candidates.

Figure 3: BAMI procedure: (1) initial prediction and cropping; (2) diverse candidate generation via masking; (3) prompt-based ranking and selection.

Empirical Validation

Performance Across Benchmarks

Extensive benchmarks demonstrate the efficacy of BAMI. On ScreenSpot-Pro, BAMI improves the accuracy of TianXi-Action-7B from 51.9% to 57.8% without any additional fine-tuning, significantly outperforming existing test-time techniques and even models leveraging reinforcement learning for spatial grounding. The approach generalizes: applied to diverse open-source backbones (UI-TARS, OS-Atlas, UGround), BAMI yields consistent, substantial gains.

Figure 4: BAMI consistently enhances performance across all major GUI grounding model backbones on ScreenSpot-Pro.

Ablation and Robustness Analyses

Ablations confirm the criticality of both manipulations. Disabling either the coarse-to-fine focus or candidate selection diminishes performance (e.g., 51.9% baseline, 55.2% only with coarse-to-fine, 54.3% only with candidate selection; full BAMI: 57.8%). Extensive tests on the hyperparameters (crop ratio and number of iterations) show BAMI's stability and robustness.

Figure 5: Model accuracy is robust to variations in crop ratio and iteration count when eliminating precision bias.

The method demonstrates neither selectivity nor bias regarding target types, maintaining improvements for both icons and text-bound elements. Correctness visualizations further illustrate BAMI's ability to systematically correct previously erroneous localizations, often in highly cluttered or ambiguous semantic regions.

Figure 6: BAMI's corrected predictions (blue) rectify baselines errors (red), accurately matching ground-truth (green) in challenging GUI contexts.

Mechanistic Insights and Prompt Engineering

A non-trivial insight from this work is that candidate diversity is key: naive random sampling for candidate generation fails, as models converge to a single region (see supplementary Figure 7). In contrast, targeted masking enforces candidate diversity. Additionally, prompt design for the correction model must inject explicit coordinate-space priors and domain knowledge—vanilla selection prompts fail to attenuate ambiguity bias, while prompts encoding GUI-specific principles and stepwise reasoning are essential.

Practical and Theoretical Implications

BAMI's demonstration that significant grounding improvements can be unlocked via inference-only manipulations—without further parameter updates or training data—has both practical and conceptual significance. Practically, it enables improved performance on test-only or privacy-sensitive scenarios where retraining is infeasible. Theoretically, it suggests structural constraints and post-hoc orchestrations can partially compensate for limitations in model architectures or objectives.

Moreover, the success of a local, efficiently trained 8B correction model indicates that test-time error correction need not rely on megascale external LLMs, opening paths to cost-efficient and private GUI agent deployments.

Figure 8: Masked Prediction Distribution visualizations reveal error attribution to knowledge gap, precision bias, or ambiguity bias.

Future Directions

This methodology naturally extends to any vision-language task characterized by output representations with mismatched or quantized coordinate spaces and high semantic ambiguity. Possible avenues for future work include:

• Joint modeling or end-to-end integration of structured candidate generation and correction within a single network.
• Automated task-specific prior extraction for prompt engineering.
• Dynamic, uncertainty-aware iteration and cropping strategies steered by online model behavior.
• Application to other high-resolution structured prediction domains beyond GUIs.

Conclusion

BAMI provides a rigorous, empirically validated, and versatile methodology for addressing dominant inductive biases in GUI grounding. Through structured inference and the explicit handling of model limitations, it establishes a robust and training-free framework that extracts significant additional utility from existing models. The results underscore the importance of inference strategies and highlight fruitful directions for future spatial reasoning research in multimodal systems.