Trust the Right Teacher: Quality-Aware Self-Distillation for GUI Grounding

Abstract: Graphical user interface (GUI) grounding requires vision-LLMs (VLMs) to identify small target elements in high-resolution screenshots and predict precise screen coordinates. On-policy self-distillation (OPSD) is a promising post-training approach for this coordinate-sensitive task, since it provides dense token-level teacher signals beyond hard coordinate labels. However, naive OPSD is not well suited to GUI grounding: OPSD evaluates the teacher on student-generated prefixes, the quality of coordinate-token teacher signals can degrade when the prefix has already deviated from the target coordinate, leading to unreliable teacher signal. To mitigate this, We propose quality-aware self-distillation for VLM-based GUI grounding, which improves coordinate-token teacher-signal quality through soft correctness-aware gating and teacher-probability scaling. The soft correctness-aware gate checks whether the teacher's current coordinate-token prediction can still be completed into the ground-truth box under the student-generated prefix. If not, the corresponding teacher signal is down-weighted. Teacher-probability scaling then uses the teacher's confidence as a lightweight factor to further calibrate the strength of the gated supervision. A key empirical finding is that neither component alone improves overall performance, whereas combining them consistently improves performance. This suggests that the two mechanisms play complementary roles: correctness-aware gating suppresses unreliable coordinate-token supervision, while teacher-probability scaling calibrates the strength of the remaining signals. Experiments across six GUI grounding benchmarks show that our method consistently improves the base model and outperforms strong baselines.

• The paper introduces a quality-aware self-distillation paradigm that integrates soft correctness-aware gating and teacher-probability scaling, boosting GUI grounding accuracy to 72.23%.
• It leverages spatial verifiability to calibrate unreliable teacher signals, providing dense, contextualized supervision on high-resolution screenshots.
• Empirical results across six benchmarks demonstrate significant gains over SFT and GRPO, with improvements ranging from +2.16 to +6.37 points over state-of-the-art baselines.

Quality-Aware Self-Distillation for GUI Grounding

Problem Setting and Shortcomings of Conventional Distillation

Graphical user interface (GUI) grounding in vision-LLMs (VLMs) entails precise localization of interface elements from high-resolution screenshots via coordinate prediction, conditioned on natural-language instructions. This coordinate-sensitive task involves predicting small, often visually confusable targets within densely populated scenes. Traditional supervised fine-tuning (SFT) approaches, which train with hard-labeled coordinate pairs, are limited by sparse supervision and are ill-suited for dense token-level learning required for accurate spatial localization. Reinforcement learning variants, such as guided reward-based policy optimization (GRPO), improve outcome-level matching but suffer from sample inefficiency and weak intermediate feedback.

On-Policy Self-Distillation (OPSD) aims to rectify exposure bias and benefit from teacher signal diversity by training on distributions from teacher models, conditioned on student-generated prefixes. Despite their promise, naive OPSD instantiations are problematic for GUI grounding: once a student’s prefix diverges from the target, the associated teacher signals become unreliable, as they merely reflect plausible continuations rather than corrective guidance toward the true coordinate. Consequently, blind imitation can reinforce erroneous beliefs, rendering the supervision not only ineffective but potentially detrimental.

Main Contributions: Reliability-Aware Teacher Signal Calibration

The paper introduces a quality-aware self-distillation paradigm that leverages the unique spatial verifiability of coordinate predictions in GUI grounding to directly assess the reliability of teacher signals. The two core mechanisms—soft correctness-aware gating and teacher-probability scaling—jointly calibrate the supervision at the coordinate-token level, providing a robust alternative to proxy-based criteria such as entropy or perplexity.

• Soft Correctness-Aware Gating: For every coordinate token in the auto-regressive decoding trajectory, a prefix-aware compatibility test is performed to determine whether the most probable teacher-predicted digit can plausibly be completed into a valid coordinate inside the ground-truth bounding box. Rather than a hard discard, a soft gate (with a tunable down-weighting factor $\alpha$, default 0.5) is applied to incompatible tokens, which preserves some degree of corrective support even for off-target prefixes, but reduces their influence on the learning objective.
• Teacher-Probability Scaling: The distillation signal is further weighted by the teacher’s confidence in the predicted digit, modulating the contribution according to estimated certainty. This reduces the risk of amplifying ambiguous or unreliable teacher predictions while prioritizing modes of high-probability, high-quality supervision.

These mechanisms are implemented as token-level, prefix-dependent weights multiplying the reverse-KL distillation loss between student and teacher distributions, in effect yielding a reliability-aware, dense, and contextualized training signal.

Empirical Results and Analysis

Extensive evaluation spans six GUI grounding benchmarks, including ScreenSpot-Pro, ScreenSpot-v2, UIEG, OSWorld-G, OSWorld-G-R, and MMbench-GUI L2, each presenting unique high-resolution localization challenges.

The proposed method exhibits consistent and strong empirical gains:

• Macro-average accuracy: 72.23%
• Improvement over state-of-the-art baseline (GUI-SD): +2.16 points
• Substantial gains over SFT and GRPO: +4.14 and +6.37 points, respectively

Ablation studies demonstrate that neither soft correctness-aware gating nor teacher-probability scaling alone suffice to yield a net accuracy gain on all benchmarks. The integration of both produces complementary effects: gating suppresses supervision from incompatible prefixes, while probability scaling finely tunes the distillation strength, yielding the highest observed dataset-wide improvements.

Further, soft gating (as opposed to hard gating) retains and exploits residual signals from partially compatible or recoverable trajectories, addressing the exposure bias and error accumulation characteristic of autoregressive generation. Scaling coefficient sweeps reveal that overly aggressive amplification can hurt generalization, underscoring the necessity of careful hyperparameter calibration to balance supervision strength with reliability.

Theoretical and Practical Implications

This paradigm exploits spatial verifiability—a property unique to coordinate-prediction tasks—for teacher signal rectification in self-distillation. By making teacher supervision both compatibility- and certainty-aware, the approach moves beyond indirect, proxy-based measures of reliability, instead grounding the training dynamics in explicit task constraints.

Practically, the method requires ground-truth bounding box access during training, limiting direct applicability to domains with spatial annotation. Nonetheless, this work substantiates the value of verifiable supervision criteria for structured prediction and opens methodological avenues for other domains (e.g., program synthesis, robot control) where prediction constraints can be checked at training time.

The findings provide actionable design principles for future GUI grounding models and more generally for self-distillation in tasks where autoregressive decisions can be incrementally checked or rolled back. Incorporating structured verification and probability scaling into distillation routines could become a blueprint for scaling dense, reliability-aware learning signals to ever-larger models and more complex multimodal interaction tasks.

Conclusion

Quality-aware self-distillation for GUI grounding sets a new bar in leveraging task-specific verifiability for teacher signal calibration. The dual mechanism of soft correctness-aware gating and teacher-probability scaling systematically suppresses unreliable teacher guidance and amplifies high-quality supervision, leading to consistent gains across diverse high-resolution GUI benchmarks. This approach highlights the critical importance of reliability-aware distillation in coordinate-structured prediction tasks and prompts further research into generalizing these insights to other domains with structured constraints and verifiable outputs.

For full experimental results, methodology, and discussion, see "Trust the Right Teacher: Quality-Aware Self-Distillation for GUI Grounding" (2606.18101).