ScreenSpot-Pro Benchmark Overview

• ScreenSpot-Pro Benchmark is a comprehensive evaluation suite that measures multi-modal models' ability to accurately ground full-screen GUIs in professional, high-resolution environments.
• It features 1,581 expert-annotated screenshots paired with natural language instructions across diverse applications and operating systems to mimic real-world computing scenarios.
• The benchmark drives innovation with methods like ScreenSeekeR visual search and continuous reward modeling, enhancing spatial reasoning and precise GUI localization.

ScreenSpot-Pro Benchmark is a rigorously constructed evaluation suite designed to measure the grounding capabilities of multi-modal LLMs (MLLMs) and vision-LLMs (VLMs) within high-resolution, professional computer environments. It emphasizes realistic, full-screen GUI grounding—a task that requires models to accurately interpret instructions and identify specific graphical user interface elements within densely populated desktop applications spanning multiple industries, operating systems, and workflows (Li et al., 4 Apr 2025). ScreenSpot-Pro responds to the unique challenges of professional GUI scenarios, where the interface complexity, target size, and contextual distractions surpass those found in prior benchmarks.

1. Motivation and Scope

The development of ScreenSpot-Pro arises from the observed inadequacies of previous GUI grounding benchmarks, which typically involved cropped screenshots and simplified interactions. Professional use cases—such as those encountered in software development, CAD, scientific analysis, and creative production—feature ultra-high-resolution screens and highly diverse, intricate user interfaces. In these settings, the target UI elements (e.g., buttons, icons, menu entries) often constitute only 0.07% of the total image area, compared to 2.01% in mainstream benchmarks. This presents substantial challenges for current GUI grounding models, particularly in faithfully mapping natural language instructions to precise interface locations when significant visual clutter and nuanced contextual dependencies are present (Li et al., 4 Apr 2025).

2. Dataset Composition and Domain Coverage

ScreenSpot-Pro consists of 1,581 expert-annotated, authentic high-resolution screenshots, each paired with compositional natural language instructions. The dataset captures a representative cross-section of professional computing through coverage of 23 applications across five industrial domains:

• Development and Programming (e.g., Visual Studio Code, PyCharm)
• Creative Suite (e.g., Photoshop, Illustrator, Premiere)
• CAD and Engineering (e.g., AutoCAD, SolidWorks)
• Scientific/Analytical Tools (e.g., MATLAB, Origin)
• Office Suites (e.g., Word, Excel, PowerPoint)

These applications run across Windows, macOS, and Linux. The benchmark ensures both diversity and authenticity by sourcing from real-world usage and including variable operating system chrome, toolbars, palettes, and multitasking contexts. Approximately 62.6% of instructions target text-based UI elements, with the remainder focused on iconography, thereby covering the spectrum of interface semantics encountered in expert workflows (Li et al., 4 Apr 2025).

3. Evaluation Protocol and Baseline Performance

The primary task in ScreenSpot-Pro is language-conditioned GUI element localization: given a full-screen, high-resolution screenshot and a natural language instruction, models are expected to return the precise bounding box of the referenced GUI element. Evaluation is primarily based on accuracy, computed as the percentage of examples for which the predicted region matches the expert-annotated ground truth.

Initial assessments revealed that models with strong general performance on web and mobile interfaces perform poorly in the ScreenSpot-Pro scenario. For example, the OS-Atlas-7B model achieved only 18.9% accuracy, and other end-to-end methods often scored below 10% (Li et al., 4 Apr 2025). This reflects both the difficulty of the benchmark and the need for advanced spatial reasoning and contextual narrowing strategies capable of high-resolution, cluttered environments.

Table: Baseline Model Performance on ScreenSpot-Pro

Model	Parameter Count	Accuracy (%)
OS-Atlas-7B	7B	18.9
UI-TARS-2B	2B	27.7
Qwen-GUI-3B	3B	28.7
GUI-G²-7B	7B	47.5

Data as reported in (Li et al., 4 Apr 2025, Hsieh et al., 30 Jun 2025, Tang et al., 21 Jul 2025)

4. Methodological Innovations: ScreenSeekeR and Reward Modeling

ScreenSpot-Pro catalyzed methodological advances in GUI grounding:

4.1 ScreenSeekeR Visual Search

ScreenSeekeR is an agentic visual search method developed in conjunction with the benchmark. It leverages the planning and commonsense reasoning of a strong LLM planner (GPT-4o) to recursively identify and narrow down likely regions of interest before applying grounding predictions. Its operation involves:

• Position inference: Parsing instructions to suggest candidate regions based on expected element locations and neighboring UI patterns.
• Centrality-based scoring: Assigning higher probability to regions closer to the predicted center using a function

$$s = \exp\bigg(-\frac{(x'-0.5)^2 + (y'-0.5)^2}{2\sigma^2}\bigg)$$

with $x', y'$ denoting normalized coordinates and $\sigma$ set to 0.3, reinforcing visual attention heuristics.

• Recursive cropping: Iteratively zooming into the highest-scoring region until it is within the model’s feasible input size or a maximum recursion depth is reached.

ScreenSeekeR elevated OS-Atlas-7B’s accuracy from 18.9% to 48.1% on ScreenSpot-Pro without additional model training, confirming the benefit of hierarchical, planner-driven search in high-resolution contexts (Li et al., 4 Apr 2025).

4.2 Continuous Reward Modeling: GUI-G²

GUI-G² introduced a continuous, Gaussian-based reward framework for GUI grounding within reinforcement learning architectures. Rather than using a binary hit/miss signal, the method deploys:

• Gaussian point rewards: The modeled reward is

$$R_{point} = \exp\left( -\frac{1}{2} \left[ \frac{(c_x^p - c_x^{gt})^2}{(\sigma_x^{gt})^2} + \frac{(c_y^p - c_y^{gt})^2}{(\sigma_y^{gt})^2} \right] \right)$$

where $(c_x^p, c_y^p)$ and $(c_x^{gt}, c_y^{gt})$ are predicted and ground-truth centers; $(\sigma_x^{gt}, \sigma_y^{gt})$ adapt to element width/height.

• Gaussian coverage rewards: Evaluate the full overlap of probability distributions between prediction and ground truth, using measures such as the Bhattacharyya coefficient.
• Adaptive variance mechanism: Standard deviations are proportional to the UI element’s size ($\sigma_x = \alpha(x_2-x_1)$, $\sigma_y = \alpha(y_2-y_1)$ with $\alpha=0.5$).

This approach delivers dense, smooth gradients during training and mitigates performance cliffs inherent in discrete reward schemes, resulting in state-of-the-art accuracy (47.5%, a 24.7% gain over UI-TARS-72B) (Tang et al., 21 Jul 2025).

5. Data Diversity, Model Adaptation, and Scalability

Subsequent research demonstrates that the composition and quality of training data as well as training strategies have substantial impact on ScreenSpot-Pro outcomes:

• Rich Annotation Taxonomy: The transition from two UI element types (text, icon) in ScreenSpot-Pro to 32 in OSWorld-G and related datasets enables models to generalize compositional UI semantics rather than relying on repeated patterns (Xie et al., 19 May 2025).
• Synthetic and Real-World Data Mixing: Expanding grounding corpora (e.g., Jedi dataset with 4 million synthesized examples) and supplementing with real-world screenshots enhance trained models’ robustness and agentic performance on ScreenSpot-Pro (Xie et al., 19 May 2025).
• Fine-Tuning Paradigms: ZonUI-3B demonstrates that a lightweight (3B parameter) architecture can achieve 28.7% accuracy on ScreenSpot-Pro via two-phase fine-tuning: first, cross-platform generalization, then domain-specific resolution specialization. Balanced sampling and redundancy reduction further boost generalization without excessive scale-up of model size or dataset volume (Hsieh et al., 30 Jun 2025).

6. Significance, Applications, and Future Directions

ScreenSpot-Pro has established itself as the reference benchmark for evaluating GUI grounding under conditions mimicking real professional use. Its impact includes:

• Benchmarking agentic planners and grounding models: It exposes limitations of standard MLLMs and VLMs while revealing the value of agentic, search-based, or reward-incentivized grounding methods.
• Driving innovation in reward modeling and hierarchical search: The successes of ScreenSeekeR and GUI-G² inform broader research into spatial reasoning and active interface search.
• Enabling practical progress on agent frameworks: Improvements in grounding ability—driven by compositional training and more realistic benchmarks—translate directly into higher utility and reliability for autonomous computer-use agents in complex desktop environments.

Planned directions, prompted by the gaps highlighted in ScreenSpot-Pro performance profiles, include enhancing image partitioning strategies, developing more sophisticated context modeling for ambiguous UI element discrimination, and facilitating agent planning/execution pipelines that integrate expert grounding modules (Li et al., 4 Apr 2025). Cross-lingual generalization and continual expansion of interface taxonomies remain ongoing challenges.

7. Comparative Perspective and Benchmark Ecosystem

ScreenSpot-Pro stands as a bridge between simplified GUI benchmarks (e.g., early ScreenSpot, which used cropped, low- to mid-res screenshots) and new multi-tasking platforms such as OSWorld-G and Jedi that focus on fine-grained manipulation, refusal handling, and multi-perspective decomposition (Xie et al., 19 May 2025). The evolution of model training—from purely accuracy-driven to multi-objective and reward-optimized—mirrors parallel advances in visual SLAM benchmarking (e.g., SLAMBench2), which underscores the importance of unified, extensible, and application-affine evaluation standards (Bodin et al., 2018).

In summary, ScreenSpot-Pro Benchmark fulfills a critical role in quantifying and advancing GUI grounding research for professional high-resolution computing. By providing a challenging, diverse dataset, enabling fine-grained evaluation, and catalyzing methodological innovation, it sets the standard for the next generation of GUI agent development and deployment.