ScreenParse: GUI Parsing Datasets & Benchmarks

Updated 4 July 2026

ScreenParse is a framework of benchmarks and datasets for dense GUI parsing that recovers complete interface structures from screenshots.
It enables end-to-end evaluation by combining UI element detection, semantic labeling, and hierarchical structure recovery with dense annotations.
Practical insights include significant improvements in recall, semantic alignment, and inference speed compared to sparse grounding techniques.

Searching arXiv for the cited ScreenParse-related papers and closely related UI parsing benchmarks. ScreenParse is a name used in recent GUI-perception literature for datasets and benchmarks that move interface understanding beyond sparse single-target grounding toward dense, screen-level structural recovery. In one usage, introduced with SparkUI-Parser, ScreenParse is a benchmark for end-to-end GUI parsing across multilingual and multi-platform screenshots, measuring whether a model can locate every visible element and attach a semantic description (Jing et al., 5 Sep 2025). In a second usage, introduced in subsequent work on complete screen parsing supervision, ScreenParse is a large-scale web dataset with dense annotations of all visible UI elements, their 55-class types, text, and hierarchy, and it serves as the training substrate for ScreenVLM (Gurbuz et al., 15 Feb 2026). Both uses are closely connected to the broader screen parsing problem formulated earlier as predicting UI elements and their relationships from screenshots (Wu et al., 2021).

1. Conceptual background and historical placement

The immediate precursor to ScreenParse is the broader problem of screen parsing. In the 2021 formulation, screen parsing is defined as the task of ingesting a raw GUI screenshot and producing a structured, machine-readable model of the interface, specifically a UI hierarchy whose leaves correspond to visible UI elements and whose internal nodes represent semantically coherent groupings (Wu et al., 2021). A valid parse was described as complete, grounded, and abstractive: every visible UI element must appear exactly once as a leaf, each leaf must be tied to a bounding-box detection, and internal nodes need not correspond one-for-one with OS view objects.

That earlier work framed screen parsing as a structured-prediction problem rather than a flat detection problem. Its pipeline combined Faster-R-CNN detection, a top-down transition-based hierarchy parser using actions $\text{Emit}$ , $\text{Arc}$ , and $\text{Pop}$ , and a Deep Averaging Network for container labeling (Wu et al., 2021). The emphasis on hierarchy, grouping, and semantic container inference established an important distinction between merely finding UI elements and reconstructing an interface’s latent structure.

ScreenParse inherits this agenda but redirects it toward dense supervision and complete screen coverage. In both later formulations, the central concern is no longer just whether a model can ground a queried element, but whether it can recover the full visible state of the screen, including localization and semantics at screen scale. This suggests a shift from task-conditioned grounding toward interface-state reconstruction as a primary primitive for computer-use agents and GUI understanding systems.

2. ScreenParse as the SparkUI-Parser benchmark

In "SparkUI-Parser: Enhancing GUI Perception with Robust Grounding and Parsing," ScreenParse is introduced as a benchmark designed to assess a model’s end-to-end ability to locate every visible element on a GUI screenshot and attach a correct semantic description to each element (Jing et al., 5 Sep 2025). Its stated purpose is to go beyond “find this one button” grounding tasks and support full-screen, fine-grained parsing.

The benchmark is multilingual and multi-platform. It supports English and Chinese interfaces and covers mobile, desktop, and web screenshots. The annotated elements include free-form text blocks such as labels, headings, and paragraphs, and icons such as buttons, glyphs, logos, and interactive widgets. For each element, ScreenParse records bounding-box coordinates and a short, human-readable semantic label. It does not explicitly annotate parent-child widget hierarchies, but its dense element-level annotations are described as implicitly capturing common layout groupings such as toolbars, lists, and dialog panels (Jing et al., 5 Sep 2025).

Its construction combines existing benchmark material and new data collection. The source screens consist of all 823 screenshots from the open-source ScreenSpot benchmark plus approximately 400 additional real-world Chinese app screens, yielding approximately 1,200 unique screenshots. Pre-annotation used Grounding DINO for icon box proposals, Paddle OCR for text box proposals, non-maximum suppression to fuse overlapping boxes, and a pretrained MLLM to generate provisional icon and text labels. Human annotators then manually inspected and corrected every box and label. On average, each screen contains 36 elements, with 57.5% text and 42.5% icons (Jing et al., 5 Sep 2025).

The benchmark also organizes evaluation by language and interface density. The English subset contains 400 screenshots spanning mobile, desktop, and web; the Chinese subset contains 400 screenshots from popular finance, social, and utility apps. Layout complexity is grouped informally by element count: simple layouts contain at most 20 elements, medium layouts contain 20–50 elements, and complex layouts contain more than 50 elements. Representative examples include a “Sign In” screen, a chat conversation view with about 30 text bubbles and icons, and a mobile banking home page with more than 60 elements (Jing et al., 5 Sep 2025).

3. ScreenParse as a large-scale complete screen parsing dataset

In "Moving Beyond Sparse Grounding with Complete Screen Parsing Supervision," ScreenParse denotes a large-scale, web-centric dataset for “complete screen parsing,” defined as recovering all visible UI elements on a screen with their bounding boxes, semantic types, and text (Gurbuz et al., 15 Feb 2026). This dataset is much larger in scale than the benchmark introduced with SparkUI-Parser and is paired with a compact vision-LLM, ScreenVLM.

The dataset contains 771,458 rendered web-page screenshots and 21,094,243 total UI element annotations. Its train split has 693,975 images and 18,968,147 annotations; the validation split has 38,850 images and 1,062,552 annotations; the test split has 38,633 images and 1,063,544 annotations (Gurbuz et al., 15 Feb 2026). The taxonomy comprises 55 classes, ranging from structural and navigational elements such as Table, Navigation Bar, Side Bar, Toolbar, and Window, to common widgets such as Button, Checkbox, Radiobox, Select, Text Input, Search Bar, and Progress bar, and to content classes such as Text, Heading, Image, Chart, Logo, and Code snippet.

For each element $e_i = (b_i, c_i, t_i)$ , the annotation format stores pixel coordinates $b_i = (x_1, y_1, x_2, y_2)$ , a class label $c_i \in \{1..55\}$ , and an optional visible text string $t_i$ from the DOM or an OCR fallback. Unlike the SparkUI-Parser benchmark version, this ScreenParse preserves hierarchy explicitly: each element records parent and child indices, distinguishing container and leaf elements (Gurbuz et al., 15 Feb 2026).

The dataset is generated by an automated pipeline called Webshot. Webshot samples 1 million unique URLs from the public “45 Million Websites” dataset, curates a balanced subset covering e-commerce, news, social, and blogs, renders pages through Playwright in headless Chromium at a viewport of $1440 \times 900$ , disables animations, waits 800 ms post-load, and captures viewport-only screenshots. DOM-driven annotation extraction traverses the DOM, including iframes, and records raw bounding boxes, coarse HTML or ARIA types, DOM text content, and accessibility metadata for elements overlapping the viewport. Optional extraction of fine-grained text spans is supplemented with Tesseract OCR for crops missing DOM text (Gurbuz et al., 15 Feb 2026).

Cleaning and filtering procedures are substantial. Degenerate boxes, out-of-viewport boxes, tiny artifacts with area below $4 \text{ px}^2$ except critical widgets, and overly large wrappers exceeding 50% of the viewport are removed. Near-duplicate boxes with IoU greater than 0.95 are suppressed with a preference for interactive or semantic classes. Sample-level deduplication uses a Hamming radius of 8, and per-class duplicate suppression on final ground truth clusters boxes with IoU greater than 0.65, keeping either the largest container or the smallest atomic element. Label refinement uses Qwen-3-VL-8B-Instruct to reclassify each element into one of the 55 ScreenTag classes and predict an “interactable” flag. Quality filtering again uses Qwen-3-VL-8B-Instruct, which scores annotation quality along coverage, false positives, duplication, and localization, with pages discarded when overall quality is below 70; thresholds are calibrated with targeted human validation (Gurbuz et al., 15 Feb 2026).

4. Annotation formalisms, supervision signals, and representations

The two ScreenParse artifacts differ substantially in their representational commitments. The benchmark in SparkUI-Parser uses dense element-level annotations of boxes and short semantic labels but does not explicitly annotate parent-child hierarchies (Jing et al., 5 Sep 2025). As a consequence, its notion of structure is evaluated indirectly through high detection recall and precision together with semantic alignment. The large-scale ScreenParse dataset, by contrast, directly stores hierarchy and class labels, making it suitable for training models that decode full structural representations (Gurbuz et al., 15 Feb 2026).

SparkUI-Parser formalizes matching between ground-truth elements $y = \{(t_i, b_i)\}_{i=1..N}$ and predictions $\text{Arc}$ 0 through a modified Hungarian matching cost

$\text{Arc}$ 1

It solves a one-to-one assignment minimizing total cost and accepts a match only if $\text{Arc}$ 2, with $\text{Arc}$ 3 in practice. This same formalism is used for evaluation and training, and it is complemented by a rejection mechanism in which the model emits either a special token $\text{Arc}$ 4 to trigger coordinate regression or $\text{Arc}$ 5 to signal that no such element exists, thereby reducing hallucinated false positives (Jing et al., 5 Sep 2025).

The large-scale ScreenParse dataset uses a different representational strategy through ScreenTag markup. ScreenVLM serializes each screen as a depth-first sequence of XML-like tags,

$\text{Arc}$ 6

where the tag belongs to the 55-class vocabulary, coordinates are quantized to a $\text{Arc}$ 7– $\text{Arc}$ 8 grid, and siblings are ordered by top-left reading order. The tokenizer is extended with special tokens for opening and closing tags and for coordinate bins (Gurbuz et al., 15 Feb 2026). This representation is explicitly hierarchical and sequence-decodable, aligning the dataset with autoregressive VLM training.

These two formalisms reflect two distinct approaches to structural GUI perception. One centers on dense matching of elements and semantic descriptions without explicit tree labels; the other makes hierarchy first-class through markup serialization. A plausible implication is that the name ScreenParse now spans both a benchmarking paradigm for complete interface parsing and a fully supervised structural dataset for training compact parsers.

5. Evaluation protocols and reported results

The SparkUI-Parser benchmark defines two main tasks. Element Detection (Bounding-Box Parsing) asks a model to predict a set of boxes $\text{Arc}$ 9 that covers the ground-truth boxes $\text{Pop}$ 0, evaluated using element recall $\text{Pop}$ 1, element precision $\text{Pop}$ 2, and optionally $\text{Pop}$ 3. Semantic Labeling (Description Parsing) asks the model to output a short description $\text{Pop}$ 4 for each matched box, measured by semantic similarity $\text{Pop}$ 5, for example via cosine similarity between embeddings of predicted and reference labels. Because the benchmark does not explicitly annotate widget hierarchies, no dedicated parent-child metric is used (Jing et al., 5 Sep 2025).

The paper reports English and Chinese parsing test-set performance averaged over all scenarios. Off-the-shelf general MLLMs perform poorly on dense parsing, while SparkUI-Parser improves both detection and semantic alignment substantially.

Model	English $\text{Pop}$ 6	Chinese $\text{Pop}$ 7
GPT-4o	5.7 / 11.7 / 0.586	4.9 / 7.8 / 0.406
Claude Computer Use	17.1 / 35.3 / 0.758	19.3 / 31.0 / 0.511
Qwen2.5-VL-7B	18.8 / 36.6 / 0.596	25.0 / 52.6 / 0.854
InternVL2.5-8B	11.7 / 30.4 / 0.615	28.6 / 47.9 / 0.812
SparkUI-Parser-8B	77.2 / 77.9 / 0.918	87.1 / 89.5 / 0.946

The same results table reports per-element inference time on one A100 GPU: Qwen2.5-VL-7B at 0.534 s/element, InternVL2.5-8B at 0.558 s/element, and SparkUI-Parser-8B at 0.154 s/element. The accompanying summary states that off-the-shelf general MLLMs locate only a handful of elements with recall below 20%, whereas SparkUI-Parser achieves more than 75% recall and semantic similarity above 0.9 on both languages, at approximately $\text{Pop}$ 8 faster per-element inference than the next best open model (Jing et al., 5 Sep 2025).

The large-scale ScreenParse dataset introduces a different evaluation suite. PageIoU averages the intersection-over-union between the unions of predicted and ground-truth boxes over screens. Label PageIoU restricts agreement to pixels with matching labels. Recall@50 measures the fraction of ground-truth elements matched by a predicted box with IoU at least 0.5, with label-aware variants requiring class agreement. mAP@50 uses standard one-to-one greedy matching at IoU at least 0.5. For sparse annotations, PixCov measures the fraction of target pixels covered (Gurbuz et al., 15 Feb 2026).

On the ScreenParse test split, ScreenVLM reaches PageIoU 0.606, Label PageIoU 0.197, and mAP@50 0.303. By comparison, Qwen3-VL-8B reports PageIoU 0.294, InternVL3-2B reports PageIoU 0.111, Label PageIoU 0.030, and mAP@50 0.000, and Qwen3-VL-2B reports PageIoU 0.228, Label PageIoU 0.051, and mAP@50 0.023. Detector-based baselines fine-tuned with ScreenParse also perform strongly: YOLO + ScreenParse reaches PageIoU 0.533, Label PageIoU 0.133, and mAP@50 0.299, while RT-DETRv2 + ScreenParse reaches PageIoU 0.600, Label PageIoU 0.172, and mAP@50 0.362 (Gurbuz et al., 15 Feb 2026).

The paper also reports transfer to GroundCUA and ScreenSpot. On GroundCUA, ScreenVLM attains PageIoU 0.251 and Label PageIoU 0.043, while OmniParser V2 + ScreenParse and RT-DETRv2 + ScreenParse report 0.398/0.061 and 0.388/0.059 respectively. On ScreenSpot, ScreenVLM achieves Web Recall 0.557 and Web PixCov 0.746, PC Recall 0.222 and PC PixCov 0.839, and Mobile Recall 0.066 and Mobile PixCov 0.847; RT-DETRv2 + ScreenParse reports 0.768/0.857 on Web, 0.590/0.699 on PC, and 0.584/0.736 on Mobile (Gurbuz et al., 15 Feb 2026).

6. Models trained on ScreenParse and transfer effects

SparkUI-Parser and ScreenVLM exemplify two distinct model design responses to complete screen parsing supervision. SparkUI-Parser modifies a pre-trained MLLM with an additional token router and coordinate decoder so that coordinates are modeled continuously rather than through probability-based discrete autoregressive token generation. According to the paper, this is intended to mitigate limitations of discrete output characteristics and token-by-token generation, thereby improving localization precision and inference speed. Robustness is further enhanced by the rejection mechanism based on modified Hungarian matching, allowing the model to identify non-existent elements and reduce false positives (Jing et al., 5 Sep 2025).

ScreenVLM is a compact 316M-parameter VLM with a SigLIP-2 vision encoder, a Granite-165M autoregressive decoder, and projection layers between them. It is fine-tuned on ScreenParse for 287,500 steps on $\text{Pop}$ 9 NVIDIA H100 GPUs with effective batch size 64. Training uses a structure-aware weighted cross-entropy loss,

$e_i = (b_i, c_i, t_i)$ 0

with $e_i = (b_i, c_i, t_i)$ 1 for tag tokens, $e_i = (b_i, c_i, t_i)$ 2 for coordinate tokens, and $e_i = (b_i, c_i, t_i)$ 3 otherwise, where $e_i = (b_i, c_i, t_i)$ 4 and $e_i = (b_i, c_i, t_i)$ 5 in all experiments. Sequence length is capped at 8192 tokens, and the optimizer is AdamW (Gurbuz et al., 15 Feb 2026).

An ablation on ScreenVLM compares standard cross-entropy to the structure-aware loss. The structure-aware variant improves ScreenParse PageIoU from 0.592 to 0.606 and Label PageIoU from 0.192 to 0.197; on GroundCUA it improves PageIoU from 0.226 to 0.251 and Label PageIoU from 0.039 to 0.043; on ScreenSpot it raises recall from 0.541 to 0.557 on Web, from 0.129 to 0.222 on PC, and from 0.052 to 0.066 on Mobile (Gurbuz et al., 15 Feb 2026).

The same paper emphasizes transfer learning effects from dense screen supervision. Finetuning InternVL3-2B on ScreenParse raises its ScreenParse PageIoU from 0.111 to 0.509 and Label PageIoU from 0.030 to 0.174; on GroundCUA it improves from 0.025/0.006 to 0.203/0.036; on WebSpot it moves from Recall@50 0.002 and PixCov 0.057 to 0.172 and 0.592. Finetuning Qwen3-VL-2B on ScreenParse raises ScreenParse performance from 0.228/0.051 to 0.585/0.166, GroundCUA from 0.030/0.005 to 0.090/0.019, and WebSpot from 0.232/0.292 to 0.477/0.720 (Gurbuz et al., 15 Feb 2026). The paper interprets these gains as evidence that dense, complete screen-level supervision injects transferable structural priors for UI understanding.

7. Relation to hierarchy-centric screen parsing and downstream significance

The ScreenParse line of work should be distinguished from, yet also situated within, the earlier hierarchy-centric screen parsing literature. The 2021 screen parsing system explicitly targeted reconstruction of the UI hierarchy, using Hungarian matching on IoU to align detected and true boxes, edge-based $e_i = (b_i, c_i, t_i)$ 6, graph edit distance, and Container Match to assess the predicted tree (Wu et al., 2021). On AMP, the dynamic-oracle parser reported $e_i = (b_i, c_i, t_i)$ 7, $e_i = (b_i, c_i, t_i)$ 8, GED $e_i = (b_i, c_i, t_i)$ 9, and CM $b_i = (x_1, y_1, x_2, y_2)$ 0; on RICO, it reported $b_i = (x_1, y_1, x_2, y_2)$ 1, $b_i = (x_1, y_1, x_2, y_2)$ 2, GED $b_i = (x_1, y_1, x_2, y_2)$ 3, and CM $b_i = (x_1, y_1, x_2, y_2)$ 4 (Wu et al., 2021). The paper also states that the dynamic oracle yields up to +23% relative gain on $b_i = (x_1, y_1, x_2, y_2)$ 5.

That work highlights applications that depend on structured UI understanding rather than isolated detection: UI similarity search through screen embeddings, accessibility enhancement by grouping elements for swipe navigation, and code generation from screenshots into SwiftUI via traversal of the predicted hierarchy (Wu et al., 2021). ScreenParse does not replicate all of these downstream evaluations, but its emphasis on complete screen coverage aligns with the same general objective: constructing screen representations rich enough for computer-use agents, accessibility tooling, and interface reverse engineering.

A recurring misconception is that screen understanding is equivalent to target grounding. The ScreenParse papers argue against that equivalence in different ways. The SparkUI-Parser benchmark is explicitly motivated by the limitation of prior methods that “can only locate predefined sets of elements and are not capable of parsing the entire interface” (Jing et al., 5 Sep 2025). The large-scale ScreenParse paper similarly argues that sparse supervision limits coverage and generalization, and that practical deployment requires dense and efficient recovery of what is visible, where it is, and what text it contains (Gurbuz et al., 15 Feb 2026). Taken together, these works position complete screen parsing as a more stringent and more operationally relevant objective than sparse grounding alone.