SparkUI-Parser: Multimodal GUI Parsing

Updated 4 July 2026

SparkUI-Parser is a multimodal GUI parsing framework that unifies robust grounding and fine-grained parsing by extracting interface elements with semantic labels and continuous bounding boxes.
It features a route-then-predict architecture that decouples language processing from direct bounding box regression, resulting in improved localization precision and reduced inference time.
The system employs a modified Hungarian matching and a dedicated vision adapter to stabilize multi-target outputs, achieving significant performance gains on benchmarks like ScreenSpot and ScreenParse.

Searching arXiv for the cited papers and closely related work to ground the article. SparkUI-Parser is an end-to-end multimodal LLM framework for GUI perception that unifies robust grounding and fine-grained parsing. In the grounding setting, it takes a GUI image and an instruction such as “click the settings icon” and predicts the bounding-box coordinates of the referenced elements, possibly multiple at once. In the parsing setting, it takes a GUI image and extracts a comprehensive set of interface elements together with categories, semantic labels, and bounding boxes for the whole interface. The system was introduced to address two limitations in prior GUI-oriented MLLMs: discrete coordinate generation through text autoregression, which lowers localization precision and slows inference, and restricted coverage of only predefined element sets rather than the full interface (Jing et al., 5 Sep 2025).

1. Problem setting and motivation

SparkUI-Parser targets two central GUI perception tasks. The first is grounding: given a screenshot and a natural-language instruction, the model predicts one or more target bounding boxes. The second is parsing: given a screenshot, the model outputs a set or list of elements, where each element is represented as a pair $(t_i, b_i)$ consisting of a semantic feature or text label $t_i$ and a bounding box $b_i$ . ScreenParse annotations associated with the work include element types such as text and icon, semantic labels, and bounding boxes for all elements in the interface (Jing et al., 5 Sep 2025).

The motivation is explicitly tied to the mismatch between continuous image geometry and tokenized language generation. Prior GUI-oriented MLLMs predominantly model coordinates as discrete, autoregressively generated text tokens, for example by emitting sequences corresponding to $x_1, y_1, x_2, y_2$ . According to the paper, this induces quantization and tokenization artifacts and creates a speed bottleneck because coordinates for multiple elements require token-by-token generation. SparkUI-Parser replaces that regime with direct continuous regression, which the authors describe as a route-then-predict design. Across benchmarks, the paper reports grounding precision improvements of about $3\%$ on average, with speedups of approximately $5\times$ for grounding and $4\times$ for parsing compared to discrete autoregressive baselines (Jing et al., 5 Sep 2025).

A common misconception is that grounding alone is sufficient for GUI understanding. SparkUI-Parser is explicitly designed around the broader claim that interface perception should include both target localization and full-interface structural extraction. Another misconception is that the model outputs a hierarchical tree of the UI. The paper does not specify hierarchical tree structures or spatial graphs; hierarchy relations are not described. Its structured output is a set or list of elements with semantics and coordinates rather than an explicit rooted containment hierarchy (Jing et al., 5 Sep 2025).

2. Route-then-predict architecture

The system architecture consists of a fine-tuned MLLM backbone, a token router, a vision adapter, a coordinate decoder, and a training-time element matcher. The base model is InternVL2.5-8B. Given a GUI image $I$ and a user instruction $T_{\text{instruction}}$ , the MLLM produces last-layer token embeddings $f_{\text{token}}$ , while its vision encoder $t_i$ 0 generates visual features from the image. The token router reads MLLM token logits and classifies output tokens into three categories: ordinary text tokens, special visual grounding tokens denoted [VG], and rejection tokens denoted [REJ] (Jing et al., 5 Sep 2025).

Component	Function	Reported details
Token Router	Separates text, `[VG]`, and `[REJ]` tokens	Based on MLLM token logits
Vision Adapter $t_i$ 1	Enhances localization features	MLP channels `[4096, 2048, 1024, 256]`
Coordinate Decoder $t_i$ 2	Regresses continuous bounding boxes	Transformer with cross-attention, hidden dimension `256`
Element Matcher	Aligns predictions and ground truth during training	Modified Hungarian matching with rejection

This routing mechanism decouples language generation from localization. Text tokens continue through the MLLM’s standard language head, [REJ] tokens are discarded, and each [VG] token triggers coordinate regression. The coordinate decoder consumes the last-layer embedding associated with each [VG] token, written as $t_i$ 3, together with the adapted visual features $t_i$ 4, and outputs a continuous-valued bounding box:

$t_i$ 5

The paper emphasizes that each target element requires exactly one [VG] token, rather than a multi-token numeric sequence. This compresses output length and reduces latency in multi-target grounding and parsing. The design therefore separates semantics, handled by the language head, from localization, handled by a lightweight decoder. A plausible implication is that this division of labor allows the MLLM to remain language-aligned while avoiding the inefficiency of expressing geometry through a discrete vocabulary (Jing et al., 5 Sep 2025).

3. Continuous coordinate modeling, matching, and training

SparkUI-Parser models coordinates as continuous values regressed by the coordinate decoder rather than as discrete text tokens. Let the ground-truth elements be $t_i$ 6 and the predictions be $t_i$ 7. Training uses a modified Hungarian matching procedure to obtain optimal bipartite assignments using a joint semantic–spatial criterion. The matching score is defined as

$t_i$ 8

with $t_i$ 9 and $b_i$ 0 during matching, and a threshold $b_i$ 1 experimentally (Jing et al., 5 Sep 2025).

The paper’s notation calls this quantity a “cost” but uses it with a thresholded assignment rule:

$b_i$ 2

After matching, the per-element loss is

$b_i$ 3

with $b_i$ 4, $b_i$ 5, and $b_i$ 6, and the total loss over $b_i$ 7 ground-truth elements is

$b_i$ 8

Robustness is implemented at two levels. At token level, [REJ] tokens allow the model to indicate that a requested element does not exist, so no coordinate decoding occurs. At matching level, the thresholded modified Hungarian procedure creates a no-object assignment $b_i$ 9 when suitable matches are absent, reducing false positives. The paper states that unmatched elements are not penalized for localization or semantics, which is intended to avoid order and randomness effects in multi-target outputs (Jing et al., 5 Sep 2025).

The training configuration is also specified in detail. The model fine-tunes InternVL2.5-8B with LoRA applied to both vision and language parts, using rank $x_1, y_1, x_2, y_2$ 0 and $x_1, y_1, x_2, y_2$ 1, while keeping the vision encoder unfrozen. The learning-rate schedule is CosineLRScheduler with learning rate $x_1, y_1, x_2, y_2$ 2 and warmup ratio $x_1, y_1, x_2, y_2$ 3. Training uses $x_1, y_1, x_2, y_2$ 4 NVIDIA A100 80GB GPUs and 515k total samples across mobile, desktop, and web interfaces: 400k English grounding samples and 5k English parsing samples, plus 100k Chinese grounding samples and 10k Chinese parsing samples. Training instances may include multiple objectives in a multi-target instruction paradigm. For parsing data construction, the pipeline uses Grounding DINO for icons, PaddleOCR for text, non-maximum suppression, an MLLM to supplement missing icon semantics, and manual inspection and correction (Jing et al., 5 Sep 2025).

4. Benchmarks, ablations, and empirical results

The paper evaluates SparkUI-Parser on ScreenSpot, ScreenSpot-v2, CAGUI-Grounding, and ScreenParse. ScreenParse, as defined in this work, is a benchmark for full-interface parsing with approximately 800 images total, 400 per language for English and Chinese, with an average of 36 elements per interface and an element-type distribution of 57.5% texts and 42.5% icons. Parsing metrics include element recall, element precision, and semantic similarity; the paper gives the standard formulas

$x_1, y_1, x_2, y_2$ 5

while noting that the exact computation formula for semantic similarity is not specified beyond reporting it as a scalar in $x_1, y_1, x_2, y_2$ 6 (Jing et al., 5 Sep 2025).

Benchmark	SparkUI-Parser-8B	Best prior reported
ScreenSpot	Avg. `88.0`, Time `0.173 s`	Qwen2.5-VL-7B Avg. `84.9`, Time `0.763 s`
ScreenSpot-v2	Avg. `89.5`, Time `0.168 s`	Aguvis-7B Avg. `87.3`, Time `0.779 s`
CAGUI-Grounding	Func2Box `80.2`, Text2Box `82.9`, Avg. `81.6`	AgentCPM-GUI-8B Avg. `77.8`
ScreenParse English	Recall `77.2`, Precision `77.9`, SemanticSim `0.918`, Time `0.154 s`	Qwen2.5-VL-72B Recall `22.9`, Precision `43.7`; Claude Computer Use SemanticSim `0.758`
ScreenParse Chinese	Recall `87.1`, Precision `89.5`, SemanticSim `0.946`, Time `0.154 s`	Qwen2.5-VL-72B Recall `42.4`, Precision `70.5`, SemanticSim `0.867`

On ScreenSpot, SparkUI-Parser-8B reaches an average score of 88.0 with 0.173 seconds per case, compared with 84.9 and 0.763 seconds for the best prior open model, Qwen2.5-VL-7B, corresponding to a +3.1 average-point improvement and approximately $x_1, y_1, x_2, y_2$ 7 lower latency. On ScreenSpot-v2, it reaches 89.5 at 0.168 seconds, compared with 87.3 at 0.779 seconds for Aguvis-7B, corresponding to +2.2 points and approximately $x_1, y_1, x_2, y_2$ 8 lower latency. On CAGUI-Grounding, it reports Func2Box 80.2, Text2Box 82.9, and average 81.6, compared with 77.8 average for AgentCPM-GUI-8B. On the ScreenParse benchmark, English performance is Recall 77.2, Precision 77.9, SemanticSim 0.918, and Time 0.154 seconds, while Chinese performance is Recall 87.1, Precision 89.5, SemanticSim 0.946, and Time 0.154 seconds (Jing et al., 5 Sep 2025).

The ablations are important because they isolate the role of each architectural component. For ScreenSpot-v2 Text/Icon average and ScreenParse averages, the full model obtains Text 95.4, Icon 83.7, Recall 82.2, Precision 83.7, and SemanticSim 0.932. Removing the coordinate decoder reduces these to Text 91.2, Icon 80.2, Recall 77.6, Precision 78.2, and SemanticSim 0.930. Removing the vision adapter causes a collapse to Text 22.1, Icon 17.8, Recall 5.4, Precision 4.6, and SemanticSim 0.915. Removing the element matcher gives Text 94.7, Icon 83.1, Recall 79.5, Precision 80.4, and SemanticSim 0.814. Removing the parsing dataset yields Text 95.0, Icon 83.2, Recall 20.2, Precision 39.2, and SemanticSim 0.751. These numbers support the paper’s interpretation that continuous coordinate regression improves localization, the vision adapter is essential for visual grounding, the matcher stabilizes multi-target training and semantic alignment, and dedicated parsing supervision is crucial for whole-interface perception (Jing et al., 5 Sep 2025).

5. Place within the screen-parsing literature

SparkUI-Parser belongs to a broader line of work that treats GUI understanding as more than isolated point grounding. Earlier screen-parsing research defined the task as predicting a structured user interface model from raw pixels, including a complete set of visible UI elements and a hierarchical relationship represented as a directed rooted tree. One influential implementation combined Faster R-CNN element detection, a transition-based hierarchy parser, and a group labeler, and reported that dynamic-oracle training improved performance by up to 23% over a static oracle depending on dataset and metric (Wu et al., 2021).

Against that background, SparkUI-Parser marks a shift in modeling strategy. Rather than building an explicit rooted tree with latent container nodes, it represents the interface as a set or list of elements with semantics and bounding boxes. This is not a simplification in the sense of task scope: the model still aims at complete interface coverage. Instead, it changes the structural representation from explicit hierarchy prediction to joint semantic and spatial extraction. This suggests a different notion of “screen parsing,” centered on dense element-level perception rather than on reconstructing an abstractive UI tree.

The relation becomes more complex because the name ScreenParse is also used in later work for a distinct large-scale dataset and benchmark. A 2026 paper introduced a ScreenParse dataset with 771,458 rendered web screenshots, 21,094,243 annotated UI elements, and 55 classes, and used it to train ScreenVLM, a compact 316M-parameter model for dense screen parsing with a ScreenTag markup representation and a structure-aware loss. That work showed that complete screen parsing supervision can improve both dense parsing and grounding transfer, for example reporting ScreenVLM PageIoU 0.606 on its ScreenParse test split and showing substantial gains when fine-tuning foundation VLMs on the dataset (Gurbuz et al., 15 Feb 2026).

The coexistence of these resources under the same name can cause confusion. In SparkUI-Parser, ScreenParse denotes an approximately 800-image bilingual benchmark for evaluating full-interface parsing. In the later dense-screen-supervision work, ScreenParse denotes a 771K-image web dataset with 55-class labels and preserved parent-child hierarchy. The shared terminology nonetheless reflects a common research direction: moving beyond sparse grounding toward richer representations of visible screen state (Jing et al., 5 Sep 2025).

6. Practical use, resource profile, and limitations

In practical use, SparkUI-Parser accepts a screenshot $x_1, y_1, x_2, y_2$ 9 and either a natural-language instruction $3\%$ 0 for grounding or just the image for parsing. The grounding output is a text answer plus one or more bounding boxes emitted through [VG] tokens. The parsing output is a list of elements with type, semantic label, and bounding box; [REJ] tokens may indicate absent elements. The paper provides a pseudocode-style inference description in which the MLLM decodes text and special tokens autoregressively, the vision adapter processes visual features, and the coordinate decoder returns one continuous bounding box per [VG] token (Jing et al., 5 Sep 2025).

The reported resource profile is comparatively compact on the localization side. Each element requires one [VG] token generation rather than several numeric tokens, and the coordinate decoder is a small transformer with hidden dimension 256 and cross-attention over compact visual features. The paper states that the decoder’s cross-attention typically scales with $3\%$ 1, where $3\%$ 2 is small, often 1 per element, and $3\%$ 3. Empirical latency is reported as 0.173 seconds on ScreenSpot, 0.168 seconds on ScreenSpot-v2, and 0.154 seconds on ScreenParse per case, measured on NVIDIA A100 80GB. Tokens per second, FPS, exact layer counts, memory usage, and throughput are not specified (Jing et al., 5 Sep 2025).

The paper does not explicitly list limitations or failure cases. However, the ablations show that the model depends strongly on high-quality visual features: removing the vision adapter severely degrades performance. The work also notes that exact rejection criteria during evaluation are not specified. Performance under extreme UI clutter, tiny elements, or severe domain shifts is not discussed. These omissions matter because GUI perception systems often encounter dense layouts, small icons, multilingual text, and application families not represented in training. A plausible implication is that deployment quality will depend on how closely the target GUI distribution matches the mobile, desktop, and web data used for training (Jing et al., 5 Sep 2025).

Reproducibility is partially supported. The paper provides a project repository, reports the hardware and hyperparameters used for training, and specifies the principal architectural modules. At the same time, license terms, pretrained checkpoints, and full configuration files are not specified in the paper text. For research use, SparkUI-Parser is therefore best understood as a concrete example of a route-then-predict GUI perception architecture: it decouples language from localization, introduces token-level and matching-level rejection, and extends evaluation from sparse grounding to full-interface parsing with semantics and coordinates (Jing et al., 5 Sep 2025).