GUISpector: GUI Perception & Verification

• GUISpector is a research domain encompassing distinct systems for pixel-based perceptual grouping and LLM-agent requirements verification in GUI analysis.
• The perceptual grouping variant infers widget clusters using unsupervised, Gestalt-inspired methods, enhancing structural understanding of GUI layouts.
• The requirements verification variant employs multi-modal LLM agents to interact with GUI prototypes and assess natural-language requirements with structured feedback.

Searching arXiv for GUISpector and closely related GUI-agent/GUI-grounding papers to ground the article. Tool call: arxiv_search({"query":"GUISpector GUI requirements verification natural language requirements GUI prototypes MLLM agent", "max_results": 10, "sort_by": "relevance"}) GUISpector is a name used in arXiv literature for distinct systems concerned with the inspection and interpretation of graphical user interfaces. One use denotes a pixel-only, unsupervised method for inferring perceptual groups of GUI widgets from GUI images, motivated by Gestalt principles and aimed at structural GUI understanding (Xie et al., 2022). A later use denotes an MLLM-agent framework for automated verification of natural-language requirements in GUI prototypes, aimed at requirements engineering, criterion-level assessment, and feedback-driven refinement in LLM-assisted development workflows (Kolthoff et al., 6 Oct 2025). Taken together, these systems place the term at the intersection of GUI perception, interactive verification, and software-engineering support.

1. Scope and disambiguation

The available arXiv record uses the name GUISpector for at least two technically different systems rather than for a single continuous framework.

GUISpector variant	Core task	Technical basis
Perceptual-grouping GUISpector	Infer perceptual groups of widgets from GUI images	Pixel-only, unsupervised, Gestalt-inspired pipeline
Requirements-verification GUISpector	Verify whether GUI prototypes satisfy NL requirements	MLLM GUI agent with trajectory execution and criterion-level feedback

The first system is centered on the proposition that a GUI is not merely a flat collection of widgets, but a composition of higher-order perceptual units such as tabs, menus, cards, and lists. Its objective is to infer those units from pixels alone, without GUI source code, runtime hierarchy, or training data (Xie et al., 2022). The second system treats a GUI as an executable artifact to be inspected against requirements, acceptance criteria, and test data by a computer-use agent that interacts with the interface and returns structured verdicts and evidence (Kolthoff et al., 6 Oct 2025).

The available descriptions do not specify a direct lineage between these two systems. A practical consequence is that citations to “GUISpector” benefit from explicit disambiguation by task definition and arXiv identifier.

2. Perceptual grouping and structural GUI understanding

The earlier GUISpector is a pixel-only, unsupervised GUI perceptual grouping system built around the claim that human perception organizes GUI widgets into groups by visual cues rather than by implementation structure. Its method has two major phases: GUI widget detection and perceptual grouping using Gestalt principles. The grouping stage is organized around connectedness, similarity, proximity, and continuity (Xie et al., 2022).

Widget detection enhances UIED. The system replaces the EAST text detector with Google OCR, retains UIED’s image-processing-based non-text detection, and modifies merging through container recognition. It deliberately simplifies widget classification to only text and non-text, on the premise that perceptual grouping depends more on geometry and visual repetition than on fine semantic widget classes. Container recognition operationalizes connectedness by identifying a widget as a container when its boundary is rectangular, with 4 straight lines perpendicular to each other, and when the boundary behaves as a wireframe border rather than as content.

Similarity-based grouping is implemented differently for text and non-text widgets. Non-text widgets are clustered using Center_X, Center_Y, and area; text widgets are clustered using (Top, Left). The clustering algorithm is DBSCAN, and only clusters with at least two widgets are retained. When a widget belongs to multiple candidate clusters, conflict resolution uses the average widget area of the candidate groups and spacing consistency, preferring the group with the largest widget-area similarity and smallest spacing difference.

Proximity then combines nearby groups into larger blocks. If two groups are next to each other, have no other group between them, have the same number of widgets, and share the same orientation, they are merged. The system first combines groups with the same type of widgets and then groups with different types. It also allows merging when the difference in widget count is less than 4, a threshold stated to be empirically determined from the ground-truth dataset. Continuity is used for detection-error correction: when aligned subgroups reveal missing elements, the system crops the inconsistent region and reruns detection with relaxed parameters, specifically double the minimum area threshold for valid widgets; when widget types disagree across aligned positions, reclassification follows a majority-win strategy.

The evaluation formalizes grouping as a hierarchy string generated by depth-first traversal with only Text and Non-text symbols and bracket markers. Matching between predicted and ground-truth blocks uses Levenshtein edit distance, with thresholds from 0 to 4, while widget detection uses IoU > 0.9 as the correctness criterion (Xie et al., 2022).

3. Automated verification of natural-language requirements

The later GUISpector is a framework for automated verification of natural-language requirements in GUI prototypes using a multi-modal LLM optimized for computer use. Its architecture is described as having three main components: Human-in-the-verification-loop, MLLM GUI agent verification loop, and Agentic implementation-verification loop (Kolthoff et al., 6 Oct 2025).

In the human-facing entry point, a user provides the target application, natural-language requirements, and relevant test input data. A zero-shot prompted LLM parses the requirements into a structured representation, extracting the requirement description and individual acceptance criteria, and it can infer missing details from context if needed. The integrated interface supports browsing requirements, inspecting whether each requirement is fulfilled, reviewing verification trajectories, examining detailed MLLM reasoning, checking whether each acceptance criterion was met or violated, and inspecting the evidence collected during verification.

The core verification loop adapts a pretrained MLLM so that it can autonomously verify whether a requirement is implemented. The agent is instructed by a zero-shot prompt to begin from a given URL, interact with the GUI to gather evidence, use minimal actions, use supplied test input data, evaluate each acceptance criterion individually, justify decisions with concrete evidence, and output a structured JSON summary. At each step it receives the requirement text, the acceptance criteria, the test data, and a screenshot of the current GUI state. It predicts actions such as click, double click, scroll, and type, with parameters including x/y coordinates. The resulting verification episode is represented as

$$(\mathrm{GUI}_1, r_1, a_1, \ldots, \mathrm{GUI}_n, r_n, a_n).$$

The final JSON contains a requirement-level verdict, explanations, and evidence for each acceptance criterion. The framework also cleans GUI environments before and after each run so that prior runs do not contaminate later verification.

The third component turns verification into a closed development loop. GUISpector provides a Model Context Protocol (MCP) server, allowing external programming agents to retrieve requirements and acceptance criteria, inspect verification states, select unverified or violated requirements, invoke verification runs, receive actionable explanations, and modify code or prototype behavior accordingly. The implementation is described as a Django web application with HTML/CSS/JavaScript frontend, MySQL storage, Celery for background tasks and parallel verification, Redis for coordination, a Redis-based display resource manager allocating Xfce desktop environments on Xvfb virtual displays, and xdotool for simulating user input. The current MLLM verification setup uses OpenAI CUA, while other supported LLMs include OpenAI GPT, Google Gemini, and Anthropic Claude Sonnet (Kolthoff et al., 6 Oct 2025).

4. Datasets, metrics, and empirical performance

The two GUISpector systems are evaluated on markedly different benchmarks because they solve different problems.

For the perceptual-grouping system, the dataset consists of 1,091 GUI screenshots from 772 Android apps and 20 UI prototypes collected from Figma. On widget detection for the Rico Android GUI dataset, Enhanced UIED achieves Precision 0.580, Recall 0.680, and F1 0.626, compared with Precision 0.490, Recall 0.557, and F1 0.524 for Original UIED. On Android GUI grouping at edit-distance threshold ≤ 1, metadata-based grouping yields Precision 0.607, Recall 0.754, and F1 0.672 for GUISpector versus 0.131, 0.116, and 0.123 for Screen Recognition. Detection-based grouping yields Precision 0.546, Recall 0.650, and F1 0.593 for GUISpector versus 0.103, 0.083, and 0.092 for Screen Recognition. On the 20 Figma prototypes, the reported scores are Precision 0.750, Recall 0.818, and F1 0.783 (Xie et al., 2022).

For the requirements-verification system, the paper introduces a GUI-Requirements Evaluation Dataset constructed with Codex. It contains 30 functional requirements with 3 acceptance criteria each for each of 5 domain-diverse applications—Park-and-Pay, Budget Tracker, Recipe Generator, Fitness Quests, and Cleaning Booking—for a total of 150 requirements. The target distribution per app is 18 met requirements, 6 unmet requirements, and 6 partially met requirements. Two authors independently labeled 450 acceptance criteria as met or unmet. Agreement statistics are high: Krippendorff’s alpha = 0.929 with ordinal, 95% CI [0.880, 0.966], and 8 disagreements out of 150 between the LLM and humans at the requirement level; at the acceptance-criteria level, Cohen’s kappa = 0.921, pooled, nominal, with accuracy = 96.7% and 435 out of 450 ACs, with kappa ranging from 0.841 to 0.972 across apps (Kolthoff et al., 6 Oct 2025).

Average requirement-level performance is reported as follows: for Met, P = 0.915, R = 0.906, F1 = 0.905; for Unmet, P = 0.917, R = 0.852, F1 = 0.878; for Partially met, P = 0.625, R = 0.666, F1 = 0.631. At the acceptance-criteria level, the averages are P = 0.933, R = 0.950, F1 = 0.940 for Met, and P = 0.899, R = 0.849, F1 = 0.870 for Unmet. Execution statistics indicate an overall average of 24.4 steps with SD 5.8, 317.6 s with SD 68.8 per requirement, 221.7k input tokens, 2.269k output tokens, and average cost $0.665** with **SD$0.163 (Kolthoff et al., 6 Oct 2025).

5. Position within GUI-agent and GUI-analysis research

GUISpector occupies two distinct positions relative to adjacent GUI research. The perceptual-grouping system addresses structural organization in GUI images, whereas several later systems concentrate on GUI grounding or GUI interaction. SpiritSight is an end-to-end, vision-based GUI agent built on InternVL2, with GUI-Lasagne containing 5,730,496 samples, and it emphasizes grounding accuracy through Universal Block Parsing (UBP) rather than perceptual grouping (Huang et al., 5 Mar 2025). GMS is a training-free, modular, coarse-to-fine framework that separates a generalist Scanner from a specialist Locator, improving ScreenSpot-Pro accuracy from 2.0% or 3.7% as standalone components to 35.7% in the integrated framework (Li et al., 29 Sep 2025). DRS-GUI likewise addresses grounding rather than grouping or requirements verification, using Focus, Shift, and Scatter actions with MCTS, and reports about a 14-point absolute gain on ScreenSpot-Pro for both Qwen2.5-VL-7B and UGround-V1-7B (Liu et al., 15 May 2026).

The later GUISpector is closer to agentic GUI-use systems, but its objective remains verification rather than assistance or generic navigation. DOMSteer modifies the live web interface through reversible DOM interventions—insert, mutate, and recompose—to provide in-situ assistance, whereas GUISpector evaluates whether the interface satisfies requirement and acceptance-criteria statements (Hao et al., 16 Apr 2026). Explorer is adjacent in a different way: it focuses on app-specific collection of interactable elements, Screen Similarity, and Action Matching for known target applications, rather than on requirement satisfaction (Chaimalas et al., 12 Apr 2025).

This suggests that “GUI inspection” now spans at least three technical strata: structural perception of grouped widgets, coordinate-level grounding and navigation, and requirements-driven interactive verification. In that taxonomy, the two GUISpector systems anchor the first and third strata.

6. Limitations, ambiguities, and open directions

The perceptual-grouping GUISpector inherits limitations typical of image-based structural analysis. Reported failure causes include over-segmentation, under-segmentation, occluded widgets still present in metadata, and alternative but still plausible grouping partitions. The method assumes that widgets are visually detectable, that containers are often rectangular wireframes, and that text and non-text widgets should be clustered differently. The paper identifies future work on better handling of widget occlusion and modal windows, and on moving toward semantic grouping rather than purely visual grouping (Xie et al., 2022).

The requirements-verification GUISpector exhibits a different set of constraints. The weakest class is partially met, which the authors attribute to boundary ambiguity and the semantic fuzziness of partial satisfaction. Verification is also resource-intensive, with substantial per-run time and token consumption. The benchmark itself is constructed from synthetic, Codex-generated requirements and HTML apps, which supports controlled evaluation but leaves broader real-world validation open. The framework is designed to support other computer-use agents, but the reported verification setup currently uses a specific MLLM configuration (Kolthoff et al., 6 Oct 2025).

A persistent source of ambiguity is nominal rather than technical: the same name refers to two systems with different tasks, inputs, and evaluation protocols. A plausible implication is that future scholarship will need to disambiguate GUISpector not only by citation, but also by whether the subject is perceptual grouping or requirements verification. Another plausible implication is methodological: the structural priors emphasized by the 2022 system and the trajectory-based verification of the 2025 system are complementary, even though the available descriptions do not report an explicit integration.