GUI-ReRank: Two-Stage GUI Retrieval

• GUI-ReRank is a two-stage GUI retrieval framework that first employs embedding-based constrained search to narrow candidate screens based on positive and negative constraints.
• It integrates a zero-shot prompted multi-modal LLM for secondary reranking, providing nuanced semantic and visual judgments for precise GUI selection.
• The modular design supports customizable repository annotations, enabling its use in prototyping, requirements elicitation, design reuse, and retrieval-augmented generation.

GUI-ReRank is a two-stage GUI retrieval framework for natural-language search over GUI repositories. Its defining configuration combines rapid embedding-based constrained retrieval with multi-modal LLM reranking, alongside a fully customizable GUI repository annotation and embedding pipeline that allows arbitrary GUI collections to be made searchable. The framework is situated in GUI prototyping, requirements elicitation, design reuse, and retrieval-augmented generation workflows, where the objective is to retrieve relevant GUI screenshots from an existing repository rather than synthesize interfaces from scratch. In a broader methodological sense, GUI-ReRank also aligns with a wider class of GUI selection problems in which a system must narrow a candidate set and then apply stronger semantic or spatial judgment to choose the best match (Kolthoff et al., 5 Aug 2025).

1. Problem formulation and motivation

GUI-ReRank addresses natural-language-based GUI retrieval: given an NL requirement query, the system returns the most relevant GUI screenshots from a repository. The motivating use cases include requirements elicitation, design reuse, early testing, iterative evaluation, validation of design ideas, and RAG workflows for GUI generation. The underlying premise is that retrieving existing GUIs can be preferable when a domain-specific or stylistic design should be grounded in prior examples rather than generated de novo (Kolthoff et al., 5 Aug 2025).

The framework is positioned against two prior method families. Embedding-based retrieval methods are efficient and scalable, but they struggle with fine-grained constraints such as negation, combinations of attributes, and selectively emphasizing different aspects such as design versus functionality. They also face generalization difficulties across repositories and can yield suboptimal ranking quality when the query semantics exceed topical similarity. Tailored Learning-to-Rank models, including the BERT-LTR baselines used on Rico-derived benchmarks, can outperform simple retrieval models, but they are custom-built for specific repository structures and are less portable to arbitrary GUI collections. GUI-ReRank is designed to retain the efficiency of a first-stage retriever while introducing stronger semantic and visual judgment in a second-stage reranker (Kolthoff et al., 5 Aug 2025).

A common misconception is that GUI-ReRank is merely an embedding retriever with an LLM attached post hoc. In the reported formulation, the pipeline is more structured: repository annotations are dimension-specific, queries are decomposed into positive and negative constraints over those dimensions, and reranking can operate either on annotations or on raw GUI images. This makes the framework not only a reranker, but also a repository-construction and query-interpretation system (Kolthoff et al., 5 Aug 2025).

2. Architecture and repository construction

GUI-ReRank is organized into three components: automated GUI repository annotation and embedding, embedding-based constrained retrieval, and MLLM-based reranking. The design is explicitly modular. A repository is first annotated along configurable search dimensions, these annotations are embedded for retrieval, a user query is decomposed into constraints over the same dimensions, a first-stage ranker retrieves top-$k$ candidates, and a second-stage MLLM reranker reorders those candidates using either annotations or raw images (Kolthoff et al., 5 Aug 2025).

Component	Core operation	Function
Automated GUI repository annotation and embedding	Zero-shot prompted MLLM produces structured annotations and embeddings	Makes arbitrary repositories searchable
Embedding-based constrained retrieval	Decomposes queries into positive and negative constraints over search dimensions	Efficiently retrieves top-$k$ candidates
MLLM-based reranking	Scores candidates from annotations or images	Reorders candidates with richer semantic judgment

A distinctive feature is the customizable annotation pipeline for arbitrary GUI repositories. Given GUI images, GUI-ReRank uses a zero-shot prompted MLLM to produce structured annotations along a set of search dimensions. The default search dimensions are domain, functionality, design, GUI components, and displayed text. The framework also allows users to add or change dimensions by specifying a dimension name and a natural-language description of that dimension. This permits repository-specific indexing schemes without retraining a dataset-specific retriever (Kolthoff et al., 5 Aug 2025).

After annotation, each dimension-specific annotation is passed through an embedding model to generate an embedding for that dimension. The result is a repository in which each GUI has both structured textual descriptions and corresponding vector representations. The paper presents this not only as a retrieval mechanism but also as infrastructure for inspiration retrieval and custom RAG, because even proprietary or ad hoc GUI collections can be transformed into indexed repositories (Kolthoff et al., 5 Aug 2025).

3. Constraint-aware retrieval model

The first ranking stage is an embedding-based constrained retriever designed to handle richer requirements than ordinary semantic search. A natural-language requirement is decomposed by a zero-shot prompted LLM into positive and negative constraints for each search dimension. The canonical example is

$$SD_{Design}: pos^{+}(\text{"modern"}),\; neg^{-}(\text{"dark"}).$$

This supports queries such as “modern but not dark,” “shopping app with minimal design and no side menu,” and “finance dashboard with charts, but no map” (Kolthoff et al., 5 Aug 2025).

For each search dimension, the system computes cosine similarity between the user constraints and the GUI’s annotation embedding for that dimension. Positive similarity contributes positively to the score, while negative similarity is subtracted. The stage-1 score is expressed as

$$s_i = \frac{1}{Z}\sum_{d \in SD} w_d \bigl(\operatorname{sim}(q_d^{+}, a_{i,d}) - \operatorname{sim}(q_d^{-}, a_{i,d})\bigr),$$

where $a_{i,d}$ is GUI $i$’s annotation embedding for dimension $d$, $q_d^{+}$ and $q_d^{-}$ are the positive and negative constraint embeddings, $w_d$ is a user-defined weight for the dimension, and $k$0 is a normalization constant (Kolthoff et al., 5 Aug 2025).

This retrieval layer is zero-shot rather than a trainable listwise ranker. Its significance lies in explicit multidimensional query handling: it can model multiple dimensions, encode both inclusion and exclusion constraints, permit user prioritization through $k$1, and reduce the candidate set before expensive reranking. In operational terms, GUI-ReRank’s first stage acts as a low-cost semantic filter that is already more expressive than vanilla embedding retrieval (Kolthoff et al., 5 Aug 2025).

4. Multi-modal LLM reranking

The second stage reranks the top-$k$2 candidates produced by constrained retrieval. This stage can operate in two modes. In text-based reranking, the MLLM consumes the search-dimension annotations. In image-based reranking, it consumes the raw GUI screenshots. The text-based mode is faster and cheaper because it avoids full-image processing; the image-based mode is more detailed and generally more accurate, but it incurs greater token, runtime, and API cost (Kolthoff et al., 5 Aug 2025).

For each search dimension, the MLLM assigns a score in $k$3, where $k$4 denotes no match and $k$5 denotes perfect match. These per-dimension scores are weighted and summed into a final reranking score: $k$6 where $k$7 is the MLLM’s relevance score for GUI $k$8 on dimension $k$9. The reranker is also used in a zero-shot prompting regime, so the framework does not require task-specific fine-tuning for the benchmarked setting (Kolthoff et al., 5 Aug 2025).

The rationale for the second stage is that MLLM reranking can capture compositional semantics, visual layout details, interaction between design aspects, and subtle multidimensional relevance judgments that are not well represented by embedding similarity alone. This clarifies why GUI-ReRank is described as more than a retrieval heuristic: the reranking stage is the primary source of its reported gains over tailored LTR baselines (Kolthoff et al., 5 Aug 2025).

Another misconception is that image-based reranking should uniformly dominate text-based reranking in practical deployments. The results support a narrower claim: image-based reranking is generally strongest in effectiveness, but the framework explicitly treats text-based reranking as a meaningful efficiency-oriented operating point rather than a degraded fallback (Kolthoff et al., 5 Aug 2025).

5. Benchmarking, accuracy, and cost-efficiency trade-offs

GUI-ReRank is evaluated on the established NL-based GUI retrieval benchmark from Kolthoff et al. (2023), built on Rico. The benchmark contains 100 natural-language queries, each paired with 20 GUIs from Rico. The reported evaluation metrics are Average Precision, Mean Reciprocal Rank, Precision@k, HITS@k, and NDCG@k. The primary quantitative baselines are BERT-LTR-1, BERT-LTR-2, and BERT-LTR-3 (Kolthoff et al., 5 Aug 2025).

Reranker	AP	MRR
BERT-LTR baselines	0.486–0.501	0.618–0.631
GPT-4.1, text-based	0.813	0.927
GPT-4.1 Mini, text-based	0.797	0.923
GPT-4.1 Nano, text-based	0.662	0.842
GPT-4.1, image-based	0.840	0.928
GPT-4.1 Mini, image-based	0.811	0.930
GPT-4.1 Nano, image-based	0.582	0.790

The reported pattern is clear. Both text-based and image-based GPT reranking substantially outperform the BERT-LTR baselines across all metrics. Among the listed results, image-based GPT-4.1 attains the highest AP at 0.840, while image-based GPT-4.1 Mini attains an MRR of 0.930. The paper emphasizes that these gains are notable because the earlier LTR methods were already specialized for the benchmark, whereas GUI-ReRank uses zero-shot MLLM reranking (Kolthoff et al., 5 Aug 2025).

The paper also provides an explicit cost and efficiency analysis. Text-based reranking is substantially less expensive and faster than image-based reranking, especially as $$SD_{Design}: pos^{+}(\text{"modern"}),\; neg^{-}(\text{"dark"}).$$0 grows. A reported comparison states that, when moving from text-based GPT-4.1 to GPT-4.1 Mini, only 1.9% AP is lost while 79.9% of the cost is saved. The deployment interpretation given in the paper is correspondingly stratified: best quality is GPT-4.1 image-based, best efficiency is text-based Mini or Nano variants, and a balanced choice is text-based GPT-4.1 Mini (Kolthoff et al., 5 Aug 2025).

6. Relation to GUI grounding and action-selection reranking

Although GUI-ReRank is introduced as a GUI retrieval framework, related work shows that reranking-style reasoning recurs across GUI grounding and GUI agent inference. In AdaZoom-GUI, the paper explicitly states that GUI grounding is closely related to GUI-ReRank-style problems because both settings are fundamentally about selecting the right element among many visually similar candidates under ambiguous language. AdaZoom-GUI does not implement an explicit listwise or pairwise reranking head; instead, “ranking” is implicit in the grounding decision, where the model must identify the single best-matching element and output a click coordinate and bounding box. Its main contributions are an instruction refinement module and a conditional zoom-in strategy, with the latter triggered when the predicted box satisfies

$$SD_{Design}: pos^{+}(\text{"modern"}),\; neg^{-}(\text{"dark"}).$$1

where $$SD_{Design}: pos^{+}(\text{"modern"}),\; neg^{-}(\text{"dark"}).$$2 and $$SD_{Design}: pos^{+}(\text{"modern"}),\; neg^{-}(\text{"dark"}).$$3. The practical implication stated in the paper is that a refined instruction can serve as a more discriminative query for downstream element selection (Pei et al., 18 Mar 2026).

A second extension appears in test-time GUI grounding refinement. GUI-RC samples multiple predictions for the same screenshot and instruction, constructs a spatial voting grid,

$$SD_{Design}: pos^{+}(\text{"modern"}),\; neg^{-}(\text{"dark"}).$$4

and extracts a maximum-consensus region for the final output. GUI-RCPO turns the same consistency signal into a reward for test-time reinforcement learning, using the region-consistency reward

$$SD_{Design}: pos^{+}(\text{"modern"}),\; neg^{-}(\text{"dark"}).$$5

and a GRPO objective. In the terminology of the supplied description, this is a test-time reranking or refinement framework for GUI grounding, because multiple sampled outputs are rescored according to consensus rather than accepted independently (Du et al., 7 Aug 2025).

A third extension is best-of-$$SD_{Design}: pos^{+}(\text{"modern"}),\; neg^{-}(\text{"dark"}).$$6 action selection for GUI agents. GUI-PRA is a training-free judge agent that compresses history with a Dynamic Memory Mechanism, gathers grounded visual evidence through Adaptive UI Perception, and scores candidate thought-action pairs with a 0–10 scale. The final choice is

$$SD_{Design}: pos^{+}(\text{"modern"}),\; neg^{-}(\text{"dark"}).$$7

This is directly a reranking pipeline in the sense that candidate actions are generated first and then scored by an external judge agent before execution (Xiong et al., 27 Sep 2025).

Taken together, these systems indicate that GUI-ReRank is best understood not only as the title of a specific retrieval framework, but also as a broader pattern of GUI inference: candidate generation or coarse retrieval followed by stronger multimodal judgment for final selection. That interpretation is explicit for retrieval, implicit for grounding, and procedural for agent action selection (Kolthoff et al., 5 Aug 2025).

7. Limitations, scope conditions, and research trajectory

The limitations stated for GUI-ReRank are largely operational. LLM and MLLM cost and latency remain non-trivial, especially for image-based reranking. Stage-1 retrieval quality still matters because reranking only sees top-$$SD_{Design}: pos^{+}(\text{"modern"}),\; neg^{-}(\text{"dark"}).$$8 candidates. The main evaluation is conducted on a benchmark derived from Rico, so broader validation on more diverse repositories would strengthen generalizability claims. The system also depends on LLM-generated annotations, which may introduce noise or inconsistency. Although the framework is customizable, the paper does not deeply analyze domain-specific adaptations beyond the general pipeline (Kolthoff et al., 5 Aug 2025).

Related reranking-style work in GUI grounding sharpens these scope conditions. AdaZoom-GUI reports that unconditional zooming can hurt performance on simpler, lower-resolution benchmarks: on ScreenSpot-v2, the base model scores 0.943, unconditional zoom drops to 0.916, and conditional zoom improves slightly to 0.945. This shows that additional second-stage refinement is not uniformly beneficial; it must be triggered selectively (Pei et al., 18 Mar 2026). GUI-RC likewise states that consistency-based reranking depends on the model’s inherent competence and is more effective for bbox-style outputs than for point-style outputs expanded into boxes by approximation (Du et al., 7 Aug 2025).

The future directions named for GUI-ReRank are to improve efficiency further, extend evaluation to more repositories and interaction settings, and refine annotation or reranking prompts and models for better robustness (Kolthoff et al., 5 Aug 2025). A plausible implication of the broader literature is that GUI retrieval, GUI grounding, and GUI action selection are converging on a common decomposition: semantic disambiguation, candidate reduction, and final multimodal judgment. Within that decomposition, GUI-ReRank is the retrieval-specific formulation that most explicitly combines configurable repository construction, constraint-aware search, and zero-shot multimodal reranking.