Phi-Ground: Precise GUI Grounding Model

• Phi-Ground is a specialized GUI grounding model that converts textual instructions and screenshots into exact screen coordinates.
• It employs a two-stage design where a large multimodal LLM expands instructions into detailed references before a compact model localizes coordinates.
• Empirical evaluations show state-of-the-art performance on multiple benchmarks, with improvements driven by data augmentation and image-token optimization.

Searching arXiv for the Phi-Ground paper and closely related GUI grounding literature. Phi-Ground is a family of specialized GUI grounding models for Computer Use Agents (CUAs), designed to map a textual reference expression and a screenshot to precise screen coordinates for actions such as clicking and typing (Zhang et al., 31 Jul 2025). In the formulation reported for Phi-Ground, GUI grounding is separated from temporal planning: a large multimodal LLM first expands a short instruction into a detailed “Long RE,” and a compact grounding model then performs localization on the screen. The reported system is positioned within a two-stage agent setting in which the grounding component is optimized for perception and coordinate prediction rather than high-level task reasoning (Zhang et al., 31 Jul 2025).

1. Concept and scope

Within a modern CUA pipeline, Phi-Ground treats GUI grounding as the stage that determines where on the screen to act, in contrast to temporal planning, which determines what operation to perform (Zhang et al., 31 Jul 2025). The underlying decomposition is explicit: GUI grounding consists of spatial planning, which identifies the visual element to manipulate, and localization, which predicts the precise screen coordinates for the click (Zhang et al., 31 Jul 2025). Phi-Ground is defined as a family of models specialized for the latter task.

The paper’s framing implies a modular separation of concerns. A large multimodal LLM such as GPT-4O or O4-mini expands a short instruction into a detailed Long RE containing functional, positional, and appearance sub-references; the Phi-Ground model then consumes that Long RE together with the screenshot and outputs exact click coordinates (Zhang et al., 31 Jul 2025). This suggests a deliberate architectural choice to offload semantic disambiguation and longer-horizon reasoning to a larger planner while constraining the grounding model to high-precision perception.

A plausible implication is that Phi-Ground should be understood not as a general-purpose agent model, but as a perception-localization subsystem within an agent pipeline. That interpretation is consistent with the paper’s stated emphasis on precise localization and with its explicit claim that current end-to-end grounding models remain below deployment-ready accuracy on challenging benchmarks (Zhang et al., 31 Jul 2025).

2. Architectural design

Phi-Ground models are fine-tuned vision-language decoders built on pretrained VLM backbones, including Phi-3.5-Vision-Instruct with a CLIP image encoder at 4.1 B parameters and Phi-4-MM with SigLIP at 7 B parameters (Zhang et al., 31 Jul 2025). The architecture uses image patch cropping through a “num_crops” mechanism: each screenshot is scaled and white-padded to a grid such as $3\times 5$ or $4\times 7$, producing $N$ image crops plus one global crop, for a total of $N+1$ image tokens (Zhang et al., 31 Jul 2025).

The tokenization order is text first and image second. If the reference-expression tokens are $w_1,\dots,w_L$ and the image patch embeddings are $v_1,\dots,v_N$, the sequence is

$[w_1,\dots,w_L,v_1,\dots,v_N].$

In the notation given for the model,

$X=[E_{\text{text}};E_{\text{img}}],$

where $$E_{\text{text}}=[\mathrm{emb}(w_1),\dots,\mathrm{emb}(w_L)]$$ and $$E_{\text{img}}=[\mathrm{emb}(v_1),\dots,\mathrm{emb}(v_N)]$$ (Zhang et al., 31 Jul 2025). A standard decoder $4\times 7$0 then maps the concatenated sequence to hidden states, and next-token prediction is used to produce output coordinates.

The coordinate representation is textual rather than direct regression. Relative coordinates $4\times 7$1 are normalized to $4\times 7$2, scaled by 1000, and rounded to integers; the model then generates $N$9 using next-token prediction with cross-entropy loss (Zhang et al., 31 Jul 2025). This places coordinate prediction inside the generative token stream, aligning the output format with the decoder-only training paradigm.

The empirical analyses reported for Phi-Ground strongly favor this design. Text-to-image input order outperforms image-to-text by up to $4\times 7$3 points on ScreenSpot-pro, and direct point prediction yields the highest click accuracy, whereas XYXY boxes yield the best IoU (Zhang et al., 31 Jul 2025). The paper also reports that, under a fixed GPU-day budget, increasing image tokens up to approximately 2000 yields larger gains on high-resolution benchmarks than merely adding parameters, indicating that visual token budget is a key systems variable for grounding performance (Zhang et al., 31 Jul 2025).

3. Data construction and preprocessing

Phi-Ground training uses approximately 40 M samples derived from four sources (Zhang et al., 31 Jul 2025). The first source is open-source GUI data, approximately 10 M samples, from OS-Atlas, SeeClick, E2ISynth, and GUIAct; all elements are re-annotated with “Long-gold” reference expressions via GPT-4O (Zhang et al., 31 Jul 2025). The second source is CommonCrawl Rendering, approximately 10.5 M elements, created by rendering filtered CC-MAIN webpages at random resolutions and aspect ratios, retaining interactive elements through JS tag, role, and class heuristics, applying rule-based filtering, uniformly resampling across a $4\times 7$4 grid, and then annotating with GPT-4O (Zhang et al., 31 Jul 2025).

The third source is Web Search Data, approximately 158 K examples, obtained from Bing Image Search queries over 66 top Windows apps; screenshots are filtered by a CLIP classifier, bounding boxes are produced with OmniParserV2, and GPT-4O generates the reference expressions (Zhang et al., 31 Jul 2025). The fourth source is Human-labeled In-domain Data, approximately 80 K examples, collected from screen recordings in target Windows software, with UIA-tree or OmniParser bounding boxes manually cleaned and then annotated with GPT-4O “Long-gold” reference expressions (Zhang et al., 31 Jul 2025).

Two augmentation procedures are reported. Random Crop proportionally crops each side with probability 0.3 and uses $4\times 7$5; Random Resize+Padding shrinks the original image by a random scale in $4\times 7$6 onto a fixed white canvas (Zhang et al., 31 Jul 2025). The reported gain from always applying random resize is $4\times 7$7 points on ScreenSpot-pro (Zhang et al., 31 Jul 2025).

This data pipeline is one of the central claims of the work. The paper characterizes itself as an empirical study spanning data collection and model training, and the breadth of the data sources suggests that Phi-Ground’s performance is tied not only to backbone selection but also to the normalization of reference expressions, large-scale GUI rendering, and targeted in-domain data construction (Zhang et al., 31 Jul 2025).

4. Training methodology

The pre-training setup reported for Phi-Ground uses up to 40 M total samples, a batch size of 8192, learning rate $4\times 7$8, $4\times 7$9, $N$0, and a warmup of 6% of training steps (Zhang et al., 31 Jul 2025). Training compute is reported as approximately 200–450 A100-GPU days depending on model and configuration (Zhang et al., 31 Jul 2025).

The loss function is standard next-token cross-entropy. The authors explicitly state that alternative losses they ablated, including tokenized coordinates, label smoothing, and loss re-weighting, showed no benefit at scale (Zhang et al., 31 Jul 2025). This is important because it narrows the source of performance gains to data, representation, cropping strategy, and post-training rather than to bespoke objective engineering.

For in-domain post-training on a Photoshop subset of 200 K samples over approximately 3 epochs each, the paper evaluates three strategies: mixing domain data during pre-training, fine-tuning only on domain data, and introducing a small domain mix in pre-training followed by fine-tuning on domain data (Zhang et al., 31 Jul 2025). The third strategy, labeled Strategy C, is reported to best balance general and in-domain performance (Zhang et al., 31 Jul 2025).

The post-training stage explores supervised fine-tuning, curriculum learning, and preference-based reinforcement learning variants including Reject Sampling and DPO variants (Zhang et al., 31 Jul 2025). The strongest reported result is that multi-round DPO with a sigmoid objective yields $N$1 points on ScreenSpot-pro and $N$2 points on Gold-S over SFT (Zhang et al., 31 Jul 2025). This places preference optimization as a meaningful contributor to the final model family.

A plausible implication is that Phi-Ground’s training recipe is deliberately conservative in loss design but aggressive in dataset engineering and post-training alignment. The paper’s ablations support that interpretation by reporting no benefit from alternative token-level losses while showing consistent gains from input order, data composition, image-token scaling, and DPO-based post-training (Zhang et al., 31 Jul 2025).

5. Evaluation and benchmark performance

Phi-Ground evaluates GUI grounding with click accuracy

$N$3

and with Intersection-over-Union for box outputs (Zhang et al., 31 Jul 2025). The paper reports performance across five grounding benchmarks and emphasizes results for models under 10 B parameters in both end-to-end and agent settings.

In the end-to-end setting with short reference expressions, the reported results are as follows (Zhang et al., 31 Jul 2025):

Model	ScreenSpot-pro	UI-Vision AVG	Showdown	Gold-S
UI-TARS-1.5-7B	42.6	22.3	67.2	86.7
Phi-Ground-4B-16C-DPO	38.0	24.5	58.2	87.2
Phi-Ground-7B-16C-DPO	43.2	27.2	62.5	84.4

In the agent setting, where long reference expressions are generated via O4-mini, the reported results are (Zhang et al., 31 Jul 2025):

Model	ScreenSpot-pro	UI-Vision AVG	Showdown	Gold-S
UI-TARS-1.5-7B	48.8	29.4	71.6	90.4
Phi-Ground-4B-16C-DPO	51.5	35.6	73.5	95.2
Phi-Ground-7B-16C-DPO	55.0	36.2	73.9	93.8

The article’s central quantitative claim is that Phi-Ground achieves state-of-the-art performance across all five grounding benchmarks for models under $N$4 B parameters in agent settings, and that in the end-to-end setting it reaches $N$5 on ScreenSpot-pro and $N$6 on UI-Vision (Zhang et al., 31 Jul 2025). The agent-setting improvement over end-to-end evaluation is consistent with the paper’s two-stage design, in which the long-reference-expression planner reduces the reasoning burden on the grounding model.

The ablation results provide more specific causal attribution. Adding 30% resampled CommonCrawl and 3% BingSearch improves ScreenSpot-pro by $N$7 point; input order matters; and DPO-sigmoid further improves performance over SFT (Zhang et al., 31 Jul 2025). The error analysis on ScreenSpot-pro with $N$8 assigns 30.7% of errors to planning omissions, 24.8% to planner hallucinations, 9.3% to reference-expression style sensitivity, 12.3% to non-English text issues, and 22.9% to intrinsically hard cases (Zhang et al., 31 Jul 2025). This distribution indicates that a substantial fraction of failures lies upstream of localization itself.

6. Limitations, interpretation, and relation to broader grounding research

The paper explicitly states several limitations. Privacy concerns arise when uploading user screenshots, irreversible GUI errors require robust verification, and the current focus is on click grounding, with keyboard or text grounding delegated to the LLM (Zhang et al., 31 Jul 2025). These limitations clarify that Phi-Ground addresses a narrower but operationally critical subproblem in agent execution.

The reported error taxonomy also constrains interpretation. Because 30.7% of errors are attributed to vague reference expressions and 24.8% to planner hallucinations, benchmark performance for Phi-Ground in the agent setting should not be read as a pure measure of localization capability (Zhang et al., 31 Jul 2025). Instead, it reflects the joint performance of long-reference-expression generation and coordinate grounding. This suggests that the two-stage architecture improves results partly by transforming the supervision interface rather than solely by improving screen understanding.

A common misconception would be to treat GUI grounding as equivalent to generic multimodal reasoning. The Phi-Ground report argues against that view by defining grounding as a specialized perceptual localization problem and by presenting a training recipe centered on image-token budgeting, crop strategy, annotation style, and coordinate output formatting (Zhang et al., 31 Jul 2025). Another misconception would be to assume that larger parameter count alone dominates performance; the reported scaling analysis instead indicates that, at fixed compute budget, increasing image tokens can matter more than merely adding parameters (Zhang et al., 31 Jul 2025).

The future directions identified by the work are privacy-preserving on-device grounding, post-grounding verification modules to avoid destructive actions, and extension of grounding techniques to richer multimodal perception tasks (Zhang et al., 31 Jul 2025). These directions are consistent with the system’s current limitations and with its framing as a modular component for reliable CUA deployment.

7. Significance

Phi-Ground occupies a specific place in the emerging systems stack for Computer Use Agents: it is a grounding subsystem optimized for localization under realistic GUI variability, trained at large scale, and evaluated in both end-to-end and planner-assisted settings (Zhang et al., 31 Jul 2025). Its principal technical contributions are a two-stage architecture separating instruction expansion from localization, a large-scale multimodal data pipeline with Long RE annotation, a text-first decoder formulation for coordinate generation, and an empirically validated training recipe emphasizing augmentation, image-token scaling, and DPO-based post-training (Zhang et al., 31 Jul 2025).

The broader significance of the work lies in its operational definition of GUI grounding as a perception problem with measurable failure modes. The benchmark results show that strong gains are possible within the sub-10 B regime, especially when the grounding model is paired with long reference expressions in an agent setting (Zhang et al., 31 Jul 2025). At the same time, the residual error profile and the stated concerns about privacy and destructive misactions indicate that the problem remains unresolved for deployment-critical environments.

In that sense, Phi-Ground is best understood as a technically focused step toward robust GUI actuation: not a complete agent, but a specialized model family that attempts to make screen-level action selection precise enough to support broader multimodal agent pipelines (Zhang et al., 31 Jul 2025).