UI-AGILE: Advanced GUI Agent Framework

• UI-AGILE is a framework that unifies agile UI design and sophisticated GUI-agent methodologies to enhance precise grounding in cluttered, high-resolution interfaces.
• It employs continuous grounding rewards, constrained reasoning, and cropping-based resampling to improve localization accuracy while mitigating visual noise.
• The framework achieves state-of-the-art performance by integrating training-time and inference-time optimizations, validating its superiority over traditional methods.

UI-AGILE refers, in its most specific contemporary usage, to a 2025 framework for improving Multimodal LLM (MLLM) GUI agents at both training time and inference time, with emphasis on precise grounding in cluttered, high-resolution interfaces (Lian et al., 29 Jul 2025). In a broader research lineage, closely related “UI-Agile” usage had already denoted agile software processes that integrate UX, UCD, prototyping, and usability evaluation directly into iterative development rather than treating interface work as a downstream activity (López et al., 2023). Across these usages, the unifying theme is that user interfaces are treated as primary technical objects of reasoning, validation, and iteration.

1. Terminological scope and research lineage

Before the appearance of the 2025 GUI-agent framework, “UI-Agile” was already used in software engineering discourse to describe agile development practices that make interface and user-centered concerns first-class. In one influential educational process model, an Android development course was organized around a tailored hybrid of Scrum, Extreme Programming, User Experience, and User-Centered Design. That process grouped work into preproduction, production, and postproduction; emphasized client interviews, personas, navigation and UI design, low-fidelity prototyping before substantial coding, and usability testing with target users using SUS as part of validation (López et al., 2023). A related graduate-course integration of Lean UX into Scrum, GLUX, structured sprint work around Hypotheses, Design Studio, Experiment Stories, MVP, and Weekly User Experiments, making UX learning loops part of Scrum cadence rather than a separate track (Alhammad et al., 2022).

The 2025 framework titled “UI-AGILE” marks a terminological shift from process integration toward agentic interface intelligence. In this later sense, UI-AGILE no longer primarily denotes a development methodology for people; it denotes a technical framework for training and deploying GUI agents. The continuity between the two senses is conceptual rather than procedural: both reject the idea that UI work is peripheral. The earlier literature foregrounded stakeholder collaboration, preproduction design, and quality in use. The later framework foregrounds precise grounding, reward shaping, and inference-time visual decomposition. This suggests a broad evolution from human-centered agile UI process design toward model-centered UI reasoning and action.

2. Core problem formulation in the 2025 framework

In the 2025 sense, UI-AGILE addresses GUI action prediction from a screenshot and a user instruction, in settings where an agent must identify both what action to perform and where to ground it on the screen. The framework is motivated by three coupled problems in prior GUI-agent pipelines: a dilemma in reasoning design, ineffective reward signals during reinforcement fine-tuning, and severe visual noise on large screenshots (Lian et al., 29 Jul 2025).

The reasoning dilemma is that elaborate chains of thought can help with non-grounding decisions such as action-type prediction, but can also hurt grounding precision and increase latency. UI-AGILE therefore rejects both extremes—verbose reasoning and “No Thinking”—and instead rewards concise, task-appropriate reasoning. The reward problem is that binary inside-the-box localization rewards are too coarse for GUI tasks: a point near the semantic center of a target is treated no differently from a point barely inside its boundary, and difficult samples often yield only zero reward. The visual-noise problem is that high-resolution screens produce very large visual token sequences, making small targets difficult to localize even for otherwise competent models.

The framework modifies both training and inference. During training, it fine-tunes Qwen2.5-VL-3B-Instruct and Qwen2.5-VL-7B-Instruct with GRPO-style reinforcement learning over roughly 9k GUI examples sampled from UI-R1, GUI-R1, Aguvis, and Grounding-R1, filtered using OmniParser. During inference, it replaces direct full-screen grounding with a decomposition-and-selection pipeline that grounds on overlapping crops and then selects among candidates (Lian et al., 29 Jul 2025).

3. Training-time design: continuous reward, constrained reasoning, and resampling

UI-AGILE’s central training contribution is a continuous grounding reward that favors center-seeking localization over mere box inclusion. The paper defines:

$$R(x,y)= \begin{cases} 1+\exp(-4 d_{\text{norm}}^2), & (x,y)\in \text{BBox} \ 0, & \text{otherwise} \end{cases}$$

with normalized distance

$$d_{\text{norm}}=\max\left(\frac{|x-c_x|}{w_h}, \frac{|y-c_y|}{h_h}\right),$$

where $(c_x,c_y)$ is the box center and $(w_h,h_h)$ are half-width and half-height (Lian et al., 29 Jul 2025). The use of $L_\infty$ geometry aligns reward contours with rectangular GUI elements rather than circular distance fields. Operationally, the reward approaches 2 near the center, remains above 1 near the edges, and drops to 0 outside the box.

The second training contribution is the “Simple Thinking” reward, which ties reasoning reward to successful grounding and favors an ideal interval of reasoning length rather than unlimited verbosity. The paper expresses the high-level form as

$$R_{\text{think}} = I(R_{\text{grounding}}>0)\cdot\left(R_{\text{length}}(L)+R_{\text{bonus}}\right),$$

where $R_{\text{length}}(L)$ rewards an ideal length range with cosine ramps outside that interval, and $R_{\text{bonus}}$ rewards syntactically complete thoughts (Lian et al., 29 Jul 2025). The important point is not a long chain of thought per se, but sufficient reasoning for action-type prediction without degrading grounding.

The third contribution is cropping-based resampling for sparse-reward cases. When all generated responses for a screenshot–instruction pair receive zero grounding reward, UI-AGILE crops the screenshot into easier views that still fully contain the target bounding box, then resamples on those views. Crop size is defined as

$$(w_{\text{crop}}, h_{\text{crop}}) = (w_o \times f, h_o \times f),$$

with default cropping factor $f=0.6$, and scan strides are

$$d_{\text{norm}}=\max\left(\frac{|x-c_x|}{w_h}, \frac{|y-c_y|}{h_h}\right),$$0

where $$d_{\text{norm}}=\max\left(\frac{|x-c_x|}{w_h}, \frac{|y-c_y|}{h_h}\right),$$1 are target-box dimensions (Lian et al., 29 Jul 2025). This avoids the shortcut of always centering the target in the crop and functions as a curriculum-like difficulty adjustment. The paper reports that in the first epoch only 61.8% of training steps were fully successful on the initial attempt, implying that without resampling at least 19.1% of training samples would have contributed no learning signal.

4. Inference-time grounding: decomposition and selection

UI-AGILE’s inference-time method, “Decomposed Grounding with Selection,” is designed for high-resolution interfaces where direct full-screen grounding is unreliable (Lian et al., 29 Jul 2025). The paper motivates it with an oracle experiment on ScreenSpot-Pro in which cropping each image to $$d_{\text{norm}}=\max\left(\frac{|x-c_x|}{w_h}, \frac{|y-c_y|}{h_h}\right),$$2 while retaining the ground-truth target raises UGround-V1-7B from 31.6 to 56.0 average grounding accuracy. Since oracle cropping is impossible at test time, UI-AGILE approximates its benefit by splitting the full screenshot into four overlapping sub-images, each scaled to 60% of the original dimensions with 10% overlap.

The agent grounds independently on each sub-image, yielding multiple candidate coordinates. Around each predicted point, the system extracts a local element image using a simple box whose width and height are 14% of the sub-image width and height. A selector VLM, Qwen2.5VL-7B-Instruct in the reported experiments, is then prompted with the user instruction and the candidate element image; the logit for the token “Yes” is used as the relevance score, and the highest-scoring candidate is remapped back to the original screenshot (Lian et al., 29 Jul 2025).

The method reduces noise in two ways: it increases target scale relative to input size, and it adds semantic verification after candidate generation. The paper also gives a favorable complexity intuition: if one image with $$d_{\text{norm}}=\max\left(\frac{|x-c_x|}{w_h}, \frac{|y-c_y|}{h_h}\right),$$3 tokens is split into four sub-images of roughly $$d_{\text{norm}}=\max\left(\frac{|x-c_x|}{w_h}, \frac{|y-c_y|}{h_h}\right),$$4 tokens each, self-attention cost scales as

$$d_{\text{norm}}=\max\left(\frac{|x-c_x|}{w_h}, \frac{|y-c_y|}{h_h}\right),$$5

so prefilling can become cheaper despite multiple crops (Lian et al., 29 Jul 2025). In practice, on ScreenSpot-Pro, baseline grounding with UI-AGILE-7B takes 30 minutes, decomposed grounding 35 minutes, and the selection stage adds 4 minutes.

5. Empirical performance and benchmark position

UI-AGILE reports state-of-the-art results on both ScreenSpot-Pro and ScreenSpot-v2. On ScreenSpot-Pro, UI-AGILE-7B reaches 44.0 average grounding accuracy without inference enhancement and 48.7 with decomposed grounding and selection, which the paper describes as a 23% improvement over the best baseline, JEDI-7B at 39.5 (Lian et al., 29 Jul 2025). UI-AGILE-3B also performs strongly, rising from 37.9 to 45.0 with decomposition. On ScreenSpot-v2, UI-AGILE-7B reaches 92.1 average, exceeding UI-TARS-7B at 91.6 and UI-TARS-72B at 90.3.

The training-only gains are also notable. UI-AGILE-7B at 44.0 surpasses several reinforcement-fine-tuned or zero-shot GUI models on ScreenSpot-Pro, including UI-R1-E at 33.5, InfiGUI-R1-3B at 35.7, GUI-R1-7B at 32.1, and UI-TARS-72B at 38.1. On AndroidControl, UI-AGILE-7B attains step success rates of 77.6 on AndroidControl-Low and 60.6 on AndroidControl-High, while UI-AGILE-3B reaches 74.3 and 56.9, respectively (Lian et al., 29 Jul 2025).

The ablation narrative attributes most grounding gains to the continuous reward and cropping-based resampling. The paper reports that adding the continuous grounding reward improves ScreenSpot-Pro by about 10%, and adding cropping-based resampling improves it by about 12.4%. “Simple Thinking” slightly reduces pure grounding accuracy relative to “No Thinking,” but improves AndroidControl step success by 15.5% in the Low setting and 3.4% in the High setting, supporting the claim that concise reasoning is useful for broader agentic behavior even if pure pointing can benefit from less reasoning (Lian et al., 29 Jul 2025).

The inference method is also presented as plug-and-play. On ScreenSpot-Pro, decomposed grounding improves OS-Atlas-7B from 18.9 to 33.1 and Aguvis-7B from 20.4 to 36.5, corresponding to relative improvements of 75.1% and 78.9%, respectively (Lian et al., 29 Jul 2025). This indicates that the visual-noise problem is not specific to UI-AGILE’s own policy model.

6. Adjacent research, limitations, and broader significance

UI-AGILE occupies one region of a larger research landscape in which neighboring systems isolate different UI-agent bottlenecks. AURORA addresses semantics-aware mobile exploration by classifying screens into learned design motifs and invoking motif-specific heuristics, improving method coverage over VET by 19.6% (Khan et al., 2024). RUIG addresses metadata-free screenshot-based instruction grounding and reaches 81.16 and 73.92 accuracy on App Seen and App Unseen mobile settings, respectively, without requiring view hierarchies or pre-extracted candidate boxes (Zhang et al., 2023). Agent+P adds UTG-based symbolic planning to Android UI agents, improving success rates on AndroidWorld by up to 14% and reducing action steps by 37.7% (Ma et al., 7 Oct 2025). UIExplore-Bench, in turn, evaluates exploration independently of downstream task solving and reports 77.2% and 59.0% of human performance for UIExplore-AlGo in Structured and Screen modes at 2,000 steps (Nica et al., 21 Jun 2025). A plausible implication is that UI-AGILE solves the grounding-and-reward portion of the GUI-agent stack, but does not by itself subsume exploration policy, symbolic planning, or long-horizon world modeling.

The framework also has explicit limitations. The paper does not provide a full GRPO loss specification, exact coordinate tokenization, or detailed response schema. The selection model is a general-purpose pretrained VLM rather than a selector specialized on UI candidate choice. The failure analysis is narrower than the success analysis, and the authors explicitly identify the selection stage as a target for future fine-tuning (Lian et al., 29 Jul 2025). More broadly, the framework focuses on point grounding and action-type prediction rather than rich workflow abstractions or multi-step symbolic plans.

In this broader sense, UI-AGILE is best understood as a major step in the specialization of GUI agents: it formalizes center-sensitive localization reward, treats reasoning length as an optimizable design variable, and introduces an inference-time decomposition mechanism for realistic high-resolution screens. Earlier “UI-Agile” work had shown that interface concerns become more effective when integrated into the development loop rather than postponed; the 2025 framework carries a related principle into agent design by integrating precise grounding concerns into both optimization and deployment rather than treating them as downstream postprocessing.