FineState-Bench: Fine-Grained GUI Control

• FineState-Bench is a benchmark for fine-grained GUI control that assesses an agent's ability to set exact target states rather than simply completing high-level tasks.
• It employs dual bounding-box schemes and a Visual Diagnostic Assistant to isolate precise control localization from general visual perception.
• Empirical evaluations reveal significant performance gaps across platforms, underscoring the urgent need for improved low-level spatial grounding in GUI agent interactions.

Searching arXiv for FineState-Bench and related GUI-agent benchmarking papers. FineState-Bench is the name of two closely related arXiv benchmark formulations for evaluating fine-grained GUI control by multimodal agents: “FineState-Bench: A Comprehensive Benchmark for Fine-Grained State Control in GUI Agents” and “FineState-Bench: Benchmarking State-Conditioned Grounding for Fine-grained GUI State Setting” (Ji et al., 12 Aug 2025, Ji et al., 30 Apr 2026). In both formulations, the central object of evaluation is not coarse task completion, but whether an agent can identify the intended UI control and manipulate it to the exact target state required by an instruction. The benchmark is motivated by the claim that prevailing GUI-agent evaluations can create an “illusion of capability”: systems may perform well on high-level workflows while failing on the low-level precision needed to set sliders, dates, colors, toggles, layout controls, or reordered content to the precise requested state (Ji et al., 12 Aug 2025).

1. Definition and conceptual scope

FineState-Bench targets what the papers call fine-grained GUI control. In the 2025 formulation, this is framed as evaluation of “fine-grained GUI proxy operations,” where a GUI agent must not only identify a relevant widget but also manipulate it to the exact target state demanded by the instruction. In the 2026 formulation, the emphasis is “state-conditioned grounding”: grounding an instruction to the specific operation location needed to set the intended control to an exact target state (Ji et al., 12 Aug 2025, Ji et al., 30 Apr 2026).

This definition distinguishes FineState-Bench from benchmarks centered on broad task success. A control can be semantically correct yet operationally wrong: a slider may be on the right track but at the wrong value, a date picker may be in the right calendar yet on the wrong day, and a color picker may be on the right component yet at the wrong color. FineState-Bench therefore treats GUI interaction as a state-setting problem rather than a mere click-grounding problem.

The later paper makes this framing more explicit by defining the benchmark around “single-step, fine-grained, state-conditioned GUI exact state setting.” That formulation isolates a precise capability: given a screenshot and instructions, predict the point that grounds the target control and the point that realizes the desired goal state. This suggests a shift from end-result proxy evaluation toward a more explicit separation between control identification, operation-point precision, and exact state attainment (Ji et al., 30 Apr 2026).

2. Benchmark versions, scale, and composition

The name FineState-Bench refers to two benchmark versions with the same core thesis but different dataset realizations and formalizations.

Version	Scale	Reported structure
2025 paper (Ji et al., 12 Aug 2025)	2,257 high-quality static samples	Desktop 814, Web 737, Mobile 706; four major task components; 23 distinct component types
2026 paper (Ji et al., 30 Apr 2026)	2,209 instances	Desktop 810, Web 701, Mobile 698; four interaction families; 23 UI component types

Both versions organize the benchmark around four task families. Their naming differs slightly, but the underlying categories are the same:

1. Numerical and Range Adjustment: sliders, knobs, steppers, seek bars, and chart points.
2. State Toggling and Option Selection: switches, checkboxes, radio groups, tabs, segmented controls, and accordions.
3. Specific Data-Type Selection: rating widgets, color pickers, date pickers, time pickers, and listbox/dropdown menus.
4. Content Organization and View Manipulation: drag reorder, zoom and pan, resizable panes, carousels, tree views, splitters, and table columns (Ji et al., 12 Aug 2025, Ji et al., 30 Apr 2026).

The 2025 paper describes construction through large-scale screening of applications across web, desktop, and mobile, including VLM-based pre-filtering from the os-atlas data source, followed by manual auditing, manual collection of missing scenarios, LabelImg annotation, and JSON storage with image, localization, instruction, and state metadata. The 2026 paper presents a five-step curation pipeline: filtering from OS-Atlas, manual screenshot supplementation, human annotation, LLM-assisted drafting of instruction–state pairs, and manual verification. Both descriptions place manual verification and exact state labeling at the center of benchmark quality control (Ji et al., 12 Aug 2025, Ji et al., 30 Apr 2026).

3. Task formalization and annotation scheme

The 2025 paper defines the benchmark task in terms of a natural-language instruction $I$, an initial UI state $W_{initial}$, an action sequence $A=\{a_1,\dots,a_T\}$, and a target state $G$. Each UI state $W_t$ contains interactable controls $C_t$, and each control $c\in C_t$ has a fine-grained state value $s(c)$. The operational goal is to drive the state of a target control $c^*$ to the required target value $s_{goal}$ (Ji et al., 12 Aug 2025).

The 2026 paper makes the state-conditioned grounding formulation explicit by defining each instance as

$W_{initial}$0

where $W_{initial}$1 is the screenshot, $W_{initial}$2 and $W_{initial}$3 are the locating and goal-directed instructions, $W_{initial}$4 is the target control, $W_{initial}$5 is the desired goal state, and the four boxes describe control extent and interactable core in current and target configurations. The agent predicts two points $W_{initial}$6: one for locating the target control and one for the operation intended to achieve the target state (Ji et al., 30 Apr 2026).

A central annotation innovation in both papers is the dual bounding-box scheme. The 2025 paper distinguishes a locate bounding box and an interact bounding box, normalized to $W_{initial}$7, to separate “seeing the target” from “acting precisely on the target.” The 2026 paper generalizes this into four boxes: $W_{initial}$8, $W_{initial}$9, $A=\{a_1,\dots,a_T\}$0, and $A=\{a_1,\dots,a_T\}$1, allowing evaluation against both current and target configurations. This matters because control geometry may change during interaction, as with moving slider knobs, expanded tree nodes, switched tabs, or resized panes (Ji et al., 12 Aug 2025, Ji et al., 30 Apr 2026).

The 2026 paper also reports an annotation reliability study on a stratified subset of $A=\{a_1,\dots,a_T\}$2 instances. It gives median IoU agreement of 83.5% for the locate box and 68.0% for the interactable-core box, with goal-state agreement of 97.5% overall, 93.0% for categorical components, and 88.5% for numeric components; the disagreement rate requiring adjudication is 6.7%. These values indicate that the interactable core is inherently more difficult to annotate than the larger control extent, while the exact goal-state labels remain comparatively stable (Ji et al., 30 Apr 2026).

4. Metrics and the perception-to-state diagnostic ladder

The 2025 paper defines a “four-phase indicator for comprehensive perception-to-control assessment” through four metrics for static tasks:

• Locate Success Rate (Loc SR):

$A=\{a_1,\dots,a_T\}$3

• Interact Success Rate (Int SR):

interaction lands on the target and drives the target to the correct state.

• Single-Action Locate Success Rate (SA-Loc SR):

$A=\{a_1,\dots,a_T\}$4

• Single-Action Interact Success Rate (SA-Int SR):

$A=\{a_1,\dots,a_T\}$5

The paper identifies SA-Int SR as the “gold standard,” because it requires correct localization and exact state-setting in the first action, with “no approximation allowed” in state matching (Ji et al., 12 Aug 2025).

The 2026 paper reformulates the diagnostic stack as FineState-Metrics. It conceptually aligns the pipeline with perception, localization, interaction, and state correctness, but formally defines four reported metrics:

$A=\{a_1,\dots,a_T\}$6

$A=\{a_1,\dots,a_T\}$7

$A=\{a_1,\dots,a_T\}$8

$A=\{a_1,\dots,a_T\}$9

Among these, ES-SR@Int is the primary metric (Ji et al., 30 Apr 2026).

The diagnostic interpretation is structurally consistent across the two papers. High localization success with much lower interaction success indicates that the model can find the right control but not the state-changing subregion. High interaction-region success with lower exact-state success indicates failure in translating a plausible operation into the precise target state. The 2026 paper further notes a common empirical pattern in which SR@Int is close to ES-SR@Int, suggesting that once the interactable core is truly hit, exact state attainment often follows. This suggests that the dominant failure mode is frequently not abstract task reasoning but precise interactable-core grounding (Ji et al., 30 Apr 2026).

5. Visual Diagnostic Assistant and controlled diagnosis of failure

A major innovation of FineState-Bench is the Visual Diagnostic Assistant (VDA), introduced to address what both papers describe as a diagnostic gap: benchmark-level failure does not by itself reveal whether the bottleneck lies in visual grounding, state perception, reasoning, or execution (Ji et al., 12 Aug 2025, Ji et al., 30 Apr 2026).

The 2025 paper describes VDA as a plug-and-play external visual localization system with a two-stage “describe-then-locate” pipeline. In Stage 1, GPT-4o produces a detailed description of the target UI element, including functional purpose, current state, visual features, spatial relations, and distinguishing cues. In Stage 2, GPT-4o predicts a high-precision normalized bounding box $G$0, which is then provided to the tested model in addition to the original screenshot and instruction. The benchmark stresses that VDA is diagnostic rather than a new agent architecture: the model’s reasoning path is intended to remain unchanged while visual localization is externally controlled (Ji et al., 12 Aug 2025).

The 2026 paper formalizes VDA around two outputs: a Description and a bounding-box Localization Hint $G$1. It defines a recoverability gap

$G$2

meant to estimate the portion of error recoverable through improved visual grounding. Controlled comparisons keep the evaluated agent, decoding, and point-based interface fixed while adding only the Description and/or Localization Hint (Ji et al., 30 Apr 2026).

The principal reported result is that Gemini-2.5-Flash gains +14.9 ES-SR@Int points with VDA localization hints. The 2025 paper reports the same headline improvement as a 14.9% boost from ideal visual localization. The 2026 ablations further report that Description-only yields essentially no improvement, Localization Hint alone yields most of the gain, and the full Description-plus-Hint setting performs best. The papers interpret this as evidence that current GUI agents are strongly bottlenecked by basic visual localization and interactable-core grounding rather than by description-level semantic augmentation alone (Ji et al., 12 Aug 2025, Ji et al., 30 Apr 2026).

6. Empirical performance and identified bottlenecks

The main empirical message of FineState-Bench is that exact fine-grained GUI state control remains poor. The 2025 paper states that “the most advanced models achieve only 32.8% fine-grained interaction accuracy.” The 2026 paper sharpens this by reporting that ES-SR@Int peaks at 32.8% on Web and 22.8% on average across platforms, with UGround-7B as the strongest baseline in its main table (Ji et al., 12 Aug 2025, Ji et al., 30 Apr 2026).

The 2026 baseline results also show strong platform heterogeneity. UGround-7B reports ES-SR@Int of 19.6 on Mobile, 32.8 on Web, and 16.0 on Desktop; Gemini-2.5-Flash reports 17.6, 11.8, and 0.7, respectively. This indicates that fine-grained GUI control is not a single monolithic capability; performance varies substantially by platform and, plausibly, by the interaction priors present in training data. A similar pattern is already visible in the 2025 paper, which notes that mobile, web, and desktop subsets expose different strengths and weaknesses (Ji et al., 12 Aug 2025, Ji et al., 30 Apr 2026).

Both papers emphasize the large drop from coarse localization to exact interaction. Representative average results from the 2025 paper include 53.0 / 22.8 / 36.6 / 22.8 for UGround-7B on Loc SR / Int SR / SA-Loc SR / SA-Int SR, 36.4 / 10.0 / 13.9 / 10.0 for Gemini-2.5-Flash, and 24.8 / 4.3 / 6.8 / 4.3 for GPT-4o. The 2026 baseline table exhibits the same pattern with SR@Loc substantially exceeding SR@Int for many models. The benchmark interprets this gap as evidence that many systems know roughly where to act but cannot identify the exact state-changing point (Ji et al., 12 Aug 2025, Ji et al., 30 Apr 2026).

Component-level results identify especially difficult classes. The 2026 appendix reports near-zero interact results for many C-family tasks, including rating, color picker, date picker, time picker, and list box/dropdown. It also highlights severe failures on precision-sensitive continuous controls such as sliders, knobs, and seek bars. By contrast, some discrete controls, such as steppers, radio groups, and tabs, are markedly more tractable. This suggests that the hardest cases are those in which a small coordinate error yields a materially different internal state (Ji et al., 30 Apr 2026).

The 2025 paper additionally identifies four qualitative failure modes: Localization Ambiguity, Visual Feature Confusion, Fine-Grained State Perception Failure, and Interaction Context Ignorance. VDA is reported to reduce these by 85%, 72%, 45%, and 23%, respectively. The ranking supports the interpretation that the largest reducible errors are low-level spatial grounding failures, while higher-order contextual failures remain more resistant (Ji et al., 12 Aug 2025).