FineState-Bench: Benchmarking State-Conditioned Grounding for Fine-grained GUI State Setting

Abstract: Despite the rapid progress of large vision-LLMs (LVLMs), fine-grained, state-conditioned GUI interaction remains challenging. Current evaluations offer limited coverage, imprecise target-state definitions, and an overreliance on final-task success, obscuring where and why agents fail. To address this gap, we introduce \textbf{FineState-Bench}, a benchmark that evaluates whether an agent can correctly ground an instruction to the intended UI control and reach the exact target state. FineState-Bench comprises 2,209 instances across desktop, web, and mobile platforms, spanning four interaction families and 23 UI component types, with each instance explicitly specifying an exact target state for fine-grained state setting. We further propose \textit{FineState-Metrics}, a four-stage diagnostic pipeline with stage-wise success rates: Localization Success Rate (SR@Loc), Interaction Success Rate (SR@Int), Exact State Success Rate at Locate (ES-SR@Loc), and Exact State Success Rate at Interact (ES-SR@Int), and a plug-and-play \textit{Visual Diagnostic Assistant} (VDA) that generates a Description and a bounding-box Localization Hint to diagnose visual grounding reason via controlled w/ vs.\ w/o comparisons. On FineState-Bench, exact goal-state success remains low: ES-SR@Int peaks at 32.8\% on Web and 22.8\% on average across platforms. With VDA localization hints, Gemini-2.5-Flash gains +14.9 ES-SR@Int points, suggesting substantial headroom from improved visual grounding, yet overall accuracy is still insufficient for reliable fine-grained state-conditioned interaction \href{https://github.com/FengxianJi/FineState-Bench}{Github.}

• The paper introduces FineState-Bench, a benchmark that isolates GUI state-setting tasks with instance-level verification and precise visual grounding.
• It employs a four-stage protocol (SR@Loc, SR@Int, ES-SR@Loc, ES-SR@Int) to diagnose failures in localization and interaction accuracy.
• Empirical results show low exact state success rates for LVLM-based agents, emphasizing the need for enhanced visual grounding techniques.

FineState-Bench: Diagnostic Benchmarking of Fine-Grained State-Conditioned GUI Grounding

Motivation and Limitations of Prior Work

Existing large vision-LLMs (LVLMs) and multimodal GUI agents exhibit strong performance on a variety of interface control tasks, but evaluation methodologies remain coarse-grained in two key aspects: imprecise target-state specification, and conflation of perception, grounding, and action into single-aggregate task success rates. Tasks requiring exact—rather than just plausible—UI control state changes (e.g., setting a slider to 74%, selecting a particular date in a calendar, or picking an explicit color value) are poorly captured both in data resources and metrics used by prior benchmarks. Most current datasets for GUI and web/mobile agent evaluation prioritize click-level or end-to-end task completion, with state-conditioned success being underrepresented, and provide insufficient diagnostic resolution to attribute failures in the full perception-localization-interaction pipeline (2604.27974).

FineState-Bench: Dataset and Problem Formulation

FineState-Bench is introduced as a dedicated benchmark explicitly targeting single-step, fine-grained, state-conditioned GUI state setting, with instance-level goal state verification. It contains 2,209 human-annotated instances sampled across desktop, web, and mobile platforms. Each instance provides:

• A screenshot of the UI.
• Two natural language instructions (for component localization and goal-directed operation).
• Exact goal state and structured dual-region (box) annotations: control-extent (locate) and interactable-core for both current and target configuration.
• Explicit decoupling of the localization point ($p^0$) and interaction point ($p^1$).

Four interaction families are comprehensively covered: numerical/range adjustment, toggling/option selection, specific data-type selection, and view/content re-organization, spanning 23 component types. The benchmark uses a single-step, point-based interface to isolate the agent's ability to set controls to precise visual/semantic states, independent of multi-step corrective trajectories.

Diagnostic Evaluation: FineState-Metrics

To provide granularity and enable precise failure attribution, FineState-Bench introduces FineState-Metrics—a four-stage diagnostic protocol:

1. Localization Success Rate (SR@Loc): $p^0$ falls inside the control's visible bounding box. Grounding at the component level.
2. Interaction Success Rate (SR@Int): $p^1$ falls inside the interactable region that actually triggers the targeted state change. Captures fine point precision.
3. Exact State Success Rate at Locate (ES-SR@Loc): Verifies attainment of the exact goal state when $p^0$ is within the target (post-action) configuration.
4. Exact State Success Rate at Interact (ES-SR@Int): Strictest metric, requiring $p^1$ inside the interactable core in the target configuration and the exact desired final state.

This stage-wise decomposition exposes where agents typically fail: transition from component localization to interactable-core localization, and from correct interaction location to state-setting correctness.

Visual Diagnostic Assistant (VDA)

FineState-Bench introduces the Visual Diagnostic Assistant (VDA) as an analysis tool for controlled ablation studies. The VDA produces, given the screenshot and instruction, a structured description and a tight bounding-box Localization Hint for the target operation region. VDA is used to augment agent inputs—enabling precise performance quantification for cases where visual grounding (rather than perception/semantics or action implementation) is the dominant bottleneck. VDA is not proposed as a system component for deployment but as a plug-and-play diagnostic module for benchmarking.

Ablation studies reveal that only the Localization Hint—rather than the textual description—gives substantial gains, directly implicating visual core-localization as the key limiting factor.

Empirical Findings: Strong Numerical and Diagnostic Results

Extensive evaluation of eight representative GUI agents (three closed-source LVLMs and five open-source GUI-specialized agents) reveals consistently low exact goal state success under strict metrics:

• Best model (UGround-7B) achieves only 32.8% ES-SR@Int on Web, and 22.8% average across all platforms.
• Success is higher on discretized selection controls (e.g., radio buttons, tabs) than on continuous controls (e.g., sliders, seek bars), where even with good component localization, state setting precision remains poor.
• The gap between SR@Loc and SR@Int is substantial, particularly for precision-sensitive and continuous controls, directly quantifying the prevalence of errors in interactable-core localization.
• Using the VDA, Gemini-2.5-Flash obtains a +14.9% ES-SR@Int increase, localizing the principal recoverable gains to visual grounding improvements.

Component-wise analysis demonstrates the largest SR@Loc-to-SR@Int drops on sliders, knobs, seek bars, and zoom/pan interfaces, while categorical selectors remain systematically difficult for goal-state verification (near-zero interact success).

Implications and Future Directions

Practical Implications: Current LVLM-based GUI agents are not yet reliable for task automation scenarios demanding fine-grained, exact state specification—e.g., enterprise software, scientific instruments, or accessibility interfaces—all of which require strong guarantees about control setting without iterative manual correction. The observed bottlenecks clarify that improved general language-vision models, without explicit advances in visual grounding and spatial precision for GUI artifacts, will not close the gap.

Theoretical Implications: The diagnostic paradigm enforced by FineState-Bench advocates for decomposition of task success into sub-factors (perception, component grounding, interactable-core precision, and state-setting logic). Systematic evaluation with explicitly decoupled points and fine-grained geometric supervision enables clearer attribution, necessary for both training objectives and benchmarking trajectories as models continue to scale.

Open Problems and Future Work:

• Extension to multi-step, interactive task settings, combining the diagnostic rigor of FineState-Bench with long-horizon reasoning and correction.
• Integration of VDA-like grounding supervision into end-to-end system pipelines, potentially as auxiliary objectives or self-supervised targets for future models.
• More aggressive data augmentation and synthesis for underrepresented component types, rare UI patterns, and dynamic/adaptive layouts.

Conclusion

FineState-Bench provides the first cross-platform, exact-state, fine-grained benchmark for single-step, state-conditioned GUI state setting, and introduces a diagnostic framework (FineState-Metrics and VDA) capable of attributing failures to specific perception and grounding sub-tasks. The low exact goal-state success rates, persistent across architectures and platforms, underline the remaining gap for reliable state-aware GUI agents. The benchmark's evaluation paradigm is expected to foster more precise, robust, and diagnostically interpretable models and methodologies in the field of multimodal UI automation and agentic software control.

Citation:

FineState-Bench: Benchmarking State-Conditioned Grounding for Fine-grained GUI State Setting (2604.27974)