Adaptive Stage Controlling for GUI Grounding

• Adaptive Stage Controlling (ASC) is a learned stage-selection mechanism in GUI grounding that dynamically decides whether to trigger an extra refinement stage.
• It leverages a two-phase training approach—supervised fine-tuning followed by reinforcement fine-tuning with GRPO—to optimize stage control decisions.
• Empirical evaluations demonstrate that ASC significantly improves performance on complex benchmarks by invoking refinement only when necessary.

Adaptive Stage Controlling (ASC) denotes a learned stage-selection mechanism for GUI grounding in the GUI-ARP framework. Its purpose is to decide whether a screenshot–instruction pair should be resolved by a single global inference pass or should invoke an additional refinement stage via Adaptive Region Perception (ARP), so that simple cases remain single-stage while difficult cases receive further observation (Ye et al., 19 Sep 2025). In this formulation, stage depth is not fixed heuristically: it is expressed as an explicit policy through structured control tokens and optimized jointly with grounding behavior.

1. Terminological scope and disambiguation

Recent arXiv usage makes the acronym ASC highly overloaded. The explicit use of Adaptive Stage Controlling appears in GUI grounding, whereas several nearby literatures either use the same acronym for unrelated concepts or discuss stage-adaptive behavior without adopting this name.

Usage	Meaning	Relation to Adaptive Stage Controlling
(Ye et al., 19 Sep 2025)	Adaptive Stage Controlling	Explicit usage
(Xu et al., 2023)	Appearance and Structure Consistency	Unrelated medical-image UDA
(Lu et al., 2022)	Atomic Skill Completion	Different embodied-AI hierarchy
(Yokoyama et al., 2023)	Adaptive Skill Coordination	Related skill coordination, different term
(Shams et al., 2023)	No ASC term	Stage-based traffic control, interpretive only

In GUI-ARP, ASC is a named component and is technically specific: it is the mechanism that determines whether ARP should be invoked for finer-grained observation (Ye et al., 19 Sep 2025). By contrast, the fetal-brain MRI paper defines ASC as Appearance and Structure Consistency, a teacher–student unsupervised domain adaptation framework with Fourier appearance transformation and structure perturbation, with no notion of stages or stage control (Xu et al., 2023). The embodied-AI paper defines ASC as Atomic Skill Completion, a hierarchical skill-composition method rather than a stage-controller in the GUI-ARP sense (Lu et al., 2022). The mobile-manipulation paper uses Adaptive Skill Coordination, which is close in spirit to reactive stage/skill selection but is explicitly not called Adaptive Stage Controlling (Yokoyama et al., 2023).

A further source of ambiguity arises in adaptive traffic signal control. The taxonomy review distinguishes phase-based and stage-based operation and notes that “Elsewhere, stage-based control is more common,” but it does not define an ATSC family called Adaptive Stage Controlling. Any mapping from “ASC” into that taxonomy is therefore interpretive and depends on whether the controller remains cyclic, pseudo-cyclic, or becomes real-time acyclic (Shams et al., 2023).

2. Functional role within GUI-ARP

ASC was introduced to address a specific defect of fixed inference depth in GUI grounding. GUI-ARP targets screenshot grounding under high resolution, dense layouts, and visually similar elements, where existing VLM-based methods often rely on a single pass over the whole screenshot. The paper attributes part of the difficulty to the patchify operation in ViT backbones, which weakens fine-grained spatial perception (Ye et al., 19 Sep 2025). A fixed single-stage strategy is efficient but often too coarse; a fixed multi-stage strategy incurs unnecessary overhead and can remove useful global context on easy cases.

Within GUI-ARP, ASC and ARP are complementary. ARP determines how to crop a task-relevant region from the model’s attention map. ASC determines whether such refinement should occur at all. The paper describes this as a move from passive perception to active visual cognition: first observe the whole screenshot, then zoom in only when further observation is necessary (Ye et al., 19 Sep 2025).

Operationally, ASC is not a separate controller network. It is implemented as a structured output decision of the main VLM, using the explicit interface <tool_call> yes </tool_call> or <tool_call> no </tool_call>. A “yes” triggers ARP and another grounding pass on a refined crop; a “no” terminates inference after the current stage. The paper summarizes the role succinctly: given a screenshot and instruction, ASC assesses task difficulty and controls the multi-stage process (Ye et al., 19 Sep 2025).

3. Decision mechanism and training formulation

ASC is learned in a two-phase pipeline consisting of Supervised Fine-Tuning (SFT) and Reinforcement Fine-Tuning with Group Relative Policy Optimization (GRPO) (Ye et al., 19 Sep 2025). During SFT, the model is trained to emit Chain-of-Thought reasoning and the structured <tool_call> decision token. The supervision signal for stage control is synthesized rather than manually authored: samples solved correctly in one stage are labeled easy, while samples requiring ARP for correct grounding are labeled challenging. This directly couples stage selection to empirical difficulty.

The SFT objective combines next-token prediction and attention supervision:

$$\mathcal{L}_{NTP}(\theta)=-\frac{1}{T} \sum_{t=1}^{T} \log P\left(x_{t} \mid x_{0}, x_{1}, \ldots, x_{t-1} ; \theta\right),$$

$$\mathcal{L}_{Attn} = \sum_{i=1}^{M} p_i \log \frac{p_i}{a_i}, \quad p_i = \frac{y_i}{\sum_{j=1}^{M} y_j + \epsilon},$$

$$\mathcal{L} = \mathcal{L}_{NTP} + \mathcal{L}_{Attn}.$$

ASC is trained primarily through $\mathcal{L}_{NTP}$ because the tool-call token is part of the generated sequence, while $\mathcal{L}_{Attn}$ supports ARP indirectly by improving the attention map used for cropping (Ye et al., 19 Sep 2025).

GRPO sharpens the policy using an explicit reward for correct stage decisions. The ASC-specific term is

$$\mathcal{R}_{tool} = \begin{cases} 1, & \text{if}\quad (\text{tool call = no} \land (c_x^p, c_y^p) \in gt_{bbox}) \ & \quad \lor\ (\text{tool call = yes} \land (c_x^p, c_y^p) \notin gt_{bbox}), \ 0, & \text{otherwise.} \end{cases}$$

This rewards no refinement when the current prediction is already inside the ground-truth box, and refinement when the current prediction is outside it. The total reinforcement signal is

$$\mathcal{R} = \mathcal{R}_{format} + \mathcal{R}_{tool} + \mathcal{R}_{point}.$$

Thus ASC is optimized as a token-level routing policy embedded in the VLM rather than as a standalone decision head (Ye et al., 19 Sep 2025).

The paper is explicit about what is not provided. There is no separate ASC classifier architecture, no confidence threshold, no entropy-based stopping criterion, and no formal arbitrary-depth controller. This suggests that ASC is best understood as a learned binary stage-gating policy, not as a fully specified generalized controller over many refinement levels (Ye et al., 19 Sep 2025).

4. Inference workflow and stage semantics

A stage in GUI-ARP is one grounding pass over an image view. Stage 1 operates on the full screenshot. The model jointly encodes screenshot and instruction, and a special contextual anchor token <ACTOR> attends over visual patch tokens to produce an attention map. That attention distribution indicates spatial relevance to the target (Ye et al., 19 Sep 2025).

After the first grounding pass, ASC emits the tool-call decision. If the output is <tool_call> no </tool_call>, the system stops and returns the current localization result. If the output is <tool_call> yes </tool_call>, ARP is invoked. ARP thresholds the attention map,

$$\mathcal{S} \gets \{(i,j) \mid \mathbf{A}_{i,j} \ge \tau \cdot \max(\mathbf{A}) \},$$

splits the selected support into connected components,

$$\{\mathcal{G}_1, \mathcal{G}_2, \dots\} \gets \text{ConnectedComponents}(\mathcal{S}),$$

selects the top-$k$ regions, computes their weighted centers,

$$\mathcal{L}_{Attn} = \sum_{i=1}^{M} p_i \log \frac{p_i}{a_i}, \quad p_i = \frac{y_i}{\sum_{j=1}^{M} y_j + \epsilon},$$0

and constructs a crop from the extrema of those centers:

$$\mathcal{L}_{Attn} = \sum_{i=1}^{M} p_i \log \frac{p_i}{a_i}, \quad p_i = \frac{y_i}{\sum_{j=1}^{M} y_j + \epsilon},$$1

The inference hyperparameters reported for ARP are $$\mathcal{L}_{Attn} = \sum_{i=1}^{M} p_i \log \frac{p_i}{a_i}, \quad p_i = \frac{y_i}{\sum_{j=1}^{M} y_j + \epsilon},$$2 and $$\mathcal{L}_{Attn} = \sum_{i=1}^{M} p_i \log \frac{p_i}{a_i}, \quad p_i = \frac{y_i}{\sum_{j=1}^{M} y_j + \epsilon},$$3 (Ye et al., 19 Sep 2025).

The paper consistently describes the overall method as adaptive multi-stage inference, but its operational descriptions are centered on a two-stage pattern: a global first pass and an ARP-refined second pass. This suggests that “multi-stage” is used in a general sense, whereas the concretely specified mechanism is primarily a learned decision about whether to trigger one additional observation stage (Ye et al., 19 Sep 2025).

5. Empirical characteristics

The most direct evidence for ASC is the relationship between tool-call rates and benchmark difficulty. The paper reports the following ablation results:

Model state	ScreenSpot-Pro	ScreenSpot-v2
Baseline	44.6, Tool Call 0.0%	91.0, Tool Call 0.0%
After SFT	54.3, Tool Call 29.7%	91.6, Tool Call 11.5%
After GRPO	60.8, Tool Call 92.4%	91.8, Tool Call 33.4%

These results show that tool usage is much higher on ScreenSpot-Pro than on ScreenSpot-v2, matching the claim that ASC learns dataset-dependent difficulty assessment rather than issuing refinement indiscriminately (Ye et al., 19 Sep 2025).

The single-stage-only ablations further clarify the role of stage control. After SFT, the single-stage-only version scores 50.4 on ScreenSpot-Pro and 92.0 on ScreenSpot-v2; after GRPO, the corresponding scores are 50.7 and 92.1. Relative to these values, adaptive refinement materially improves the harder benchmark while offering little benefit on the easier one, where preserving global context can be slightly preferable (Ye et al., 19 Sep 2025). This is precisely the tradeoff ASC is meant to exploit.

At the system level, GUI-ARP-7B reaches 60.8% on ScreenSpot-Pro and 30.9% on UI-Vision. The paper also reports that GUI-ARP-7B is competitive with open-source 72B-scale systems, citing UI-TARS-72B at 38.1% (Ye et al., 19 Sep 2025). The empirical significance of ASC lies less in raw token generation than in selective inference-depth control: it is the mechanism that lets GUI-ARP use extra visual scrutiny heavily on complex screenshots while suppressing unnecessary refinement on easier layouts.

6. Conceptual position, limitations, and related stage-adaptive research

ASC, in the strict sense, is currently a narrow term. Its explicit formulation is a GUI-grounding policy that decides whether another visual observation stage is necessary. It is therefore better characterized as learned stage gating than as a general theory of staged control (Ye et al., 19 Sep 2025). The paper itself leaves several issues open: ASC internals are not deeply specified, “multi-stage” is described more generally than it is operationalized, and efficiency is argued mainly through tool-call rates rather than through direct latency or FLOP measurements (Ye et al., 19 Sep 2025).

Outside GUI grounding, several bodies of work are conceptually adjacent but terminologically distinct. Adaptive horizon multi-stage MPC adapts the prediction horizon $$\mathcal{L}_{Attn} = \sum_{i=1}^{M} p_i \log \frac{p_i}{a_i}, \quad p_i = \frac{y_i}{\sum_{j=1}^{M} y_j + \epsilon},$$4 online while keeping the robust horizon $$\mathcal{L}_{Attn} = \sum_{i=1}^{M} p_i \log \frac{p_i}{a_i}, \quad p_i = \frac{y_i}{\sum_{j=1}^{M} y_j + \epsilon},$$5 fixed; this suggests a horizon-adaptive analogue of stage control rather than an explicit ASC mechanism (Mdoe et al., 2023). The SAXO+ second-stage adaptive-optics controller studies standalone versus disentangled cascaded AO, which suggests stage-aware inter-loop coordination but not a learned stage-selection policy (Dinis et al., 2024). The synchronisation-oriented adaptive-control formulation decomposes behavior into fast synchronization dynamics and slower collective dynamics, which suggests a control-theoretic analogue of stage separation rather than a named ASC framework (Cho et al., 2024).

A similar caution applies to traffic signal control. Stage-based signal operation is recognized as an alternative to phase-based operation, but no top-level ATSC family called Adaptive Stage Controlling is defined. A plausible implication is that “ASC” in that domain would be a descriptive label imposed on top of an existing taxonomy, not a native classification (Shams et al., 2023).

Taken together, these literatures suggest that the distinctive content of ASC is not the generic fact of operating in stages, but the explicit conversion of stage selection into a learned policy. In current arXiv usage, ASC is therefore most precisely understood as a VLM-integrated decision mechanism for adaptive inference depth in GUI grounding, with broader connections to staged adaptation remaining conceptual rather than terminological.