GUI-Anchoring in Flux (GUI-AiF) Framework

• The paper introduces GUI-AiF, a framework that rewards diverse anchoring points to counteract overfitting and improve generalization in fluctuating GUI environments, achieving accuracy gains up to 83.5%.
• It presents two novel reward mechanisms, APR-iF and ARR-iF, that quantify spatial variance in candidate predictions to reinforce exploration and resilience against domain shifts.
• Empirical analyses show that integrating these rewards via reinforcement fine-tuning significantly mitigates catastrophic forgetting and enhances performance under continual digital interface changes.

GUI-Anchoring in Flux (GUI-AiF) is a reinforcement fine-tuning framework designed to stabilize continual learning in GUI agents exposed to dynamic, non-stationary digital environments. By explicitly encouraging exploration and alignment with shifting user interaction points and regions, GUI-AiF addresses the longstanding challenge of performance deterioration caused by overfitting to static interface cues as GUI distributions evolve over time, such as domain or resolution changes. The framework introduces two novel reward mechanisms—Anchoring Point Reward in Flux (APR-iF) and Anchoring Region Reward in Flux (ARR-iF)—that jointly guide grounding policies toward robust generalization in the presence of continual distribution shifts (Liu et al., 28 Jan 2026).

1. Motivation and Problem Statement

Conventional GUI agent training strategies rely on static point‐based instruction rewards, typically optimizing for minimal Euclidean distance or maximal IoU overlap between a predicted anchor point and a fixed ground-truth target. This setup induces a tendency for agents to collapse their click prediction policies onto a single coordinate, resulting in significant performance degradation following any subsequent domain transition (e.g., from mobile UI to web UI, or variation in display resolution). The GUI-Anchoring in Flux (GUI-AiF) paradigm addresses these limitations by rewarding diversity in proposed interaction points and regions, thus mitigating overspecialization and promoting resilience under continual data distribution flux (Liu et al., 28 Jan 2026).

2. Formal Definition and Mathematical Framework

GUI-AiF operationalizes its robustness incentives through two structured reward components. For the point-based signal, Anchoring Point Reward in Flux (APR-iF) is defined over a group of candidate bounding box predictions $\{\mathbf{b}^p_i\}_{i=1}^N$ per instruction. Each prediction is a tuple $$\mathbf{b}^p_i = [x^p_{1,i}, y^p_{1,i}, x^p_{2,i}, y^p_{2,i}] \in \mathbb{R}^4$$, with its center computed as

$$\mathbf{c}^p_i = \left( \frac{x^p_{1,i} + x^p_{2,i}}{2},\, \frac{y^p_{1,i} + y^p_{2,i}}{2} \right).$$

The centroid $\bar{\mathbf{c}}$ across the $N$ predicted centers defines the group mean,

$$\bar{\mathbf{c}} = \frac{1}{N}\sum_{j=1}^N \mathbf{c}^p_j.$$

The APR-iF reward is then

$R_{\mathrm{APR\mbox{-}iF}}\left(\mathbf{b}^p_{1:N}\right) = \mathcal{R}_p = \frac{1}{N}\sum_{i=1}^N \left\| \mathbf{c}^p_i - \bar{\mathbf{c}} \right\|_2^2,$

quantifying the empirical spatial variance of candidate anchor points. This reward is maximized when agent click predictions are spatially diverse. The overall GUI-AiF reward integrates point and region terms as $$R_{\text{AiF}} = \alpha\,\mathcal{R}_p + \gamma\,\mathcal{R}_r$$, where $\mathcal{R}_r$ captures a region diversity objective (ARR-iF), and $\alpha,\gamma$ are tunable hyperparameters (Liu et al., 28 Jan 2026).

3. Training Procedure and Algorithmic Integration

During reinforcement fine-tuning, the policy samples $N$ bounding box predictions per instruction, computes their centers and centroid, and evaluates $\mathcal{R}_p$ as above. The exploration incentive is combined with a group-normalized advantage computed via Group Relative Policy Optimization (GRPO): for each prediction $i$ in the group, the advantage is $A_i = \big(r_i - \text{mean}(r)\big)/\text{std}(r)$, where $r_i$ measures standard per-sample ground-truth reward (e.g., IoU or center distance). The final objective for each candidate is

$$J_i = \operatorname{ratio}(\pi_\theta,\pi_\text{ref}) \cdot (A_i + R_{\text{AiF}}) - \beta\,\mathrm{KL}[\pi_\text{ref}\,||\,\pi_\theta],$$

where $\pi_\text{ref}$ is a frozen reference policy and $\beta$ is the KL penalty weight. This objective both stabilizes updates and maintains continual learning progress. Training employs mixed-precision, efficient attention mechanisms, and gradient checkpointing to enable large-scale parallelization. Recommended settings are $N=4$; $\alpha$ in $[1,15]$; $\gamma$ in $[0.1,1]$; and a KL penalty $\beta \approx 0.04$ (Liu et al., 28 Jan 2026).

4. Comparison to Static Reward Schemes

The traditional static point-based reward is defined as $-\|\mathbf{c}^p - \mathbf{c}^\text{gt}\|_2$ or as standard IoU, which inexorably steers the agent to “lock on” to a single ground-truth location. Such over-adapted policies rapidly lose efficacy under domain or resolution shift, the primary context of continual learning. In contrast, APR-iF optimizes for the variance of anchors, actively rewarding outcome diversity. Empirical findings demonstrate that while this could be suboptimal in stationary benchmarks, in fluxing settings it dramatically curbs catastrophic forgetting and enhances out-of-distribution transfer (Liu et al., 28 Jan 2026).

5. Empirical Analysis and Ablation Studies

Experiments on sequential domain-shifted datasets (ScreenSpot-V1/V2) establish that GUI-AiF (full APR-iF and ARR-iF) achieves significant performance improvements versus baselines. Baseline reinforcement fine-tuning without any flux rewards yields average accuracy of 75–76%. Integrating only APR-iF increases accuracy to 76.9% (SSv1) and 80.0% (SSv2); ARR-iF alone yields 75.4% / 77.7%. The combination of APR-iF and ARR-iF gives the highest results: 81.7% / 83.5%. Hyperparameter sweeps confirm that increasing $\alpha$ (the APR-iF weight) most strongly drives performance under domain shift, while $\gamma$ (ARR-iF) tunes robustness to resolution drift. Reward trace plots confirm that APR-iF is a dominant exploration bonus, with high variance and adaptivity during training (Liu et al., 28 Jan 2026).

Reward configuration	ScreenSpot-V1 (%)	ScreenSpot-V2 (%)
Baseline (no APR/ARR)	76	75
+ APR-iF only	76.9	80.0
+ ARR-iF only	75.4	77.7
Full GUI-AiF (APR + ARR)	81.7	83.5

6. Implementation Considerations

The practical deployment of GUI-AiF requires multiple roll-out samples per instance to compute group variance-based rewards. APR-iF is scalable, as the required compute grows linearly with the group size $N$ per instruction. The reference policy and KL-regularization are essential for training stability, especially in a continual learning regime. Efficient hardware utilization via mixed-precision and optimized attention kernels is necessary for large parameter models (e.g., 3B). According to the implementation, a single epoch of reinforcement fine-tuning is often sufficient to observe substantial retention and forward transfer gains across domain progressions (mobile $\rightarrow$ desktop $\rightarrow$ web or 1080p $\rightarrow$ 4K) (Liu et al., 28 Jan 2026).

7. Significance and Context within Continual Learning

GUI-AiF represents the first continual learning framework explicitly tailored for GUI agents operating in non-stationary environments. By departing from precision-centric reward schemes and incentivizing exploration, it demonstrates that RL objectives tuned for variance can outperform conventional targets in sequential, shift-prone UI distributions. A plausible implication is that similar anchoring-in-flux strategies may yield gains in other interaction-rich, evolving data contexts where rigid spatial grounding is a liability (Liu et al., 28 Jan 2026). This framework lays foundational groundwork for robust, future-proof GUI agent design in the context of lifelong learning.