StainFlow: Entity-Stain Tracking and Evidence Linking for Process Rewards in GUI Agents

Abstract: Reinforcement Learning (RL) has become a promising approach for improving GUI Agents in long-horizon, stochastic digital environments, but trajectory-level success feedback is too sparse to provide reliable credit assignment for intermediate exploration steps. To mitigate this issue, recent studies introduce Process Reward Models (PRMs), which provide finer-grained training feedback through global milestone verification or local step-level evaluation. However, these methods still suffer from two level-specific limitations: global milestone decomposition is subjective and singular, making it difficult to accommodate the multiple valid execution paths in real GUI tasks, while fixed local judging windows may miss long-range key evidence or dilute the decision signal with irrelevant frames. Inspired by stain-tracing mechanisms in network flow analysis, we propose StainFlow, an entity-stain-flow process reward model for GUI Agents. To reduce the subjectivity of global partitioning, we introduce the Global Entity Stain Tracking module, which extracts visually verifiable task entities and tracks how their stain concentrations and states evolve along the trajectory, allowing task phases to be objectively separated by changes in the entity evidence flow. To improve the accuracy of local verification, we introduce the Local Stain Evidence Linking module. Centered on the triggering entities of each candidate key node, it retrieves relevant steps based on their stain concentrations and state changes, and dynamically constructs high-density evidence windows for verifying true key nodes. Extensive experiments on AndroidWorld and OGRBench show that StainFlow relatively improves online RL success by 3.2% and trajectory completion judgment accuracy by 1.8%.

• The paper introduces StainFlow, an entity-centric process reward model that dynamically tracks GUI task entities to provide fine-grained, objective rewards.
• It employs global entity stain tracking combined with adaptive evidence linking to overcome challenges of sparse, delayed rewards in traditional reinforcement learning.
• Empirical results show that StainFlow improves policy performance and trajectory accuracy across multiple platforms, outperforming milestone-based approaches.

StainFlow: Process Reward Modeling for GUI Agents via Entity-Stain Tracking and Evidence Linking

Motivation and Problem Statement

Sparse and delayed trajectory-level reward signals impose significant challenges for GUI agent training in long-horizon, stochastic environments. Traditional reinforcement learning pipelines for GUI agents rely on success/failure feedback at episode termination, impeding efficient credit assignment to intermediate exploration steps. This leads to suboptimal policy learning, especially when detours, redundant actions, or diverse valid solution paths exist. Recent Process Reward Models (PRMs) have attempted to provide denser feedback by decomposing trajectories via global milestone chains or localized stepwise judgments, but these approaches suffer from subjectivity in partitioning, rigidity in context windowing, and insufficient handling of trajectory diversity.

StainFlow: Architectural Framework

StainFlow introduces an entity-centric process reward model that leverages dynamic "entity-stain" signals along GUI task trajectories. The model centers on two modules:

• Global Entity Stain Tracking: StainFlow grounds visually checkable task entities using VLMs. Each entity is assigned a "stain concentration" that reflects its recognition confidence and dynamic state. Stain signals are updated at each step (rise from visibility, decay otherwise), allowing objective partitioning of task stages according to detected evidence flow, rather than static milestones or hand-constructed subgoal chains.
• Local Stain Evidence Linking: For each candidate key node, this module traces triggering entities' stain histories across the trajectory, constructing an adaptive and high-density evidence window. Rather than using a fixed context, evidence may be aggregated from both temporally proximate and distant states linked by entity-centric changes, supporting robust key-node verification against reward dilution and evidence loss.

This architecture yields fine-grained, objective step rewards by fusing a continuous term (weighted mean stain concentration) and a discrete term (verified key-node indicator).

Figure 1: StainFlow dynamically partitions task stages and introduces flexible evidence windows for key-node verification, yielding more accurate step rewards.

Figure 2: The workflow of StainFlow, highlighting its entity-centric evidence organization for reward modeling.

Algorithmic Design

Entity Extraction and Tracking: Entities are extracted based on interface-observable attributes using an auxiliary VLM, ensuring rewards are grounded in visual cues. Per-step, stain concentration is updated by explicit rules considering visibility, recognition confidence, and entity-specific decay ($\gamma_i$). Significant stain magnitude or abrupt changes flag candidate key nodes without predefining static milestones.

Evidence Windowing and Key Node Detection: Local windows for candidate verification are not fixed in length; instead, they are constructed by aggregating all contextually relevant steps tied to entity stain spikes or transitions, extending over nonlocal trajectories when necessary. Only when new, task-relevant entity progress is detected—distinct from previous key nodes—is a candidate accepted, which suppresses overlabeling and noisy rewards.

Reward Assignment: Step reward $\hat{r}_t$ is an affine combination of continuous evidence (stain term) and discrete high-level events (key node term):

$\hat{r}_t = \lambda_r p_t + (1 - \lambda_r) k_t,$

where $p_t$ is the weighted sum of entity stains, $k_t$ is the discrete key-node indicator, and $\lambda_r$ is a tunable mixing parameter.

Empirical Results

Online Reinforcement Learning

Experiments on AndroidWorld with multiple VLM verifiers (Qwen3.5 9B/27B) demonstrate that StainFlow yields superior online RL policy performance relative to stepwise (GUI-Critic-R1) and milestone-based (ADMIRE, OS-Themis) PRMs. In particular:

• Success rate: 62.28% with Qwen3.5-27B verifier, representing a 3.2% absolute gain over OS-Themis.
• Reward distribution: Larger reward separation between successful and failed trajectories (gap = 0.42), with failed runs retaining partial credit rather than receiving uniform penalty or reward saturation.

Scaling verifier strength further amplifies the benefit: stronger models lead to more reliable entity recognition and evidence linking, which translates directly into higher reward quality and policy improvement for StainFlow.

Inference-Time Trajectory Completion

Evaluation on OGRBench across multiple platforms (Ubuntu, Mobile, MacOS, Windows, Web) shows that StainFlow achieves the highest overall accuracy (Acc./F1 = 88.2/88.2 with Gemini-3-Flash), outperforming both trajectory-level baselines and PRMs that rely on strict milestone chains. StainFlow's entity-tracing is robust to variations in interface layout and execution diversity—demonstrated by strong results across all domains.

Ablations and Analysis

Ablation studies support the necessity of both modules:

• Removing Global Entity Stain Tracking or Local Stain Evidence Linking significantly degrades both reward informativeness and policy performance.
• Adaptive, entity-linked evidence windows recover long-range dependencies missed by local or fixed-size approaches; analysis shows required evidence spans often exceed a single-step context, especially in real-world desktop/mobile trajectories.

Hyperparameter ablations indicate optimal ranges for $\eta$ (advantage weight) and $\gamma_i$ (entity decay factors), confirming practical tunability and highlighting the importance of balancing trajectory-level reward with stepwise process evidence.

Figure 3: Tuning the advantage weight $\eta$ and stain decay $\gamma_i$ for persistent and transient entities affects success and stability.

Figure 4: StainFlow's reward learning curves demonstrate improved stability and evidence span coverage compared to baseline PRMs.

Qualitative Examples

Case studies on mobile and desktop GUI tasks illustrate StainFlow's ability to extract entity-stain flows that accurately localize key nodes across diverse operating sequences. Red-border nodes mark high-certainty, evidence-backed actions, supporting both precise credit assignment and interpretable RL supervision.

Figure 5: Example of StainFlow-based staining and reward assignment for a mobile GUI trajectory with identified key nodes.

Figure 6: Example of StainFlow-based staining and reward assignment for a desktop GUI trajectory; red borders signify detected key steps.

Implications and Future Directions

StainFlow establishes an entity-centric paradigm for process reward modeling in GUI agents, addressing both subjectivity in milestone progression and context rigidity in local evaluators. By tying rewards to visual, verifiable entity states and their dynamic transitions, the method generalizes across platforms and task variants. Practical implications include:

• Policy stability and efficiency: Denser, more accurate rewards lead to improved policy learning and sample efficiency, particularly in sparse-reward or multi-solution tasks.
• Interpretability: Entity-stain traces provide interpretable skeletons of agent behavior, facilitating robustness analysis and error diagnosis.
• Scalability: The design supports verifier upgrades and transfer to broader environments, pending further validation on longer training horizons and larger data.

Anticipated theoretical developments include deepening the formal connection between network flow analysis and credit assignment in high-dimensional vision-action loops, and applying StainFlow to non-GUI domains requiring dynamic entity tracking.

Conclusion

StainFlow presents a process reward framework that integrates dynamic visual entity recognition with adaptive evidence relinking for GUI agent RL. The architecture simultaneously reduces the subjectivity in milestone definition and compensates for the myopia of fixed context windows. Empirical results on challenging benchmarks demonstrate consistent performance gains and robust reward separability. Thematically, StainFlow advocates entity-centric, evidence-driven supervision as a foundation for generalist interactive agent training and evaluation.

(2606.07027)