Orcust: Unified RL for GUI Agents

• Orcust is a unified reinforcement-learning framework designed to improve GUI agents by integrating principle-guided reward modeling with online VM-based trajectory generation.
• It combines PCRM and OVTC to provide stepwise, interpretable feedback, addressing challenges like coarse reward signals and static trajectory datasets.
• Quantitative evaluations show significant performance gains over baselines, with improved task success rates and enhanced long-horizon decision making.

Searching arXiv for the Orcust paper and related GUI-agent benchmark papers. Orcust is a unified reinforcement-learning framework designed to endow vision–language GUI agents with both reliable reasoning and data-efficient policy learning. It combines Principle-Constrained Reward Modeling (PCRM) and Online VM-Grounded Trajectory Construction (OVTC) to bridge pixel-level perception, natural-language instruction understanding, and long-horizon decision making in GUI environments. In the reported formulation, these two modules provide stepwise, interpretable reward signals and autonomously generated high-fidelity GUI interaction traces, with the stated goal of addressing coarse or sparse reward signals, static trajectory datasets, and opaque reasoning that cannot be formally verified (Lu et al., 22 Sep 2025).

1. Problem setting and design objective

Orcust is situated in the context of GUI agents that must map screenshots and instructions to executable interaction sequences. The target regime includes perceptual grounding, foundational operations, and end-to-end task execution, all under environment dynamics that can render static datasets stale. Typical GUI agents are described as suffering from coarse or sparse reward signals, such as final success or failure only; static trajectory datasets that do not adapt as environments change; and opaque reasoning that cannot be formally verified (Lu et al., 22 Sep 2025).

The framework’s central design objective is to improve reasoning reliability and data efficiency in interactive GUI tasks. It does so by injecting “principles” into the reward model and by spinning up lightweight, instrumented virtual machines to synthesize subgoal-annotated interaction traces on the fly. The explicit emphasis on safety, correctness, interpretability, and dense feedback indicates that Orcust is not merely an action-prediction system; it is a policy-learning framework intended to support long-horizon decision making under structured constraints (Lu et al., 22 Sep 2025).

This suggests a shift from terminal-outcome supervision toward intermediate constraint satisfaction. A plausible implication is that Orcust treats GUI interaction as a sequential control problem in which reward shaping must remain interpretable and resistant to reward hacking.

2. System composition and operational logic

Orcust comprises two complementary modules: Principle-Constrained Reward Modeling and Online VM-Grounded Trajectory Construction. Their combination is presented as the mechanism by which the framework obtains both reliable reward signals and scalable data generation (Lu et al., 22 Sep 2025).

PCRM injects both human-authored domain rules and LLM-derived heuristics into the reward process. OVTC, by contrast, focuses on trajectory construction through instrumented virtual machines seeded with task templates. Together, they yield dense, stepwise feedback via deterministic checks and LLM-based critiques, while also generating high-fidelity trajectories without manual labeling.

The operational logic is therefore bifurcated. One branch determines how trajectories are evaluated; the other determines how trajectories are produced. This separation is technically consequential because it decouples reward interpretation from data acquisition while still allowing both modules to co-evolve during reinforcement fine-tuning. In the reported system, OVTC continuously produces current-policy rollouts, and PCRM assigns rewards that are tied either to environment-verifiable predicates or to principle-grounded critiques of reasoning (Lu et al., 22 Sep 2025).

This architecture implies that Orcust addresses two recurrent failure modes of GUI agents simultaneously: low-fidelity supervision and low-diversity rollout data. The paper’s framing treats these as coupled problems rather than independent deficiencies.

3. Principle-Constrained Reward Modeling

PCRM decomposes the per-step reward into two orthogonal components: Environment-Verifiable Principle (EVP) reward and LLM-Derived Principle (LDP) reward. The principle set is defined as

$\mathcal{P} \;=\;\{\,p^{\mathrm{human}_1,\dots,p^{\mathrm{human}_m\}\;\cup\;\{\,p^{\mathrm{GPT}_1,\dots,p^{\mathrm{GPT}_n\}\,,$

where each principle $p_i(s_t,a_t)$ is a Boolean predicate, or side-effect check, on the GUI state–action pair (Lu et al., 22 Sep 2025).

Given a trajectory $\tau=\{(s_t,a_t,c_t)\}_{t=1}^T$, where $c_t$ is the agent’s chain-of-thought at step $t$, the total trajectory reward is defined as

$R(\tau)\;=\;\sum_{t=1}^T\Bigl[ r^{\mathrm{env}\bigl(s_t,a_t\mid\mathcal P\bigr) \;+\; r^{\mathrm{critique}\bigl(c_t\mid\mathcal P\bigr) \Bigr]\,.$

In this construction, $r^{\mathrm{env}(s_t,a_t)$ is a deterministic score, with examples including cursor-in-bounds, correct widget clicked, and no crashes. The term $r^{\mathrm{critique}(c_t)\in[1,10]$ is emitted by a generative reward model $r_\theta$, described as a large LLM that critiques the agent’s reasoning against $\mathcal P$. The resulting optimization target is a GRPO objective that maximizes the advantage

$p_i(s_t,a_t)$0

subject to a KL-divergence trust-region constraint,

$p_i(s_t,a_t)$1

The stated rationale is that each term in $p_i(s_t,a_t)$2 is verifiable, giving zero risk of exploitation, while each $p_i(s_t,a_t)$3 score is tied back to explicit principles. The intended outcome is interpretable, stepwise rewards that guide long-horizon reasoning. This formulation is important because it constrains chain-of-thought reasoning and rule-based feedback using a shared principle set rather than treating reward modeling as a purely latent preference-estimation problem (Lu et al., 22 Sep 2025).

A common oversimplification would be to view Orcust as relying only on symbolic checks. The framework explicitly combines deterministic environment verification with generative critique, and the reported ablations distinguish EVP-only, LDP-only, and hybrid reward construction.

4. Online VM-Grounded Trajectory Construction

OVTC deploys a pool of instrumented QEMU/KVM virtual machines. Each VM is seeded with a task template specifying a high-level goal, such as “export document as PDF,” together with a sequence of requisite sub-tasks and their success criteria, such as “open File menu,” “click Export,” and “confirm Save dialog” (Lu et al., 22 Sep 2025).

At runtime, OVTC performs a fixed interaction loop. It spawns VMs and loads a random task template; invokes the current agent policy $p_i(s_t,a_t)$4 in a loop; records at each step the RGB screenshot $p_i(s_t,a_t)$5, the input event, and a DOM snapshot for precise state verification; detects self-labeled subgoals when the agent emits [MILESTONE: …] in its chain-of-thought; and assigns intermediate rewards upon milestone completion or penalties if skipped. The compact pseudocode supplied in the source describes this as a trajectory-generation procedure returning $p_i(s_t,a_t)$6 after repeated observation, action execution, milestone detection, and reward assignment (Lu et al., 22 Sep 2025).

The recorded state includes both rendered pixels and structured DOM state. This dual capture is significant because it allows precise state verification while preserving the screenshot-based observation channel used by the policy. The explicit use of self-labeled milestones also introduces a procedural scaffold into trajectories, making subgoal progress part of the training signal rather than an after-the-fact annotation.

By continuously generating 15 K such trajectories during training, OVTC is reported to provide diverse, up-to-date data without manual labeling. The paper presents this as a remedy for static dataset limitations and as a mechanism for improving data efficiency. A plausible implication is that OVTC serves both as an online data engine and as a controllable environment for reward attribution at subgoal granularity.

5. Training procedure and optimization regime

Orcust’s training proceeds in two phases. The first is Supervised Fine-Tuning (SFT), in which $p_i(s_t,a_t)$7 is initialized on a small curated set of high-quality GUI trajectories comprising 1–5 K samples. The second is Reinforcement Fine-Tuning (RFT) with PCRM and OVTC (Lu et al., 22 Sep 2025).

During RFT, each iteration consists of five steps: running OVTC to collect a minibatch of trajectories $p_i(s_t,a_t)$8; computing per-step rewards $p_i(s_t,a_t)$9 via PCRM; estimating advantages $\tau=\{(s_t,a_t,c_t)\}_{t=1}^T$0; updating $\tau=\{(s_t,a_t,c_t)\}_{t=1}^T$1 by maximizing the clipped surrogate PPO loss

$\tau=\{(s_t,a_t,c_t)\}_{t=1}^T$2

and enforcing

$\tau=\{(s_t,a_t,c_t)\}_{t=1}^T$3

The framework’s training description emphasizes stepwise reward timing. Reward feedback is provided every 1–8 steps, and the ablation results state that 4-step averaging works best. This is presented as a mechanism for improving credit assignment in long-horizon tasks (Lu et al., 22 Sep 2025).

Technically, the training regime combines policy initialization from curated demonstrations with online policy improvement under principle-constrained rewards. This suggests that Orcust does not rely on a purely offline preference dataset or a purely exploratory RL loop; instead, it uses SFT to establish a competent prior and then uses VM-grounded interaction plus stepwise reward decomposition to refine policy behavior.

6. Benchmarks, results, and ablations

The reported evaluation covers ScreenSpot and ScreenSpot-Pro for visual grounding on 23 apps across 5 industries, as well as GUI-Act-Web, OmniAct-Web/Desktop, AndroidControl-Low/High, and GUI-Odyssey. The metrics are type accuracy, grounding (element/coordinate) accuracy, stepwise success rate, and overall success on high-level tasks (Lu et al., 22 Sep 2025).

The main quantitative results position Orcust-7B against the Qwen2.5-VL-7B base model and against GUI-R1-7B on selected suites. On ScreenSpot-Pro, Orcust-7B improves by +23.9 percentage points over the base model; on ScreenSpot, by +22.2 percentage points. On low-level action suites, Orcust-7B achieves 96.4% type, 92.4% grounding, and 87.8% success, compared with 83.3% success for GUI-R1-7B. On high-level tasks, Orcust-7B attains 62.9% success rate on AndroidControl-High and 56.3% on GUI-Odyssey, compared with 51.7% and 38.8% respectively for GUI-R1-7B (Lu et al., 22 Sep 2025).

The ablation study is methodologically important because it clarifies where gains are attributed. EVP+LDP yields +2.7 percentage points on AndroidControl-Low and +2.4 percentage points on AndroidControl-High over EVP-only. Reward step size matters: 4-step averaging outperforms 1-step, 2-step, and 8-step variants. Trajectory diversity is reported to matter more than sheer quantity, with a 5× increase in task variety giving +6 percentage point gains (Lu et al., 22 Sep 2025).

These results are relevant to two common interpretive errors. First, benchmark gains are not presented as arising solely from a larger model, since the paper frames improvements relative to the Qwen2.5-VL-7B base model. Second, the ablations do not support the view that more trajectories alone are sufficient; the reported data emphasize the role of diversity and the hybrid reward design.

7. Strengths, limitations, and extensions

The strengths identified in the source are interpretable, verifiable rewards that prevent silent exploitation; dense, stepwise feedback that improves credit assignment and stability; and dynamic, high-fidelity data generation via OVTC that yields strong generalization (Lu et al., 22 Sep 2025).

The limitations are equally explicit. Orcust depends on simulated VM environments, and real-world GUIs may differ. It has high computational cost because of VM orchestration and LLM critique queries. The framework also raises potential privacy and security concerns when scaling to real user data (Lu et al., 22 Sep 2025).

The proposed extensions are domain adaptation to real UIs via sim-to-real transfer learning, lightweight on-device deployment through model distillation, and augmenting OVTC with adversarial interface generation for robustification. These proposals indicate that the framework is presently optimized for instrumented, simulation-centric settings rather than unconstrained deployment contexts (Lu et al., 22 Sep 2025).

The summary claim in the source is that Orcust represents the first GUI-agent paradigm to marry principle-guided, stepwise reward modeling with scalable VM-grounded trajectory construction, achieving new state-of-the-art performance across a spectrum of GUI interaction challenges. Within the reported evidence, the distinguishing contribution is not a single algorithmic primitive but the integration of principle-constrained reward decomposition, online trajectory construction, and stepwise policy optimization into a single RL pipeline.