Online VM-Grounded Trajectory Construction (OVTC)

• OVTC is a method that uses instrumented VMs to autonomously collect structured, self-annotated, and milestone-aware GUI interaction trajectories.
• It ensures scalability, diversity, and fidelity by capturing multi-modal data (pixels, DOM, and input logs) across various platforms through an orchestrated VM lifecycle.
• By integrating with PCRM, OVTC delivers dense, stepwise rewards for improved credit assignment and faster convergence in GUI-agent training pipelines.

Searching arXiv for the specified paper and closely related GUI-agent work. Online VM-Grounded Trajectory Construction (OVTC) is a data-generation and supervision mechanism introduced as a core component of Orcust, a framework for interactive GUI tasks that combines Principle-Constrained Reward Modeling (PCRM) with online trajectory construction to improve reasoning reliability and data efficiency (Lu et al., 22 Sep 2025). In this setting, OVTC spins up instrumented virtual machines to autonomously collect structured GUI interaction trajectories with explicit procedural and structural objectives. The collected traces are self-annotated, milestone-aware, and paired with dense, stepwise reward signals, so that downstream training is no longer limited to static, manually labeled traces or coarse sparse rewards at task completion (Lu et al., 22 Sep 2025).

1. Motivation and problem setting

OVTC is motivated by three limitations in existing GUI-agent pipelines: they rely on static, manually labeled traces or coarse sparse rewards at task completion; this fails to capture new bug or failure modes as GUIs evolve; it provides poor credit assignment for long-horizon workflows; and it requires costly human annotation to refresh data (Lu et al., 22 Sep 2025). OVTC is designed to address those limitations by automatically spinning up thousands of lightweight, instrumented VMs, executing the agent policy end-to-end, and collecting self-annotated, milestone-aware trajectories with dense, stepwise reward signals.

The design goals are stated explicitly. Scalability is to generate $O(10^4$–$10^6)$ trajectories without manual effort. Diversity is to cover multiple platforms, including desktop, web, and mobile, as well as multiple application types. Fidelity is to capture pixel (RGB), semantic (DOM), and input-event logs for each step. Fine-grained feedback is to embed self-labeled sub-goals through "[MILESTONE:...]" tokens so that intermediate rewards can be assigned via PCRM (Lu et al., 22 Sep 2025).

Within Orcust, OVTC is therefore not an isolated logging utility. It is the mechanism that creates on-policy, instrumented interaction data in a form suitable for stepwise reward modeling. This suggests that the term “VM-grounded” denotes grounding in an executable, instrumented GUI environment rather than in a static corpus.

2. Architecture and VM lifecycle

The OVTC harness is organized around three subsystems: a VM Orchestrator that starts and stops QEMU/KVM instances, an Instrumentation Layer, and a Task Template Manager (Lu et al., 22 Sep 2025).

Component	Role	Output
VM Orchestrator	Starts / stops QEMU/KVM instances	Running VM instances
Screen Capture Service	Captures GUI state	PNG or raw frame buffer
Input-Event Logger	Records interaction events	mouse/keyboard streams
DOM Snapshotter	Captures semantic state	element tree at each step
Task Template Manager	Supplies high-level tasks and subtask definitions	task templates

The per-VM lifecycle is specified in four stages. During Initialization, OVTC launches a headless QEMU/KVM VM with approximately 200 MB RAM from a golden snapshot, injects an instrumentation agent composed of a shared library and event bus, and loads one task template

$T = (\mathcal{G}=\{G_1,\dots,G_n\}, \phi),$

where each $G_i$ describes a sub-task and $\phi$ denotes success predicates evaluated via DOM or event-bus checks (Lu et al., 22 Sep 2025).

During Rollout, the system captures

$s_t \coloneqq \{RGB_t, DOM_t, InputLog_t\}$

at each time step, sends $s_t$ together with past chain-of-thought to the agent, receives an action $a_t$ and a possible [MILESTONE:] marker, executes the action inside the VM, and records the post-state. The environment-verifiable reward $r^{\mathrm{env}}(s_t,a_t\mid\mathcal{P})$ checks cursor-bounds, widget-state changes, and related deterministic predicates. If the agent emitted a milestone and the relevant predicate $\phi_i(s_{1:t}) \to \mathrm{True}$, OVTC assigns a sub-goal reward (Lu et al., 22 Sep 2025).

During Termination, if the final goal $10^6)$0 is satisfied or the maximum number of steps is reached, OVTC finalizes the trajectory

$10^6)$1

and persists it to the trajectory store. During Teardown, it cleans VM disk state, destroys the instance, and loops to the next template (Lu et al., 22 Sep 2025).

The architectural consequence is that every collected trace is grounded simultaneously in rendered pixels, semantic structure, and actual input execution. A plausible implication is that this multi-channel grounding is what permits deterministic subgoal verification without depending entirely on human relabeling.

3. Procedural and structural objectives

OVTC shapes each trajectory using two objective classes: procedural objectives and structural objectives (Lu et al., 22 Sep 2025).

The procedural component is defined by a sub-task graph. A task template $10^6)$2 specifies a directed acyclic graph of sub-tasks $10^6)$3, and each $10^6)$4 has an associated predicate

$10^6)$5

indicating completion. The procedural constraint is sequential: the agent must satisfy

$10^6)$6

in order (Lu et al., 22 Sep 2025). In the example given in the paper, a sub-task sequence can be “Open File → Export → PDF”.

The structural component is implemented through self-labeled milestones in the agent’s chain-of-thought. At step $10^6)$7, the chain-of-thought $10^6)$8 may emit tokens of the form $\phi$3 These are recognized by OVTC and trigger a positive signal if they align with the corresponding predicate $10^6)$9. The structural constraint is “no skipping”: emitting $T = (\mathcal{G}=\{G_1,\dots,G_n\}, \phi),$0 before $T = (\mathcal{G}=\{G_1,\dots,G_n\}, \phi),$1 is flagged and penalized via PCRM (Lu et al., 22 Sep 2025).

This combination yields dense, intermediate rewards rather than sparse final-only rewards. It also makes the trajectory representation milestone-aware in a way that is explicit rather than latent. A common misunderstanding is to treat milestone tokens as free-form annotations; in Orcust they are operational because they are checked against task predicates and can affect reward assignment.

4. Algorithmic specification and interface with PCRM

The algorithmic form of OVTC is given as “OVTC: Online VM-Grounded Trajectory Construction” with inputs TaskTemplates $T = (\mathcal{G}=\{G_1,\dots,G_n\}, \phi),$2, AgentPolicy $T = (\mathcal{G}=\{G_1,\dots,G_n\}, \phi),$3, and MaxTrajectories $T = (\mathcal{G}=\{G_1,\dots,G_n\}, \phi),$4 (Lu et al., 22 Sep 2025). For each rollout, the algorithm launches a KVM instance from a snapshot, samples a task template, initializes the environment, repeatedly captures state, obtains $T = (\mathcal{G}=\{G_1,\dots,G_n\}, \phi),$5, executes $T = (\mathcal{G}=\{G_1,\dots,G_n\}, \phi),$6, computes an environment-verifiable signal by EVP_Check, adds a milestone reward if MilestoneDetected(c_t) is true, appends the tuple to the trajectory, then tears down the VM and stores the trajectory.

Once OVTC has collected a buffer $T = (\mathcal{G}=\{G_1,\dots,G_n\}, \phi),$7 of trajectories, PCRM scores each trajectory

$T = (\mathcal{G}=\{G_1,\dots,G_n\}, \phi),$8

using the hybrid reward

$T = (\mathcal{G}=\{G_1,\dots,G_n\}, \phi),$9

Here, $G_i$0 comes directly from OVTC’s verifiable checks, including cursor-bounds, DOM changes, and milestone flags, while $G_i$1 is produced by the generative reward model on the chain-of-thought $G_i$2 (Lu et al., 22 Sep 2025).

Training then proceeds with a standard GRPO/RL-style update,

$G_i$3

where $G_i$4 is a baseline (Lu et al., 22 Sep 2025). The division of labor is explicit: OVTC provides the on-policy data, and PCRM computes the dense rewards for each step.

This separation is conceptually important. OVTC does not replace reward modeling; it constrains and enriches reward modeling by ensuring that trajectories arrive with environment-verifiable events and milestone structure. Conversely, PCRM does not generate data; it interprets and critiques data produced online by OVTC.

5. Empirical isolation of OVTC’s contribution

The paper isolates OVTC’s effect through supervised transfer, diversity analyses, reward-convergence analyses, and end-task comparisons (Lu et al., 22 Sep 2025).

Setting	Result	Interpretation stated in the paper
15K OVTC trajectories, SFT, Qwen2.5-VL-3B$G_i$5	+4.8 pts on ScreenSpot over zero-shot	“even without RL,” OVTC data accelerates grounding
15K OVTC trajectories, SFT, Qwen2.5-VL-7B$G_i$6	+6.4 pts	same conclusion
Low diversity, 1K→15K traces	$G_i$7 absolute gain	quantity alone is limited
High diversity, 1K→15K traces	$G_i$8 gain	diversity is the main lever
High Quality + High-Res OVTC traces	average per-step reward to $G_i$9 by 20 training steps	faster reward convergence
Low Quality + Low-Res OVTC traces	only $\phi$0 after 100 steps	lower data quality slows convergence

In supervised transfer on OVTC traces, Qwen2.5-VL-3B$\phi$1 trained with 15K OVTC trajectories via SFT improves grounding on ScreenSpot by +4.8 points over zero-shot, and Qwen2.5-VL-7B$\phi$2 gains +6.4 points. The paper interprets this as evidence that even without RL, OVTC’s self-labeled, diverse data accelerates grounding (Lu et al., 22 Sep 2025).

The diversity ablation is especially direct. Under low diversity, increasing the number of traces from 1K to 15K yields only approximately 1.5% absolute gain in average score. Under high diversity, the same scale-up yields approximately 6.1% gain. The paper therefore states that data diversity, afforded by OVTC sampling many templates, is the main lever for improved generalization (Lu et al., 22 Sep 2025).

The data-quality ablation reaches a similar conclusion. High Quality + High-Res OVTC traces drive the average per-step reward to approximately 90% by 20 training steps, whereas Low Quality + Low-Res reaches only approximately 60% after 100 steps (Lu et al., 22 Sep 2025). This suggests that OVTC’s value depends not only on online generation, but also on the fidelity of the generated traces.

6. Role in Orcust and broader significance

In end-task evaluation, Orcust is reported to achieve state-of-the-art performance, improving by 22.2% on ScreenSpot and 23.9% on ScreenSpot-Pro over the base model, Qwen2.5-VL-7B (Lu et al., 22 Sep 2025). The paper further reports that on GUI-Act/Web and OmniAct, described as low-level tasks, Orcust-7B with OVTC + PCRM achieves 88.1% overall success versus GUI-R1-7B’s 83.3%, a gain of +4.8%. On AndroidControl-High and Odyssey, described as high-level tasks, Orcust-7B reaches 67.8% versus GUI-R1-7B’s 54.7%, a gain of +13.1% (Lu et al., 22 Sep 2025).

These numbers are presented as evidence that OVTC’s automated, diverse, milestone-rich trajectory construction is a major driver behind Orcust’s gains on ScreenSpot and ScreenSpot-Pro. At the same time, the low-level and high-level task results concern the combined system with OVTC + PCRM, so they should be read as evidence for the integrated training loop rather than for OVTC in isolation (Lu et al., 22 Sep 2025).

A frequent conceptual error is to equate OVTC with reward modeling itself. The paper distinguishes the two: OVTC continuously feeds new, self-annotated, instrumented data into the PCRM-guided RL loop, while PCRM computes the dense rewards for each step. Another error is to treat OVTC as merely an online replacement for dataset collection. Its defining contribution is more specific: it constructs GUI trajectories that are simultaneously procedural, structural, environment-verifiable, and milestone-aware. In the formulation of Orcust, that combination is what closes the gap between simulated practice and robust, real-world GUI performance (Lu et al., 22 Sep 2025).