---
title: Online VM-Grounded Trajectory Construction (OVTC)
url: https://www.emergentmind.com/topics/online-vm-grounded-trajectory-construction-ovtc
type: topic
---

# Online VM-Grounded Trajectory Construction (OVTC)

Searching arXiv for the specified paper and closely related GUI-agent work.
Online VM-Grounded Trajectory Construction (OVTC) is a data-generation and supervision mechanism introduced as a core component of Orcust, a framework for interactive GUI tasks that combines Principle-Constrained Reward Modeling (PCRM) with online trajectory construction to improve reasoning reliability and data efficiency [2509.17917]. In this setting, OVTC spins up instrumented virtual machines to autonomously collect structured GUI interaction trajectories with explicit procedural and structural objectives. The collected traces are self-annotated, milestone-aware, and paired with dense, stepwise reward signals, so that downstream training is no longer limited to static, manually labeled traces or coarse sparse rewards at task completion [2509.17917].

## 1. Motivation and problem setting

OVTC is motivated by three limitations in existing GUI-agent pipelines: they rely on static, manually labeled traces or coarse sparse rewards at task completion; this fails to capture new bug or failure modes as GUIs evolve; it provides poor credit assignment for long-horizon workflows; and it requires costly human annotation to refresh data [2509.17917]. OVTC is designed to address those limitations by automatically spinning up thousands of lightweight, instrumented VMs, executing the agent policy end-to-end, and collecting self-annotated, milestone-aware trajectories with dense, stepwise reward signals.

The design goals are stated explicitly. **Scalability** is to generate $O(10^4$–$10^6)$ trajectories without manual effort. **Diversity** is to cover multiple platforms, including desktop, web, and mobile, as well as multiple application types. **Fidelity** is to capture pixel (RGB), semantic (DOM), and input-event logs for each step. **Fine-grained feedback** is to embed self-labeled sub-goals through `"[MILESTONE:...]"` tokens so that intermediate rewards can be assigned via PCRM [2509.17917].

Within Orcust, OVTC is therefore not an isolated logging utility. It is the mechanism that creates on-policy, instrumented interaction data in a form suitable for stepwise reward modeling. This suggests that the term “VM-grounded” denotes grounding in an executable, instrumented GUI environment rather than in a static corpus.

## 2. Architecture and VM lifecycle

The OVTC harness is organized around three subsystems: a VM Orchestrator that starts and stops QEMU/KVM instances, an Instrumentation Layer, and a Task Template Manager [2509.17917].

| Component | Role | Output |
|---|---|---|
| VM Orchestrator | Starts / stops QEMU/KVM instances | Running VM instances |
| Screen Capture Service | Captures GUI state | PNG or raw frame buffer |
| Input-Event Logger | Records interaction events | mouse/keyboard streams |
| DOM Snapshotter | Captures semantic state | element tree at each step |
| Task Template Manager | Supplies high-level tasks and subtask definitions | task templates |

The per-VM lifecycle is specified in four stages. During **Initialization**, OVTC launches a headless QEMU/KVM VM with approximately 200 MB RAM from a golden snapshot, injects an instrumentation agent composed of a shared library and event bus, and loads one task template
$$
T = (\mathcal{G}=\{G_1,\dots,G_n\}, \phi),
$$
where each $G_i$ describes a sub-task and $\phi$ denotes success predicates evaluated via DOM or event-bus checks [2509.17917].

During **Rollout**, the system captures
$$
s_t \coloneqq \{RGB_t, DOM_t, InputLog_t\}
$$
at each time step, sends $s_t$ together with past chain-of-thought to the agent, receives an action $a_t$ and a possible `[MILESTONE:]` marker, executes the action inside the VM, and records the post-state. The environment-verifiable reward $r^{\mathrm{env}}(s_t,a_t\mid\mathcal{P})$ checks cursor-bounds, widget-state changes, and related deterministic predicates. If the agent emitted a milestone and the relevant predicate $\phi_i(s_{1:t}) \to \mathrm{True}$, OVTC assigns a sub-goal reward [2509.17917].

During **Termination**, if the final goal $\phi(s_{1:t})$ is satisfied or the maximum number of steps is reached, OVTC finalizes the trajectory
$$
\tau = \{(s_1,a_1,c_1,r_1), \dots, (s_t,a_t,c_t,r_t)\}
$$
and persists it to the trajectory store. During **Teardown**, it cleans VM disk state, destroys the instance, and loops to the next template [2509.17917].

The architectural consequence is that every collected trace is grounded simultaneously in rendered pixels, semantic structure, and actual input execution. A plausible implication is that this multi-channel grounding is what permits deterministic subgoal verification without depending entirely on human relabeling.

## 3. Procedural and structural objectives

OVTC shapes each trajectory using two objective classes: **procedural objectives** and **structural objectives** [2509.17917].

The procedural component is defined by a sub-task graph. A task template $T$ specifies a directed acyclic graph of sub-tasks $G_i$, and each $G_i$ has an associated predicate
$$
\phi_i(s_{1:t}) \in \{0,1\}
$$
indicating completion. The procedural constraint is sequential: the agent must satisfy
$$
\phi_1 \rightarrow \phi_2 \rightarrow \cdots \rightarrow \phi_k
$$
in order [2509.17917]. In the example given in the paper, a sub-task sequence can be “Open File → Export → PDF”.

The structural component is implemented through self-labeled milestones in the agent’s chain-of-thought. At step $t$, the chain-of-thought $c_t$ may emit tokens of the form
```text
[MILESTONE: name]
```
These are recognized by OVTC and trigger a positive signal if they align with the corresponding predicate $\phi_i$. The structural constraint is “no skipping”: emitting $\phi_j$ before $\phi_i$ is flagged and penalized via PCRM [2509.17917].

This combination yields dense, intermediate rewards rather than sparse final-only rewards. It also makes the trajectory representation milestone-aware in a way that is explicit rather than latent. A common misunderstanding is to treat milestone tokens as free-form annotations; in Orcust they are operational because they are checked against task predicates and can affect reward assignment.

## 4. Algorithmic specification and interface with PCRM

The algorithmic form of OVTC is given as “OVTC: Online VM-Grounded Trajectory Construction” with inputs `TaskTemplates` $\mathcal{T}$, `AgentPolicy` $\pi_\theta$, and `MaxTrajectories` $N$ [2509.17917]. For each rollout, the algorithm launches a KVM instance from a snapshot, samples a task template, initializes the environment, repeatedly captures state, obtains $(c_t,a_t)\leftarrow \pi_\theta(s_t)$, executes $a_t$, computes an environment-verifiable signal by `EVP_Check`, adds a milestone reward if `MilestoneDetected(c_t)` is true, appends the tuple to the trajectory, then tears down the VM and stores the trajectory.

Once OVTC has collected a buffer $\mathcal{D}$ of trajectories, PCRM scores each trajectory
$$
\tau = \{(s_t,a_t,c_t)\}_{t=1}^{T}
$$
using the hybrid reward
$$
R(\tau)=\sum_{t=1}^{T}\Bigl[r^{\mathrm{env}}(s_t,a_t\mid\mathcal{P}) + r^{\mathrm{critique}}_\theta(c_t\mid\mathcal{P})\Bigr].
$$
Here, $r^{\mathrm{env}}$ comes directly from OVTC’s verifiable checks, including cursor-bounds, DOM changes, and milestone flags, while $r^{\mathrm{critique}}_\theta$ is produced by the generative reward model on the chain-of-thought $c_t$ [2509.17917].

Training then proceeds with a standard GRPO/RL-style update,
$$
\mathcal{L}(\theta)=-\mathbb{E}_{\tau\sim\pi_\theta}[A(\tau)], \qquad
A(\tau)=R(\tau)-\widehat b(\mathrm{group}),
$$
where $\widehat b$ is a baseline [2509.17917]. The division of labor is explicit: OVTC provides the on-policy data, and PCRM computes the dense rewards for each step.

This separation is conceptually important. OVTC does not replace reward modeling; it constrains and enriches reward modeling by ensuring that trajectories arrive with environment-verifiable events and milestone structure. Conversely, PCRM does not generate data; it interprets and critiques data produced online by OVTC.

## 5. Empirical isolation of OVTC’s contribution

The paper isolates OVTC’s effect through supervised transfer, diversity analyses, reward-convergence analyses, and end-task comparisons [2509.17917].

| Setting | Result | Interpretation stated in the paper |
|---|---|---|
| 15K OVTC trajectories, SFT, Qwen2.5-VL-3B$^*$ | +4.8 pts on ScreenSpot over zero-shot | “even without RL,” OVTC data accelerates grounding |
| 15K OVTC trajectories, SFT, Qwen2.5-VL-7B$^*$ | +6.4 pts | same conclusion |
| Low diversity, 1K→15K traces | $\sim 1.5\%$ absolute gain | quantity alone is limited |
| High diversity, 1K→15K traces | $\sim 6.1\%$ gain | diversity is the main lever |
| High Quality + High-Res OVTC traces | average per-step reward to $\sim 90\%$ by 20 training steps | faster reward convergence |
| Low Quality + Low-Res OVTC traces | only $\sim 60\%$ after 100 steps | lower data quality slows convergence |

In supervised transfer on OVTC traces, Qwen2.5-VL-3B$^*$ trained with 15K OVTC trajectories via SFT improves grounding on ScreenSpot by +4.8 points over zero-shot, and Qwen2.5-VL-7B$^*$ gains +6.4 points. The paper interprets this as evidence that even without RL, OVTC’s self-labeled, diverse data accelerates grounding [2509.17917].

The diversity ablation is especially direct. Under low diversity, increasing the number of traces from 1K to 15K yields only approximately 1.5% absolute gain in average score. Under high diversity, the same scale-up yields approximately 6.1% gain. The paper therefore states that data diversity, afforded by OVTC sampling many templates, is the main lever for improved generalization [2509.17917].

The data-quality ablation reaches a similar conclusion. High Quality + High-Res OVTC traces drive the average per-step reward to approximately 90% by 20 training steps, whereas Low Quality + Low-Res reaches only approximately 60% after 100 steps [2509.17917]. This suggests that OVTC’s value depends not only on online generation, but also on the fidelity of the generated traces.

## 6. Role in Orcust and broader significance

In end-task evaluation, Orcust is reported to achieve state-of-the-art performance, improving by 22.2% on ScreenSpot and 23.9% on ScreenSpot-Pro over the base model, Qwen2.5-VL-7B [2509.17917]. The paper further reports that on GUI-Act/Web and OmniAct, described as low-level tasks, Orcust-7B with OVTC + PCRM achieves 88.1% overall success versus GUI-R1-7B’s 83.3%, a gain of +4.8%. On AndroidControl-High and Odyssey, described as high-level tasks, Orcust-7B reaches 67.8% versus GUI-R1-7B’s 54.7%, a gain of +13.1% [2509.17917].

These numbers are presented as evidence that OVTC’s automated, diverse, milestone-rich trajectory construction is a major driver behind Orcust’s gains on ScreenSpot and ScreenSpot-Pro. At the same time, the low-level and high-level task results concern the combined system with OVTC + PCRM, so they should be read as evidence for the integrated training loop rather than for OVTC in isolation [2509.17917].

A frequent conceptual error is to equate OVTC with reward modeling itself. The paper distinguishes the two: OVTC continuously feeds new, self-annotated, instrumented data into the PCRM-guided RL loop, while PCRM computes the dense rewards for each step. Another error is to treat OVTC as merely an online replacement for dataset collection. Its defining contribution is more specific: it constructs GUI trajectories that are simultaneously procedural, structural, environment-verifiable, and milestone-aware. In the formulation of Orcust, that combination is what closes the gap between simulated practice and robust, real-world GUI performance [2509.17917].

Source: https://www.emergentmind.com/topics/online-vm-grounded-trajectory-construction-ovtc