VeriOS-Agent: Trustworthy GUI OS Agent

• VeriOS-Agent is a trustworthy OS agent for multimodal GUI applications that queries humans in unreliable scenarios to avoid over-execution.
• It employs a two-stage interleaved training paradigm that decouples scenario recognition from action generation, improving performance by 20.64% in untrustworthy conditions.
• The framework supports mobile, desktop, web, and tablet interfaces, setting a benchmark for safe, query-driven human-agent GUI interaction.

VeriOS-Agent is a trustworthy operating-system agent for multimodal GUI environments that was introduced together with VeriOS, a query-driven human-agent-GUI interaction framework for OS agents operating under untrustworthy conditions. Its central design objective is to prevent over-execution: rather than forcing an agent to act autonomously in every state, VeriOS-Agent is trained to execute actions in normal conditions and to proactively query humans when the environment is unreliable, the instruction is underspecified, or the action is sensitive. The framework targets mobile, desktop, web, and tablet settings, and the paper reports that VeriOS-Agent improves the average step-wise success rate by $20.64\%$ in untrustworthy scenarios over the state of the art, without compromising normal performance (Wu et al., 9 Sep 2025).

1. Problem setting and trust model

In this work, an OS agent is an MLLM-based system that receives a natural-language user instruction and executes the task by interacting with a device’s graphical user interface. The paper places VeriOS-Agent in the setting of GUI action spaces that include click, type, swipe, long press, back, home, wait, and task completion, and treats the conventional policy as

$a_t = M(i, s_t),$

where $i \in \mathcal{I}$ is the user instruction, $s_t \in \mathcal{S}$ is the GUI screenshot at step $t$, and $a_t \in \mathcal{A}$ is the action (Wu et al., 9 Sep 2025).

The paper argues that this formulation is inadequate in realistic deployments because OS agents are commonly trained and evaluated in idealized, trustworthy environments, whereas real GUI settings are often unreliable or ambiguous. VeriOS-Agent is motivated by the need to avoid over-execution in exactly those settings. The paper defines untrustworthy conditions as GUI situations where blind autonomous execution is unreliable or unsafe, and divides them into environment-side and user-side sources. Environment-side sources include environmental anomalies such as pop-ups, disconnections, OS crashes, and interface instability, as well as sensitive actions such as granting location permission. User-side sources include information missing, where the instruction is ambiguous or incomplete, and multiple choices, where several plausible options exist but user preference is unclear (Wu et al., 9 Sep 2025).

This trust model is therefore neither full autonomy nor full human takeover. Instead, it introduces a middle ground in which the agent remains the primary executor but may explicitly request semantic clarification from the human before acting. A plausible implication is that the paper treats trustworthiness as a property of interaction policy rather than only of action accuracy.

2. Query-driven human-agent-GUI interaction

VeriOS is organized around three entities: Human, Agent, and GUI environment. The interaction loop is explicit. The human provides the original task instruction; the agent observes the current GUI screenshot; the agent determines whether the current state is normal or untrustworthy; and then either emits a direct GUI action or emits an ASK action, which is a natural-language query to the human. After the human replies, the agent incorporates that answer into the next decision step (Wu et al., 9 Sep 2025).

The scenario taxonomy used by the paper is as follows.

Abbreviation	Scenario type	Source
EA	environmental anomalies	environment-side
IM	information missing	user-side
SA	sensitive actions	environment-side
MC	multiple choices	user-side
NS	normal scenarios	normal condition

The crucial behavioral rule is asymmetric. If the scenario is judged normal, the agent should act directly. If the scenario is judged untrustworthy, it should ask first and act only after receiving an answer. The paper also imposes an anti-loop rule: once the human has answered, the subsequent action at that time step must not be another ASK action (Wu et al., 9 Sep 2025).

The runtime formalization makes this explicit. At step $t$, the model first predicts scenario type and an initial action: $e_t, a_t^{(0)} = M(i, s_t).$ If $e_t$ is normal, the executed action is

$a_t = a_t^{(0)}.$

If $a_t = M(i, s_t),$0 is untrustworthy, then $a_t = M(i, s_t),$1 must be ASK. After receiving human response $a_t = M(i, s_t),$2, the model reasons again: $a_t = M(i, s_t),$3 If $a_t = M(i, s_t),$4 is non-empty, then $a_t = M(i, s_t),$5 must not be ASK, and the final executed action is

$a_t = M(i, s_t),$6

The framework therefore inserts a learned clarification phase into the policy itself rather than delegating to a confidence threshold or an external fallback rule (Wu et al., 9 Sep 2025).

3. Formalization and the two-stage learning paradigm

The paper represents each benchmark instance as

$a_t = M(i, s_t),$7

where $a_t = M(i, s_t),$8 is the system prompt, $a_t = M(i, s_t),$9 the user instruction, $i \in \mathcal{I}$0 the ground-truth action, $i \in \mathcal{I}$1 the scenario type, $i \in \mathcal{I}$2 the standard ask query, and $i \in \mathcal{I}$3 the standard human answer. For normal scenarios, $i \in \mathcal{I}$4. From each original instance, the authors derive two sub-instances corresponding to two forms of what the paper calls meta-knowledge: scenario/query knowledge and action knowledge (Wu et al., 9 Sep 2025).

The first sub-instance is

$i \in \mathcal{I}$5

This trains the model to infer the scenario type and formulate the query. The second is

$i \in \mathcal{I}$6

This trains the model to use the query-answer pair to produce the executable action. The training sequence is interleaved: $i \in \mathcal{I}$7 The stated rationale is that simple prompt concatenation of query-answer history can hurt normal performance, whereas decoupling and then interleaving the two supervision types preserves the logical link between “recognize and ask” and “use the answer to act” (Wu et al., 9 Sep 2025).

The supervised objective is standard token-level autoregressive training on both sub-instance types. For $i \in \mathcal{I}$8, with target sequence $i \in \mathcal{I}$9,

$s_t \in \mathcal{S}$0

For $s_t \in \mathcal{S}$1, with target sequence $s_t \in \mathcal{S}$2,

$s_t \in \mathcal{S}$3

The overall loss is

$s_t \in \mathcal{S}$4

No reinforcement-learning objective, explicit query-cost term, or external uncertainty estimator is introduced. Query behavior is learned implicitly through the scenario-label and ASK supervision (Wu et al., 9 Sep 2025).

4. Instantiation, benchmark construction, and evaluation protocol

VeriOS-Agent is instantiated as a single fine-tuned multimodal model rather than as a modular multi-agent system. The paper fully fine-tunes both the visual component and the LLM component of Qwen2.5-VL-7B and Qwen2.5-VL-32B, producing VeriOS-Agent-7B and VeriOS-Agent-32B respectively (Wu et al., 9 Sep 2025).

The benchmark introduced with the framework is VeriOS-Bench, a cross-platform dataset of 622 screenshots spanning mobile, desktop, web, and tablet. It contains both untrustworthy and normal scenarios in an approximate 6:4 ratio, and is split 7:3 into training and testing. Experts annotate scenario type, query content, and ground-truth action; for untrustworthy cases, crowdsourcing is used to annotate what the agent should ask and the corresponding human answer or stance (Wu et al., 9 Sep 2025).

The evaluation metric is step-wise success rate, meaning whether the predicted action matches the annotated ground-truth action at the current step. The paper also reports scenario judgment accuracy (SJA). The action-matching tolerances are operationally defined: click and long-press actions allow relative coordinate error up to 14%; type actions require text similarity of at least 80%; other actions require exact match (Wu et al., 9 Sep 2025).

The baseline set is broad and includes 13 open and proprietary models or agents: UI-TARS-2B-SFT, Qwen2.5-VL-3B, Qwen2.5-VL-7B, OS-Atlas-Pro-7B, UI-TARS-7B-SFT, UI-TARS-7B-DPO, UI-TARS-1.5-7B, Qwen2.5-VL-32B, GUI-owl-32B, Qwen2.5-VL-72B, GPT-4o, Qwen-VL-max, and GLM-4.5V. For GPT-4o, the paper adds an OCR-assisted localization model because it cannot natively predict coordinates (Wu et al., 9 Sep 2025).

Training uses 8 A800 GPUs, maximum image resolution of 262144 pixels, batch size 1 per GPU, and two epochs. The reported learning rates are $s_t \in \mathcal{S}$5 for the 7B model and $s_t \in \mathcal{S}$6 for the 32B model (Wu et al., 9 Sep 2025).

5. Empirical performance and analytical findings

The paper first reports a pilot finding that conventional OS agents are poor at recognizing untrustworthy scenarios: scenario judgment accuracy ranges from 46.52% to 60.96%, whereas VeriOS-Agent improves this to 80.21% for the 7B model in the pilot discussion. It also shows that naively appending correct query-answer history to the prompt can hurt normal-scenario performance: for example, Qwen2.5-VL-7B drops from 37.80 to 30.49, Qwen2.5-VL-3B drops from 3.66 to 2.44, and Qwen2.5-VL-72B drops from 71.95 to 70.73 (Wu et al., 9 Sep 2025).

The main reported results for VeriOS-Agent are summarized below.

Model	MC	IM	EA	SA	NS	Total
VeriOS-Agent-32B	45.24	71.43	81.82	92.95	69.51	68.98
VeriOS-Agent-7B	45.24	71.43	77.27	81.48	47.56	57.22

The paper’s headline claim is that VeriOS-Agent improves the average step-wise success rate in untrustworthy scenarios by 20.64% over the strongest baseline, while preserving normal performance. The gains are concentrated in exactly the categories where proactive querying is intended to matter most: multiple choices, information missing, environment anomalies, and sensitive actions (Wu et al., 9 Sep 2025).

The ablation studies clarify the role of architecture and training design. In the single-agent versus dual-agent comparison, the unified model is competitive or superior, especially at larger scale. For 32B, the single-agent model reaches 68.98 total and 74.87 SJA, whereas the dual-agent setup reaches 65.78 total and 70.05 SJA. For 7B, the single-agent model reaches 57.22 total and 80.21 SJA, while the dual-agent setup reaches 58.29 total and 77.54 SJA. The paper interprets this as evidence that scenario judgment, query generation, and action generation are logically coupled enough that a unified model can internalize them effectively (Wu et al., 9 Sep 2025).

Training-order ablations are even more direct. For 7B, interleaved training yields 57.22, compared with 41.18 for shuffled, 25.67 for rotating, and 30.48 for phased training. For 32B, interleaving also gives the best result at 68.98, compared with 66.31, 66.84, and 55.61 for shuffled, rotating, and phased training. This is the main empirical support for the two-stage decoupling-plus-interleaving design (Wu et al., 9 Sep 2025).

The paper also reports two forms of generalization analysis. First, when one untrustworthy scenario type is removed from training and reserved for testing, VeriOS-Agent remains better than the base Qwen2.5-VL line on all metrics, suggesting that the learned interaction logic transfers across scenario types. Second, in a continual-extension setting that adds a new scenario type later, performance remains close to the full-model result: 70.05, 67.38, 68.45, and 67.91 depending on which scenario type is held out and then added later, versus 68.98 for the full VeriOS-Agent-32B. The paper presents this as evidence of scalability and some resistance to catastrophic forgetting (Wu et al., 9 Sep 2025).

6. Position within OS-agent research

VeriOS-Agent occupies a distinctive place in the emerging OS-agent literature because its trust mechanism is query-driven clarification rather than verifier ranking or post-hoc environment inspection. Later work makes this position clearer by contrast. ColorAgent incorporates both personalized user intent recognition and proactive engagement, explicitly evaluating on VeriOS-Bench and reporting 68.98 on that benchmark; its details state that proactive engagement is based on related VeriOS work (Li et al., 22 Oct 2025). This suggests that VeriOS-Agent established both a benchmark and an interaction pattern that subsequent mobile OS-agent systems reused.

Two adjacent lines of research pursue different control principles. V-Droid uses LLMs as verifiers to evaluate candidate GUI actions before final selection, rather than relying on direct autoregressive action generation (Dai et al., 20 Mar 2025). VAGEN treats GUI verification as an interactive post-hoc problem, introducing a verifier agent that probes the environment online to determine whether a task was actually completed (Cui et al., 31 Jan 2026). Relative to those systems, VeriOS-Agent is centered on pre-action human clarification rather than candidate-action verification or post-execution verification. This suggests that the broader design space for trustworthy OS agents is already differentiating into at least three mechanisms: query-before-action, verifier-driven action selection, and agentic execution verification.

The limitations of VeriOS-Agent are also explicit. The framework relies on annotated scenario types and query-answer pairs; evaluation is step-wise rather than long-horizon end-to-end; the trade-off between autonomy and intervention is not modeled with an explicit query budget or latency cost; and the system assumes human availability when queries are issued (Wu et al., 9 Sep 2025). A plausible implication is that VeriOS-Agent is best understood as an interaction-policy advance rather than a full verification framework. It improves trustworthiness by deciding when to ask, but it does not provide formal action contracts, rollback semantics, or proof-carrying execution. Within the OS-agent literature, its enduring contribution is the claim that trustworthy GUI automation can be organized around a simple but strong rule: act in trustworthy conditions, ask in untrustworthy ones (Wu et al., 9 Sep 2025).