OS-Themis: Scalable GUI Outcome Reward Framework

• OS-Themis is a scalable multi-agent framework that evaluates GUI trajectories by decomposing them into verifiable milestones to produce binary outcome rewards.
• It employs a four-agent architecture—Selector, Verifier, Reviewer, and Judge—to robustly audit evidence and mitigate false positives in stochastic environments.
• Evaluated on OmniGUIRewardBench, OS-Themis enhances online RL and self-training workflows, achieving higher accuracy and precision compared to single-judge schemes.

to=arxiv_search.search 北京赛车微信 亚历山大发json {"^^^^^^^^3query3 OR abs:\3^^^^^^^^"OS-Themis\"","max_results":^^^^3ti:\3query3 to=arxiv_search.search 天天中彩票软件ությունները 北京赛车女郎json {"^^^^^^^^3query3 rewards\" OR \"GUI agent\" OR \"trajectory validation\") AND (critic OR judge OR reward)","max_results":^^^^3ti:\3query3^^^^,"sort_by":"submittedDate","sort_order":"descending"} to=arxiv_search.search 天天中彩票如何ություններ 鸿丰json {"^^^^^^^^3query3 AND (^^^^^^^^3ti:\3 OR abs:\3^^^^^^^^"OmniGUIRewardBench\")","max_results":^^^^3ti:\3query3 OS-Themis is a scalable critic framework for generalist GUI rewards that evaluates whether a GUI trajectory has actually completed a task in stochastic environments. It was introduced as a multi-agent alternative to single-judge reward schemes, with the central design choice of decomposing trajectories into verifiable milestones, auditing the resulting evidence chain, and then producing a binary outcome reward PRESERVED_PLACEHOLDER_^^^^3query3^^^^. The framework is paired with OmniGUIRewardBench, a cross-platform benchmark spanning Android, desktop, and web environments, and is evaluated primarily in the context of AndroidWorld, where it improves both online RL and self-training workflows (&&&^^^^3query3^^^^&&&).

^^^^3ti:\3^^^^. Definition and problem setting

OS-Themis addresses a specific failure mode in GUI-agent training: outcome rewards that are scalable but unreliable. Rule-based rewards are precise but expensive to write and maintain across heterogeneous apps and platforms, while trained critics and single-shot LLM judges often fail under stochastic layouts, sparse trajectory sampling, or long evidence chains. The framework is motivated by the claim that, in GUI environments, false positives are especially damaging for RL because they assign credit to failed trajectories and can therefore push the policy in the wrong direction (&&&^^^^3query3^^^^&&&).

The target object of evaluation is a trajectory

PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^

where PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3^^^^ is a screenshot, $a_t$ is an action, and $m_t$ is metadata such as the agent’s “think” text or operation description. Given a task instruction $\mathcal{I}$, OS-Themis evaluates the full trajectory rather than a single action or a sparse set of terminal frames. This suggests a reward model intended for long-horizon, cross-platform GUI control rather than narrowly scripted environments.

A common misconception in this setting is that checking more frames or more steps necessarily improves reward quality. The ablations reported for OS-Themis argue the opposite: dense or indiscriminate verification can dilute decisive evidence with irrelevant transitions, which lowers precision even when overall coverage increases. The framework therefore treats selective evidence extraction as a primary design variable rather than a post hoc optimization (&&&^^^^3query3^^^^&&&).

^^^^3 OR abs:\3^^^^. Formal reward structure

OS-Themis produces a trajectory-level binary outcome reward. Internally, however, it introduces a structured intermediate representation based on milestones. The Selector first proposes an initial milestone set

$\mathcal{M}_0 = \{(t_i, d_i, r_i)\}_{i=1}^k,$

where $t_i$ is a step index, $d_i$ is an assessment goal, and $r_i$ is a rationale for why that step is critical. The assessment goal is not incidental; it specifies the exact post-condition to be checked and is a central mechanism for suppressing false positives.

The final reward is produced by the Judge through

PRESERVED_PLACEHOLDER_^^^^3ti:\3query3^^^^

where PRESERVED_PLACEHOLDER_^^^^3ti:\3ti:\3^^^^ denotes verification results and PRESERVED_PLACEHOLDER_^^^^3ti:\3 OR abs:\3^^^^ denotes reviewer feedback. This is not a simple conjunction over milestone verdicts. The Judge is explicitly allowed to reason over the milestone history, unresolved issues, and the trajectory as a whole before deciding whether the task is completed, not_completed, or uncertain, with the deployed reward mapping collapsing this to PRESERVED_PLACEHOLDER_^^^^3ti:\33^^^^ (&&&^^^^3query3^^^^&&&).

For QA-style GUI tasks, OS-Themis imposes an additional compliance layer. The Judge inspects the last agent answer, verifies whether it is correct, complete, and exactly formatted as requested, and forces not_completed if the compliance verdict is violates. This makes answer formatting part of the outcome specification rather than a peripheral logging artifact.

3. Multi-agent architecture

The architecture is divided into a Milestone Verification Module and a Verdict Calibration Module. All agents are instances of a VLM, but they are assigned distinct roles through prompting.

Agent	Module	Function
Selector	MVM	Decomposes trajectory into milestones
Verifier	MVM	Checks each milestone with local visual evidence
Reviewer	VCM	Audits completeness and strictness of the evidence chain
Judge	VCM	Produces the final binary verdict

The Selector receives PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^4 and the full trajectory and proposes milestone steps that should be checked. It is instructed to prefer coverage over minimality, avoid selecting the terminate step when no after-image exists, and, in QA tasks, focus on evidence acquisition and final answer production. The Verifier then inspects the before and after screenshots for each selected step, along with the executed action and assessment goal, and returns success, failure, or uncertain, together with explicit evidence cues and optional suggestions for additional checks (&&&^^^^3query3^^^^&&&).

The Reviewer functions as a strict auditor rather than a second verifier. It inspects whether the proposed milestone set and verification history are complete, whether important failure modes remain unchecked, and whether any milestone was framed too leniently. Its feedback is represented as

PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^5

with issue identifiers, risk levels, related steps, and evidence requirements. The Selector and Verifier can then refine the evidence chain in response. The final Judge consumes the task, the verified history, the milestone evolution, and reviewer issues, and returns the trajectory-level decision (&&&^^^^3query3^^^^&&&).

Several implementation constraints are explicitly reported. The default orchestration uses selector_max_rounds = 6, reviewer_max_rounds = ^^^^3 OR abs:\3^^^^, and up to ^^^^3 OR abs:\3^^^^ retries for formatting or execution errors. The framework was instantiated with models from the Qwen3-VL family, and the reported ablations indicate that upgrading the Judge or Verifier to Qwen3-VL-^^^^3 OR abs:\33^^^^5B yields the largest gains, while upgrading the Reviewer mainly increases precision.

4. Benchmark and empirical profile

OS-Themis is evaluated on OmniGUIRewardBench, a cross-platform outcome-reward benchmark assembled from AndroidWorld, OSWorld, WindowsAgentArena, macOSArena, and WebArena-Lite-v^^^^3 OR abs:\3^^^^. The benchmark contains ^^^^3ti:\3^^^^,4^^^^3query3 trajectories in total, with 7^^^^3query3query3^^^^ positive and 7^^^^3query3^^^^9 negative examples. It covers ^^^^3 OR abs:\3^^^^7,88^^^^3 OR abs:\3^^^^ total steps and 9,9^^^^3ti:\3^^^^8 total milestones, which corresponds to approximately ^^^^3ti:\3^^^^9.8 steps per task, 7.^^^^3query3^^^^ milestones per task, and 35.6% of steps being selected as milestones (&&&^^^^3query3^^^^&&&).

The framework is compared primarily against ZeroGUI and DigiRL. Across evaluated models, the paper reports that all evaluated models achieve their best performance under OS-Themis. The aggregated overall metrics are as follows.

Framework	Accuracy	Precision	Recall	F^^^^3ti:\3^^^^
OS-Themis	8^^^^3ti:\3^^^^.6	9^^^^3query3^^^^.9	7^^^^3query3^^^^.4	78.7
ZeroGUI	73.9	85.8	57.4	65.3
DigiRL	6^^^^3 OR abs:\3^^^^.8	6^^^^3ti:\3^^^^.3	53.5	5^^^^3 OR abs:\3^^^^.5

For Qwen3-VL-^^^^3 OR abs:\33^^^^5B under OS-Themis, the reported overall metrics are Acc 88.^^^^3query3^^^^, Prec 9^^^^3 OR abs:\3^^^^.8, Rec 8^^^^3 OR abs:\3^^^^.3, and F^^^^3ti:\3^^^^ 87.^^^^3 OR abs:\3^^^^. The evaluation also shows that the advantage of OS-Themis grows as the underlying model becomes stronger, which suggests that the multi-agent evidence structure is not merely compensating for weak base models but is enabling higher-capacity evaluators to use trajectory information more effectively (&&&^^^^3query3^^^^&&&).

The ablation results clarify why the architecture matters. Removing the Selector and verifying every step reduces accuracy from 88.^^^^3query3^^^^ to 83.3 and precision from 9^^^^3 OR abs:\3^^^^.8 to 79.7. Removing the Verifier reduces accuracy to 8^^^^3ti:\3^^^^.9 and precision to 77.^^^^3 OR abs:\3^^^^. Removing the Reviewer yields accuracy 86.9, precision 85.7, and recall 88.4, while removing the Judge and replacing it with a simple milestone conjunction collapses accuracy to 5^^^^3 OR abs:\3^^^^.5 and recall to 5.^^^^3query3^^^^. These results support the framework’s central claim that scalable GUI rewards require both decomposition and post-verification auditing (&&&^^^^3query3^^^^&&&).

5. Use in reinforcement learning and self-training

OS-Themis is used in two distinct training roles on AndroidWorld: as an online RL reward and as an offline trajectory filter. In online RL, the reported setup uses GRPO in the Verl framework, Docker-based Android emulators, and automatically synthesized Android tasks. The reward source is swapped while keeping the policy initialization fixed, enabling direct comparison against SEAgent and ZeroGUI (&&&^^^^3query3^^^^&&&).

For Qwen3-VL-4B, the baseline AndroidWorld accuracy is 45.3, compared with 47.8 using SEAgent, 46.^^^^3ti:\3^^^^ using ZeroGUI, 5^^^^3query3^^^^.9 using OS-Themis with Qwen3-VL-8B as critic, and 5^^^^3ti:\3^^^^.3 using OS-Themis with Qwen3-VL-^^^^3 OR abs:\33^^^^5B as critic. For Qwen3-VL-8B, the corresponding values are 47.6, 5^^^^3query3^^^^.^^^^3query3 5^^^^3ti:\3^^^^.7, 53.4, and 54.7. In a larger scaling experiment with ^^^^3ti:\3^^^^,^^^^3query3 OR abs:\3^^^^4 training tasks and Qwen3-VL-^^^^3 OR abs:\33^^^^5B as the critic backbone, Qwen3-VL-4B reaches 55.6% accuracy on AndroidWorld from a 45.3% baseline, yielding the reported ^^^^3ti:\3query3^^^^.3% improvement when OS-Themis supports online RL training (&&&^^^^3query3^^^^&&&).

In self-training, OS-Themis is used to filter agent-generated trajectories before supervised fine-tuning. The reported raw collection contains ^^^^3ti:\3^^^^5,^^^^3ti:\3ti:\3query3^^^^ trajectories, and the filtered subsets are compared against DigiRL-filtered, ZeroGUI-filtered, and unfiltered data. Fine-tuning on unfiltered data hurts performance, whereas OS-Themis-filtered data yields a 6.9% gain for Qwen3-VL-4B and a 5.^^^^3query3^^^^% gain for Qwen3-VL-8B. This suggests that OS-Themis functions not only as a reward mechanism but also as a data-quality control layer for trajectory curation (&&&^^^^3query3^^^^&&&).

6. Interpretation, misconceptions, and limitations

A central interpretive point in OS-Themis is that high-recall critics are not automatically good RL critics. The appendix formalizes evaluator behavior with recall PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^6 and false-positive rate PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^7, which yields the induced pseudo-objective

PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^8

Under this view, reducing PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^9 is critical once recall is adequate, because false positives directly corrupt the optimization signal. This is consistent with the choice of the Reviewer in Critic mode, which trades some recall for substantially better precision (&&&^^^^3query3^^^^&&&).

Another common misconception is that the framework’s value comes from expensive large-model judging alone. The ablations do not support that interpretation. They show that assignment goals, milestone selection, verification, review, and judgment each contribute distinct error-control functions. The paper also reports that OS-Themis can be expensive in raw inference terms: average latency is about ^^^^3ti:\3ti:\3^^^^7.6 seconds per trajectory, with about ^^^^3ti:\3^^^^64,6^^^^3 OR abs:\3^^^^4 prompt tokens, 6,4^^^^3ti:\3^^^^6.8 completion tokens, and ^^^^3ti:\3^^^^4.^^^^3ti:\3^^^^ model calls. A plausible implication is that its practicality depends on asynchronous evaluation and prefix-caching rather than on single-call efficiency (&&&^^^^3query3^^^^&&&).

The reported limitations are correspondingly concrete. Current online RL experiments are bounded by infrastructure scale; the framework presently outputs binary trajectory-level rewards rather than systematic milestone-shaped rewards; and VLM-based critics remain susceptible to distribution mismatch and semantic reward hacking. On AgentRewardBench, for example, the paper reports very high precision but relatively low recall because the benchmark distribution differs from the long-horizon GUI trajectories OS-Themis is designed for. Even so, within its intended domain, OS-Themis establishes a specific template for generalist GUI outcome rewards: decomposition into verifiable milestones, explicit auditing of evidence completeness, and final reward synthesis from a reviewed evidence chain rather than from a single global judgment (&&&^^^^3query3^^^^&&&).