---
title: 'OS-Themis: Scalable GUI Outcome Reward Framework'
url: https://www.emergentmind.com/topics/os-themis
type: topic
---

# OS-Themis: Scalable GUI Outcome Reward Framework

to=arxiv_search.search  北京赛车微信  亚历山大发json
{"query":"ti:\"OS-Themis\" OR abs:\"OS-Themis\"","max_results":10,"sort_by":"submittedDate","sort_order":"descending"}
to=arxiv_search.search  天天中彩票软件ությունները  北京赛车女郎json
{"query":"(\"GUI rewards\" OR \"GUI agent\" OR \"trajectory validation\") AND (critic OR judge OR reward)","max_results":10,"sort_by":"submittedDate","sort_order":"descending"}
to=arxiv_search.search  天天中彩票如何ություններ  鸿丰json
{"query":"cat:cs.LG AND (ti:\"OmniGUIRewardBench\" OR abs:\"OmniGUIRewardBench\")","max_results":10,"sort_by":"submittedDate","sort_order":"descending"}
OS-Themis is a scalable critic framework for generalist GUI rewards that evaluates whether a GUI trajectory has actually completed a task in stochastic environments. It was introduced as a multi-agent alternative to single-judge reward schemes, with the central design choice of decomposing trajectories into verifiable milestones, auditing the resulting evidence chain, and then producing a binary outcome reward \(r \in \{0,1\}\). The framework is paired with OmniGUIRewardBench, a cross-platform benchmark spanning Android, desktop, and web environments, and is evaluated primarily in the context of AndroidWorld, where it improves both online RL and self-training workflows [2603.19191].

## 1. Definition and problem setting

OS-Themis addresses a specific failure mode in GUI-agent training: outcome rewards that are scalable but unreliable. Rule-based rewards are precise but expensive to write and maintain across heterogeneous apps and platforms, while trained critics and single-shot LLM judges often fail under stochastic layouts, sparse trajectory sampling, or long evidence chains. The framework is motivated by the claim that, in GUI environments, false positives are especially damaging for RL because they assign credit to failed trajectories and can therefore push the policy in the wrong direction [2603.19191].

The target object of evaluation is a trajectory
\[
\tau = \{(s_t, a_t, m_t)\}_{t=1}^T,
\]
where \(s_t\) is a screenshot, \(a_t\) is an action, and \(m_t\) is metadata such as the agent’s “think” text or operation description. Given a task instruction \(\mathcal{I}\), OS-Themis evaluates the full trajectory rather than a single action or a sparse set of terminal frames. This suggests a reward model intended for long-horizon, cross-platform GUI control rather than narrowly scripted environments.

A common misconception in this setting is that checking more frames or more steps necessarily improves reward quality. The ablations reported for OS-Themis argue the opposite: dense or indiscriminate verification can dilute decisive evidence with irrelevant transitions, which lowers precision even when overall coverage increases. The framework therefore treats selective evidence extraction as a primary design variable rather than a post hoc optimization [2603.19191].

## 2. Formal reward structure

OS-Themis produces a trajectory-level binary outcome reward. Internally, however, it introduces a structured intermediate representation based on milestones. The Selector first proposes an initial milestone set
\[
\mathcal{M}_0 = \{(t_i, d_i, r_i)\}_{i=1}^k,
\]
where \(t_i\) is a step index, \(d_i\) is an assessment goal, and \(r_i\) is a rationale for why that step is critical. The assessment goal is not incidental; it specifies the exact post-condition to be checked and is a central mechanism for suppressing false positives.

The final reward is produced by the Judge through
\[
r = \mathcal{J}\left(\mathcal{I}, \tau, \{\mathcal{M}_0, \ldots, \mathcal{M}\}, \mathcal{V}, \mathcal{F}\right),
\]
where \(\mathcal{V}\) denotes verification results and \(\mathcal{F}\) denotes reviewer feedback. This is not a simple conjunction over milestone verdicts. The Judge is explicitly allowed to reason over the milestone history, unresolved issues, and the trajectory as a whole before deciding whether the task is `completed`, `not_completed`, or `uncertain`, with the deployed reward mapping collapsing this to \(r \in \{0,1\}\) [2603.19191].

For QA-style GUI tasks, OS-Themis imposes an additional compliance layer. The Judge inspects the last agent answer, verifies whether it is correct, complete, and exactly formatted as requested, and forces `not_completed` if the compliance verdict is `violates`. This makes answer formatting part of the outcome specification rather than a peripheral logging artifact.

## 3. Multi-agent architecture

The architecture is divided into a Milestone Verification Module and a Verdict Calibration Module. All agents are instances of a VLM, but they are assigned distinct roles through prompting.

| Agent | Module | Function |
|---|---|---|
| Selector | MVM | Decomposes trajectory into milestones |
| Verifier | MVM | Checks each milestone with local visual evidence |
| Reviewer | VCM | Audits completeness and strictness of the evidence chain |
| Judge | VCM | Produces the final binary verdict |

The Selector receives \(\mathcal{I}\) and the full trajectory and proposes milestone steps that should be checked. It is instructed to prefer coverage over minimality, avoid selecting the terminate step when no after-image exists, and, in QA tasks, focus on evidence acquisition and final answer production. The Verifier then inspects the before and after screenshots for each selected step, along with the executed action and assessment goal, and returns `success`, `failure`, or `uncertain`, together with explicit evidence cues and optional suggestions for additional checks [2603.19191].

The Reviewer functions as a strict auditor rather than a second verifier. It inspects whether the proposed milestone set and verification history are complete, whether important failure modes remain unchecked, and whether any milestone was framed too leniently. Its feedback is represented as
\[
\mathcal{F} = \{(i_j, q_j)\}_{j=1}^n,
\]
with issue identifiers, risk levels, related steps, and evidence requirements. The Selector and Verifier can then refine the evidence chain in response. The final Judge consumes the task, the verified history, the milestone evolution, and reviewer issues, and returns the trajectory-level decision [2603.19191].

Several implementation constraints are explicitly reported. The default orchestration uses `selector_max_rounds = 6`, `reviewer_max_rounds = 2`, and up to 2 retries for formatting or execution errors. The framework was instantiated with models from the Qwen3-VL family, and the reported ablations indicate that upgrading the Judge or Verifier to Qwen3-VL-235B yields the largest gains, while upgrading the Reviewer mainly increases precision.

## 4. Benchmark and empirical profile

OS-Themis is evaluated on OmniGUIRewardBench, a cross-platform outcome-reward benchmark assembled from AndroidWorld, OSWorld, WindowsAgentArena, macOSArena, and WebArena-Lite-v2. The benchmark contains 1,409 trajectories in total, with 700 positive and 709 negative examples. It covers 27,882 total steps and 9,918 total milestones, which corresponds to approximately 19.8 steps per task, 7.0 milestones per task, and 35.6% of steps being selected as milestones [2603.19191].

The framework is compared primarily against ZeroGUI and DigiRL. Across evaluated models, the paper reports that all evaluated models achieve their best performance under OS-Themis. The aggregated overall metrics are as follows.

| Framework | Accuracy | Precision | Recall | F1 |
|---|---:|---:|---:|---:|
| OS-Themis | 81.6 | 90.9 | 70.4 | 78.7 |
| ZeroGUI | 73.9 | 85.8 | 57.4 | 65.3 |
| DigiRL | 62.8 | 61.3 | 53.5 | 52.5 |

For Qwen3-VL-235B under OS-Themis, the reported overall metrics are Acc 88.0, Prec 92.8, Rec 82.3, and F1 87.2. The evaluation also shows that the advantage of OS-Themis grows as the underlying model becomes stronger, which suggests that the multi-agent evidence structure is not merely compensating for weak base models but is enabling higher-capacity evaluators to use trajectory information more effectively [2603.19191].

The ablation results clarify why the architecture matters. Removing the Selector and verifying every step reduces accuracy from 88.0 to 83.3 and precision from 92.8 to 79.7. Removing the Verifier reduces accuracy to 81.9 and precision to 77.2. Removing the Reviewer yields accuracy 86.9, precision 85.7, and recall 88.4, while removing the Judge and replacing it with a simple milestone conjunction collapses accuracy to 52.5 and recall to 5.0. These results support the framework’s central claim that scalable GUI rewards require both decomposition and post-verification auditing [2603.19191].

## 5. Use in reinforcement learning and self-training

OS-Themis is used in two distinct training roles on AndroidWorld: as an online RL reward and as an offline trajectory filter. In online RL, the reported setup uses GRPO in the Verl framework, Docker-based Android emulators, and automatically synthesized Android tasks. The reward source is swapped while keeping the policy initialization fixed, enabling direct comparison against SEAgent and ZeroGUI [2603.19191].

For Qwen3-VL-4B, the baseline AndroidWorld accuracy is 45.3, compared with 47.8 using SEAgent, 46.1 using ZeroGUI, 50.9 using OS-Themis with Qwen3-VL-8B as critic, and 51.3 using OS-Themis with Qwen3-VL-235B as critic. For Qwen3-VL-8B, the corresponding values are 47.6, 50.0, 51.7, 53.4, and 54.7. In a larger scaling experiment with 1,024 training tasks and Qwen3-VL-235B as the critic backbone, Qwen3-VL-4B reaches 55.6% accuracy on AndroidWorld from a 45.3% baseline, yielding the reported 10.3% improvement when OS-Themis supports online RL training [2603.19191].

In self-training, OS-Themis is used to filter agent-generated trajectories before supervised fine-tuning. The reported raw collection contains 15,110 trajectories, and the filtered subsets are compared against DigiRL-filtered, ZeroGUI-filtered, and unfiltered data. Fine-tuning on unfiltered data hurts performance, whereas OS-Themis-filtered data yields a 6.9% gain for Qwen3-VL-4B and a 5.0% gain for Qwen3-VL-8B. This suggests that OS-Themis functions not only as a reward mechanism but also as a data-quality control layer for trajectory curation [2603.19191].

## 6. Interpretation, misconceptions, and limitations

A central interpretive point in OS-Themis is that high-recall critics are not automatically good RL critics. The appendix formalizes evaluator behavior with recall \(\rho = \Pr(\hat r=1 \mid \tau=g)\) and false-positive rate \(\alpha = \Pr(\hat r=1 \mid \tau=b)\), which yields the induced pseudo-objective
\[
\hat J(\theta) = \alpha + (\rho - \alpha)\, p(\theta).
\]
Under this view, reducing \(\alpha\) is critical once recall is adequate, because false positives directly corrupt the optimization signal. This is consistent with the choice of the Reviewer in `Critic` mode, which trades some recall for substantially better precision [2603.19191].

Another common misconception is that the framework’s value comes from expensive large-model judging alone. The ablations do not support that interpretation. They show that assignment goals, milestone selection, verification, review, and judgment each contribute distinct error-control functions. The paper also reports that OS-Themis can be expensive in raw inference terms: average latency is about 117.6 seconds per trajectory, with about 164,624 prompt tokens, 6,416.8 completion tokens, and 14.1 model calls. A plausible implication is that its practicality depends on asynchronous evaluation and prefix-caching rather than on single-call efficiency [2603.19191].

The reported limitations are correspondingly concrete. Current online RL experiments are bounded by infrastructure scale; the framework presently outputs binary trajectory-level rewards rather than systematic milestone-shaped rewards; and VLM-based critics remain susceptible to distribution mismatch and semantic reward hacking. On AgentRewardBench, for example, the paper reports very high precision but relatively low recall because the benchmark distribution differs from the long-horizon GUI trajectories OS-Themis is designed for. Even so, within its intended domain, OS-Themis establishes a specific template for generalist GUI outcome rewards: decomposition into verifiable milestones, explicit auditing of evidence completeness, and final reward synthesis from a reviewed evidence chain rather than from a single global judgment [2603.19191].

Source: https://www.emergentmind.com/topics/os-themis