The Art of Building Verifiers for Computer Use Agents

Abstract: Verifying the success of computer use agent (CUA) trajectories is a critical challenge: without reliable verification, neither evaluation nor training signal can be trusted. In this paper, we present lessons learned from building a best-in-class verifier for web tasks we call the Universal Verifier. We design the Universal Verifier around four key principles: 1) constructing rubrics with meaningful, non-overlapping criteria to reduce noise; 2) separating process and outcome rewards that yield complementary signals, capturing cases where an agent follows the right steps but gets blocked or succeeds through an unexpected path; 3) distinguishing between controllable and uncontrollable failures scored via a cascading-error-free strategy for finer-grained failure understanding; and 4) a divide-and-conquer context management scheme that attends to all screenshots in a trajectory, improving reliability on longer task horizons. We validate these findings on CUAVerifierBench, a new set of CUA trajectories with both process and outcome human labels, showing that our Universal Verifier agrees with humans as often as humans agree with each other. We report a reduction in false positive rates to near zero compared to baselines like WebVoyager ($\geq$ 45\%) and WebJudge ($\geq$ 22\%). We emphasize that these gains stem from the cumulative effect of the design choices above. We also find that an auto-research agent achieves 70\% of expert quality in 5\% of the time, but fails to discover all strategies required to replicate the Universal Verifier. We open-source our Universal Verifier system along with CUAVerifierBench; available at https://github.com/microsoft/fara.

• The paper introduces the Universal Verifier (UV) to rigorously score computer use agents through a non-overlapping, task-specific rubric and separate process/outcome rewards.
• UV utilizes multimodal evidence aggregation and top-K screenshot selection to precisely detect hallucinations and attribute errors to controllable versus uncontrollable sources.
• Empirical evaluations show UV’s superior performance with significantly reduced false positives and higher agreement with human annotators compared to existing baselines.

Building Reliable Web Agent Trajectory Verifiers: The Universal Verifier Approach

Motivation and Problem Setting

Assessing the success of computer use agents (CUAs) on semi-structured or complex web tasks is non-trivial due to the multimodal, extended, and ambiguous nature of real-world interaction trajectories. Robust, scalable trajectory verifiers are necessary for both trustworthy evaluation and reinforcement-learning-based improvement of web/GUI agents; yet, most extant baselines either rely on oversimplified assumptions (e.g., outcome-only reward, last-observation grounding, weak rubrics, or excessive reliance on hallucination-prone models) or conflate process and outcome, inducing noisy signal and reward hacking.

This paper introduces the Universal Verifier (UV), constructed via a set of stringent, empirically validated design principles, each directly motivated by observed failure modes in prior work. The design and analysis are underpinned by the establishment of CUAVerifierBench, a new human-annotated benchmark containing both process and outcome ground-truths for CUA-generated web interaction trajectories.

Key Principles for Reliable Web Task Verification

Specific and Non-Overlapping Criterion Rubric Design

A reliable verifier begins with a task-grounded, non-overlapping multi-criteria rubric. Empirical evidence demonstrates that LLM-driven rubric generation frequently injects "phantom" criteria unanchored in the original user intent or, conversely, produces logically entangled rubrics that cascade errors through scoring. Iterative human-in-the-loop refinement is essential for consistently isolating true failures versus rubric artifacts.

The UV enforces a clear separation between rubric generation (conditioned on the task but not trajectory) and rubric scoring (conditioning on full trajectory), sidestepping confirmation bias and overfitting. Additional mechanisms include conditional criteria, which are only activated if the environment makes them relevant, and two-pass scoring (action-history only vs. full-multimodal context) for robust hallucination detection.

Figure 1: An instance of the Universal Verifier’s criterion-level scoring flagging an agent’s error in OCR/transcription, exemplifying pinpoint hallucination detection.

Process vs. Outcome Decomposition

UV formalizes process and outcome rewards as separate, complementary signals. The process score is a normalized sum over rubric criteria (0.0–1.0), reflecting the agent’s execution quality, agnostic to uncontrollable environment stochasticity. The outcome reward is binary, capturing the end-user’s primary success intent. This separation is crucial: many benchmarks and prior verifiers conflate the two, masking reward signals due to either spurious failures (environment blockage yields outcome fail with perfect process) or reward hacking (unexpected but successful environment exploits).

Controllable Versus Uncontrollable Failure Attribution

Scoring is further partitioned according to controllability: the rubric and error taxonomy distinguish between avoidable (agent) and unavoidable (environment) failures at fine granularity. This enables process reward to be robust against exogenous blockers (e.g., CAPTCHAs, content unavailability), while outcome reward remains stringent.

Context Management via Per-Criterion Multimodal Relevance

Unlike prior baselines which feed all trajectory screenshots to the LLM (yielding context window overflows and needle-in-haystack failures) or consider only the last frames (missing critical transitions), UV scores screenshot relevance against each rubric criterion, then forwards only the top-K for each criterion to the scoring LLM. This design is computationally efficient, scalable to arbitrarily long trajectories, and substantially improves detection of subtle errors or hallucinations.

Figure 2: A relevance matrix showing how trajectory screenshots are scored with respect to rubric criteria, capturing trajectory progress and error localization.

Universal Verifier System: Pipeline and Implementation

The UV instantiates these principles in a systematic multi-step pipeline:

1. Rubric Generation: Generate per-task rubrics with conditional, specific, and semantically aligned criteria
2. Screenshot-Criterion Matrix: Compute relevance between every screenshot and every criterion
3. Top-K Selection: Prune to the most diagnostic screenshot per criterion
4. Evidence Extraction and Conditional Updates: Holistically analyze and resolve conditional criteria
5. Rubric Scoring: Attribute point deductions/additions rigorously, identifying hallucinations via cross-modal contradiction detection
6. Side Effect Pass: Detect unsolicited or off-intent agent side effects not directly anticipated by rubric
7. Outcome Aggregation: Infer binary end-state by majority on rubric, with explicit user-centric "does this fulfill primary intent?"
8. Failure Taxonomy: Attach a localized, taxonomy-anchored explanation to every failure point

Figure 3: Visualization of UV's rubric-based results for the complex web task "find the best men's face wash according to GQ or Men's Health, then buy it on Amazon".

Quantitative Evaluation

The UV is empirically compared to strong baselines (WebVoyager, WebJudge, and AgentRewardBench verifiers) on both internally annotated and third-party human-labeled datasets. Main findings:

• UV achieves outcome Cohen’s $\kappa$ of 0.64/0.58 (internal/Browserbase benchmarks) compared to WebJudge (0.44/0.26) and WebVoyager (0.31/0.13).
• False positive rates (FPR) on outcome: UV (0.01/0.08) vs. WebVoyager ($0.45/0.60$) and WebJudge ($0.22/0.40$).
• Inter-annotator agreement analysis establishes that UV’s alignment with human labels matches or exceeds human-human agreement, with median annotator scores achieving Pearson $r=0.61$ with UV predictions.

Notably, backbone model upgrades (e.g. using GPT-5.2 in place of GPT-4o for baselines) only modestly improve $\kappa$ and do not close the gap to UV, demonstrating that gains are fundamentally architectural, not a side effect of model scaling.

Figure 4: Scatter plot of UV rubric scores versus human annotator scores on the Browserbase OM2W dataset; strong monotonic agreement is observed, mirroring human-human agreement levels.

Figure 5: Outcome false positive and false negative rates across successive design iterations, showing reduction to near zero with UV’s pipeline.

Auto-Research: Human-Expertise vs. Automated Prompt/Verifier Optimization

An auto-research agent was deployed to attempt UV prompt/code design from scratch ("blank slate") or as an optimizer atop the best human-constructed variant. Results:

• Auto-research from blank achieves 70% of human expert’s $\kappa$ in $<5\%$ of annotation/iteration cost/time
• Qualitatively, human experts make critical structural changes (rubric decomposition, overhaul of hallucination detection, error taxonomy expansion) that drive $\kappa$ improvements, while the auto-research agent is conservative/incremental (tuning thresholds, adding specific rules)
• When seeded from the best human UV, the auto-research agent can optimize further, but cannot discover architectural advances ex nihilo.

Figure 6: Comparison of $\kappa$ agreement (with human label) as the human expert and auto-research agent iterate over design; step-changes coincide with the integration of the UV’s principles, which the auto-research agent independently fails to rediscover.

Failure Taxonomy and Error Analysis

The UV’s fine-grained diagnostic output enables mapping errors not only to broad categories (Selection, Hallucination, Execution, Critical Point, Side Effects), but also to specific steps in the agent trajectory. This taxonomy supports transparent diagnosis, robust benchmarking, and error-driven agent improvement pipelines.

Figure 7: An example where UV correctly detects that a reasoning error (misidentification of longest last name) does not cascade into incorrect scoring of subsequent (downstream) criteria.

Figure 8: Subtle hallucination case where UV detects a contrived metric in a model description not grounded in the agent’s observed evidence.

Qualitative Insights and Annotator Feedback

A two-stage human annotation protocol (UV-blind and UV-informed) provides insight into UV’s utility for benchmarking and training. Annotators revised 16.6% of their outcome labels after seeing UV’s reasoning, most flips being false positives (agent marked as success) found by UV. Representative human feedback highlights that UV’s structured, explainable verdicts surface errors both in the "last mile" (e.g., not delivering final answer) and subtle overclaiming/hallucination cases missed in naive manual review.

Practical and Theoretical Implications

Practically, the UV system and CUAVerifierBench establish new standards for generic, reliable evaluation of agentic interaction models, robust to length, context, and environmental noise. Detecting hallucinations, side-effects, and process/outcome divergence, UV minimizes reward hacking and training set pollution. The open-sourcing of the codebase further promulgates fair comparison and advances in robust real-world agent evaluation.

Theoretically, the necessity of explicit rubric design, context partitioning, and split reward channels supports prior findings on reward modeling in agentic LLM settings [zheng2025survey, zhang2025lessons]. The inability of automated prompt search to discover the most valuable structural improvements, even when equipped with scalable optimization, underscores that human insight and bias correction remain essential in reward/verification design for alignment-critical systems.

Future Directions

Potential directions include extending UV to:

• Reason about unstructured or multi-modal trajectories in other domains (e.g., video task demonstration, cross-app workflows).
• Incorporate active-learning or semi-supervised iteration, mixing annotator signal with systematic prompt search.
• Investigate the scalability and sample efficiency of process/outcome separation for downstream agent RL finetuning or safety certification.

Conclusion

This work precisely characterizes the construction and validation of reliable verifiers for computer use agents, centering on rigorous rubric construction, process/outcome separation, attribution of failures, and context-efficient multimodal evidence aggregation. The UV system, underpinned by CUAVerifierBench, matches human-level annotation reliability, dramatically reduces false positives across all benchmarks and scenarios, and establishes robust inductive foundations for future CUA agent research and deployment. The path to agentic evaluation that is both trustworthy and robust requires such systematic, decomposition-driven verification—one in which architecture, not only base model power, is the differentiator.