CUAJudge: Automated Evaluation for CUAs

Updated 3 July 2026

CUAJudge is a fully automatic evaluation pipeline designed to assess computer-use agents in dynamic, multimodal digital tasks through LLM-driven analysis.
It decomposes tasks into key sub-tasks and selects evidence frames from screenshots, enabling detailed outcome judgments without human annotations.
Empirical studies show high human-alignment with over 93% agreement and improved precision compared to traditional rule-based methods.

CUAJudge is a fully automatic evaluation pipeline designed for computer-use agents (CUAs) engaged in real-world, multimodal digital environments. Its central purpose is to deliver environment-grounded binary reward signals for reinforcement learning (RL) and curriculum generation, eliminating the need for human-annotated labels. CUAJudge operates entirely through prompt-based use of LLMs and is built to assess long, action-conditional event traces—spanning screenshots and action histories—while achieving high alignment with human judgments. The system was introduced as a core component of the ACuRL framework for autonomous continual learning of CUAs (Xue et al., 10 Feb 2026).

1. Motivation and Conceptual Foundation

CUAs deployed in dynamic digital environments—such as spreadsheet manipulation, graphical editing, or document organization—frequently encounter scenarios where simple rule-based or pass/fail test evaluation is insufficient. Tasks often require:

Temporal compositionality (tracking changes over extended trajectories),
Multimodal alignment (reasoning about images, UI states, and logs),
Interpretable subgoal verification (assessing intermediate outcomes).

Prior RL settings with easy-to-formalize reward structures (e.g., code synthesis, math problems) benefit from direct comparison between outputs and ground truth. In contrast, CUAs require nuanced judges capable of analyzing not just the initial and final states, but also the sequence and relevance of actions taken.

CUAJudge addresses this by incorporating LLM-mediated decomposition and evaluation, filling a critical gap for robust, scalable, and environment-sensitive agent feedback without external annotation overhead. Within the ACuRL curriculum-RL loop, CUAJudge outputs are used both for RL reward assignment and for adaptive generation of new training tasks (Xue et al., 10 Feb 2026).

2. Architectural Pipeline

CUAJudge extends the WebJudge approach with a multi-stage architecture, each mediated by specialized LLM prompts:

A. Key-Point Identification: Given a high-level, free-form task description (e.g., “Filter the spreadsheet and copy filtered rows”), the system extracts a minimal set of atomic sub-tasks ("key points") via LLM prompt. Typical output might include requirements such as “Column E (Transmission) equals ‘Manual’” and “Filtered rows moved to sheet ‘Manual Cars’.”
B. Key-Screenshot Identification: The full trajectory comprising ℓ screenshots is scored for relevance against extracted key points using a second LLM prompt. The K most informative screenshots (evidence frames) are selected to substantiate outcome assessment.
C. Outcome Judgment: Using (i) initial and final environment screenshots, (ii) evidence frames, (iii) full action history, and (iv) key points, a final LLM prompt adjudicates task success by:
- Computing pixel- or DOM-level diffs between start and end states,
- Explicitly verifying satisfaction of each key point, and
- Checking for the absence of extraneous or violating actions.

The verdict is reduced to a single binary reward, $R \in \{0,1\}$ , interpreted as success or failure (Xue et al., 10 Feb 2026).

3. Integration as an RL Reward and Feedback Signal

CUAJudge’s binary outputs are central to both policy optimization and adaptive curriculum within ACuRL:

For each sampled trajectory $\tau$ , the system computes $R(\tau) = \mathrm{CUAJudge}(\text{task}, \{I_t\}, \{a_t\})$ .
Rewards are normalized across the batch to form advantages $\hat{A}^{(i)}$ , which are then assigned uniformly to all action tokens within the trajectory.
These advantages are incorporated into the Group Relative Policy Optimization (GRPO) update:

$\tilde{A}_{t,k}^{(i)}(\theta) = \min\left\{ r_{t,k}^{(i)}(\theta) \hat{A}^{(i)},\; \mathrm{clip}(r_{t,k}^{(i)}(\theta), 1-\epsilon, 1+\epsilon) \hat{A}^{(i)} \right\}$

where $r_{t,k}^{(i)}(\theta)$ is the token-level policy ratio, and the overall objective is minimized to optimize agent policy.
The "mean success rate" guides the difficulty of subsequent tasks generated by the curriculum module, ensuring curriculum adaptation in the absence of explicit labels.

This design enables agents to autonomously cycle through exploration, curriculum generation, RL via CUAJudge-mediated reward, and re-evaluation, facilitating rapid continual adaptation in previously unseen environments (Xue et al., 10 Feb 2026).

4. Empirical Validation and Performance

CUAJudge has undergone two primary empirical evaluations:

Rule-based Baseline Comparison: On the OSWorld dataset (1,444 agent trajectories), CUAJudge achieved macro-averaged precision of 79.4%, recall 82.1%, and aggregate agreement with the rule-based oracle of 88.4%. Relative to the predecessor WebJudge, this reflects a 4.6 point gain in precision, attributed to stricter evidence-grounding via state-diff and key-point mechanisms.
Human Annotation Alignment: On a set of 288 RL trajectories sampled uniformly across six environments and three ACuRL iterations, CUAJudge attained overall agreement of 93.7% with crowdworker ground-truth judgments (precision 94.5%, recall 93.8%). Agreement remained consistently above 89% in all training iterations, indicating high and stable reliability throughout task curriculum progression.

No evidence of reward hacking or significant reliability collapse was observed as task complexity increased (Xue et al., 10 Feb 2026).

5. Implementation Details and Pseudocode

CUAJudge requires no separate ML model training; it is implemented as chained minimal LLM prompts. The system can be instantiated as follows:

Function CUAJudge(task_desc, screenshots I[0..L], actions a[1..T]):
  # Stage 1: Key Point Extraction
  key_points = LM_extract_keypoints(task_desc)
  # Stage 2: Screenshot Selection
  evidence_indices = LM_score_screenshots(I[0..L], key_points)
  evidence_frames  = { I[i] for i in evidence_indices }
  # Stage 3: Outcome Judgment
  diff     = compute_image_or_dom_diff(I[0], I[L])
  verdict  = LM_judge_outcome(
                task_desc,
                key_points,
                diff,
                evidence_frames,
                actions
           )
  If verdict == "success":
    Return 1
  Else:
    Return 0

This prompt-chain structure makes the pipeline highly adaptable, allowing researchers to swap or tune LLM components as needed. All reward computation and key-point validation are fully explained within the prompt, yielding transparent and reproducible evaluation (Xue et al., 10 Feb 2026).

6. Limitations, Ablation Findings, and Open Questions

LLM dependency: The absence of explicit ML model training means reliability derives entirely from LLM reasoning and prompt quality.
Human disagreement: Remaining 6–10% error relative to human annotation typically arises from subtle GUI modifications (e.g., minor visual changes), implicit user intentions not captured by key-point extraction, and extremely long trajectories where earlier context must be discarded.
Cost-performance tradeoff: Qwen3-VL-8B outperformed GPT-5-mini for key-screenshot selection, with higher precision (70.3% vs 65.6%) at one-fifth the compute cost to process ~100 trajectories.
Scalability: No evidence of reward hacking was detected, but generalization to tasks with complex external side-effects (e.g., persistent file changes, external APIs) remains untested.
Open questions: Can CUAJudge be reliably extended to mixed-initiative scenarios (user–agent co-adaptation)? Is it possible to fine-tune compact evaluators to maintain ≥90% agreement with reduced inference cost?

7. Comparative Position and Broader Context

Compared to other LLM-based “judge” frameworks (e.g., CompassJudger-2 (Zhang et al., 12 Jul 2025), judge-aware ranking in LLM evaluation (Xu et al., 29 Jan 2026), specialist judge models in legal AI (Zambrano, 18 Jul 2025)), CUAJudge is unique in two respects:

Its design is tailored for trajectory-based, grounding-heavy computer-use tasks, rather than pairwise LLM comparisons or single-shot legal judgments.
It functions without human-annotated data, establishing a generalizable standard for zero-shot evaluation pipelines in digital environments.

A plausible implication is that CUAJudge’s architecture establishes a robust, interpretable baseline for future work in fully automatic agent evaluation under complex multimodal feedback settings. Its demonstrated human-alignment (93+%) and plug-and-play structure make it a strong candidate for rapid iteration and benchmarking in agent-centric continual learning research (Xue et al., 10 Feb 2026).