CUA-Speedrun Benchmark: Performance Evaluation for Computer-Use Agents

• CUA-Speedrun is a platform designed to measure the performance, wall-clock speed, efficiency, and cost of computer-use agents (CUAs) operating through graphical user interfaces. It includes standardized environments, techniques, and metrics for evaluating agents, addressing benchmarks accuracy, time, and latency variations.
• The platform identifies that agent harnesses significantly impact performance and speed, emphasizing the importance of harness-specific optimizations and reducing infrastructure latency, while stressing that different reasoning efforts have variable impacts on speed and performance depending on the workload.
• Though CUAs are evaluated using consistent environments, task benchmarks, agent policies, and a sandbox layer, these agents must also learn synchronization, waiting, and state-confirmation policies to optimize the effectiveness of their actions.

cua-speedrun is a standardized benchmarking platform for measuring the performance, wall-clock speed, efficiency, and model cost of computer-use agents (CUAs). CUAs operate graphical user interfaces through screenshots, keyboard input, and mouse actions. The platform addresses reproducibility problems caused by differences in virtual machines, containers, hardware, action APIs, model-serving systems, agent harnesses, timing conventions, and benchmark implementations. Its principal contribution is a common evaluation stack that separates the agent policy, task benchmark, execution environment, and interaction harness while recording task outcomes, timing components, interaction counts, token usage, and inference cost (Aggarwal et al., 30 Sep 2026).

1. Purpose and scope

The platform is motivated by a distinction between capability and operational efficiency. Conventional CUA benchmarks primarily report whether an agent completes a task, whereas deployment also depends on how quickly and cheaply completion occurs. cua-speedrun therefore evaluates performance as a joint capability–latency–cost problem.

The platform’s stated conclusions include the following:

• No single model family is optimal simultaneously for performance, time, and cost.
• None of the evaluated open-weight systems lies on the performance–time or performance–cost Pareto frontier.
• Increasing reasoning effort can sometimes reduce total completion time by preventing repeated unsuccessful actions.
• Faster environment input-output can increase overall completion time when screenshots arrive before the desktop has visually updated.
• Agent harnesses materially affect both accuracy and speed.
• Carefully selected task subsets can preserve benchmark rankings while reducing evaluation time.

The benchmark is not limited to a particular model architecture or agent framework. It provides a common interface through which direct model APIs, Codex, Claude Code, and open-weight models served by vLLM can operate across multiple CUA benchmarks. The same agent implementation can therefore be evaluated under different task sets and execution environments.

The platform is distinct from reliability-only or task-completion benchmarks. MCPWorld, for example, evaluates GUI-only, MCP/API-only, and hybrid agents using internal application verification but does not report completion-time distributions, action counts, token consumption, or cost-adjusted performance (Yan et al., 9 Jun 2025). cua-speedrun explicitly treats these operational quantities as first-class evaluation dimensions.

2. Formal task and execution model

A task is represented as

$x_i=(E_i,s_i^0,p_i,V_i),$

where $E_i$ is the interactive environment, $s_i^0$ is the initial state, $p_i$ is the natural-language instruction, and $V_i$ is the verifier. An agent $\pi$ receives the instruction and a sequence of screenshots, emits computer actions, and eventually terminates or reaches a task limit. The resulting trajectory is $\tau_i$, with final environment state $s_i^T$ and verifier score

$r_i=V_i(\tau_i,s_i^T).$

A benchmark is a collection

$B=\{x_i\}_{i=1}^{N}.$

The supported interaction vocabulary includes mouse clicks, double- and triple-clicks, pointer motion and dragging, scrolling, keyboard text and key chords, explicit waits, screenshot requests, and completion. Desktop observations are RGB screenshots at $E_i$0. Actions use pixel coordinates, although an agent harness may transform normalized or resized coordinates into desktop pixels.

The environment and agent execute in separate sandboxes. The environment sandbox contains the desktop, applications, task state, and verifier-facing environment; the agent sandbox contains the model interface and agent loop. Communication occurs through a controlled screenshot/action interface. The runtime adapts the Gym-Anything execution model to Modal sandboxes.

Each task starts from a fresh environment. The infrastructure records trajectories, screenshots, action logs, scores, timing components, and model cost. For self-hosted open-weight models, inference servers use vLLM on fixed NVIDIA L40S GPUs. A Python library manages GPU scheduling, environment allocation, parallelism, and task assignment.

Task timing begins when the instruction is given to the agent and ends when the agent terminates or reaches its limit. It excludes VM provisioning, task setup, agent initialization, model loading, execution-slot waiting, and verification. Infrastructure failures are retried; agent failures terminate the trajectory and are scored.

Timing is decomposed into environment and agent components:

$E_i$1

where $E_i$2 is environment time, including action execution and observation return, and $E_i$3 is agent time, including model requests and agent-side processing. Explicit agent-requested waits are recorded separately and may be subtracted from environment time.

A common agent interface allows an implementation in one Python file to request screenshots, call a model function, submit one or more actions, and repeat until completion. A response may contain multiple tool calls, and a tool call may contain multiple sequential actions before another screenshot is requested. This makes action batching an explicit experimental variable rather than an incidental implementation detail.

3. Benchmarks, task populations, and representative subsets

The platform evaluates four CUA benchmarks:

1. OSWorld
2. OSWorld 2.0
3. CUA-World
4. MyPCBench

OSWorld and OSWorld2 use primarily programmatic task verification. CUA-World evaluates long-horizon computer use using a vision-language-model judge, while MyPCBench evaluates personalized computer use through rubric-based visual trajectory assessment.

The deployed task populations and representative subsets are:

Benchmark	Full set	cua-speedrun subset
OSWorld	295	50
OSWorld2	63	52
CUA-World	143	26
MyPCBench	184	38

The OSWorld population was reduced from 369 tasks after an internet-dependence audit. Three reviewers independently classified whether tasks required interaction with live public services. The final inclusion rule required unanimous inclusion and yielded 295 tasks. Pairwise reviewer agreement was 98.64%, 99.46%, and 98.10%; corresponding Cohen’s $E_i$4 values were 0.956, 0.983, and 0.939, with Fleiss’ $E_i$5.

The platform evaluates 56 configurations on OSWorld and 21 on OSWorld2. Configurations vary model family, reasoning effort, harness, batching, and input-output mode. The task subsets are intended to reduce evaluation cost while retaining the statistical structure of the full benchmark.

Representative-task selection

Task selection uses an energy-distance objective based on the behavior of evaluated agents. Let $E_i$6 be model $E_i$7’s partial score on task $E_i$8, and let

$E_i$9

denote exact completion. Each task is represented by

$s_i^0$0

The distance between tasks is

$s_i^0$1

For a subset $s_i^0$2 of $s_i^0$3 tasks from a benchmark of $s_i^0$4 tasks, the normalized energy objective is

$s_i^0$5

where $s_i^0$6 is the mean distance between distinct tasks. The objective favors subsets that represent the full distribution of task difficulty and agent failure patterns while avoiding redundant tasks.

Selection creates 100 balanced initial subsets, repeatedly swaps selected and unselected tasks when the swap reduces energy, and stops when no improving swap remains. Balanced initializations are produced from the leading principal coordinate of centered score profiles, with random phase $s_i^0$7 and seeds 7000–7099.

Leave-one-agent-out validation removes each agent’s scores during selection, estimates the held-out agent’s full-benchmark result using the selected subset, and compares the estimate with the full-benchmark score. For quantity $s_i^0$8, the chosen subset size is

$s_i^0$9

where $p_i$0 is the Spearman correlation between subset estimates and full-benchmark results.

For OSWorld, the selected 50-task subset achieved partial-score correlation of 0.983 and exact-completion correlation of 0.975. It reduced the number of tasks by $p_i$1, represented 15.1% of total recorded task time, and corresponded to a $p_i$2 reduction in trajectory time. The introduction reports an 84.9% reduction in OSWorld evaluation time. The subset recovered two of three full-set speed–performance frontier configurations, with 100% precision and 66.7% recall; pairwise dominance agreed for 91.7% of model pairs.

For OSWorld2, the selected 52-task subset achieved partial-score correlation of 0.964 and exact-completion correlation of 0.955. For CUA-World, selected subsets of 25, 26, and 27 tasks produced held-out rank correlations of 0.964, 1.000, and 1.000, respectively.

The selection procedure remains dependent on the calibration agents used to construct task representations. An automated search over 34 candidate selection methods reduced validation error from 2.43 to 0.01 percentage points but changed held-out-model error from 3.90 to 4.00 points, demonstrating overfitting to a small model pool.

4. Metrics and Pareto analysis

Performance

For an agent $p_i$3 evaluated on $p_i$4 tasks, mean verifier performance is

$p_i$5

The platform retains partial verifier scores and also reports exact completion, defined as the fraction of tasks receiving full credit. Median time on successfully completed tasks is reported alongside the mean task time.

Mean task time

Mean task time is

$p_i$6

It includes successful and unsuccessful attempts and ends at agent termination or the task limit. The platform separately records total task time, agent time, environment processing time, agent-requested wait time, model-response time, action batches, individual actions, and model calls.

Model cost

Mean model cost is

$p_i$7

Costs use provider-recorded charges or token-based prices. They exclude VM hosting, GPU rental, setup, initialization, and verification calls. Thus, model cost is primarily an inference-cost measure rather than a complete deployment-cost estimate.

Interaction counts

If $p_i$8 is the number of action batches for task $p_i$9, and $V_i$0 is the action list in batch $V_i$1, the number of individual actions is

$V_i$2

The platform distinguishes model responses, tool calls, action batches, individual actions, and generated tokens. A “step” is a request containing keyboard, mouse, or wait actions; a batch may contain several actions executed sequentially before the next screenshot.

All generated tokens are counted once per response, including reasoning tokens and other output. The effective generation rate is

$V_i$3

where $V_i$4 is the generated-token count for task $V_i$5. A normalized comparison assigns all systems a common generation rate of 100 tokens per second, retaining environment time and replacing measured generation time with generated tokens divided by 100 tokens per second.

Pareto frontiers

A configuration lies on the performance–time frontier when no evaluated configuration has both higher performance and lower time. The performance–cost frontier uses cost instead of time. The joint frontier includes performance, time, and cost: a configuration is dominated if another configuration improves one quantity without worsening the others.

This formulation prevents speed or cost from being interpreted independently of task performance. A faster agent that fails more tasks is not automatically superior, and a highly capable agent that is prohibitively slow may not be operationally preferable.

5. Infrastructure latency and FastCUA

Existing CUA runtimes typically require approximately 2–3 seconds between issuing an action and receiving the next screenshot. This delay may include fixed post-action sleeps, action execution, application processing, screenshot capture, image encoding, communication, and cleanup.

FastCUA reduces this latency by combining action execution and screenshot delivery, keeping screenshots in memory, performing encoding in the background, reusing network connections, and placing agent and environment sandboxes in the same Modal region. Reported action-to-observation latency is approximately 2–28 ms.

Action	Gym-Anything	OSWorld	FastCUA
Click	2758.70 ms	2515.41 ms	27.58 ms
Escape	2865.00 ms	2707.74 ms	2.43 ms
Ctrl+U	2858.29 ms	2708.41 ms	2.91 ms
Type 100 characters	3521.72 ms	2756.47 ms	14.98 ms

Full command-to-screenshot measurements remained below 50 ms, with medians of 38.5 ms for a click, 12.8 ms for Escape, 13.6 ms for Ctrl+U, and 29.5 ms for typing 100 characters.

Lower infrastructure latency did not necessarily reduce total task time. For GPT-6 Astra at low reasoning effort on OSWorld, normal input-output required 89.5 seconds per task, whereas FastCUA required 99.0 seconds. Mean environment processing time excluding requested waits fell from 12.68 seconds to 0.34 seconds, but the agent generated more interaction steps.

The cause was synchronization failure. Screenshots could arrive before the application had visually updated. After opening a Save As dialog, for example, the first screenshot did not show the dialog, while a later screenshot requested 5.8 seconds afterward did. The agent could interpret the stale screenshot as evidence that its action had failed, request another observation, or repeat an action.

In a five-seed FastCUA experiment, mean score was $V_i$6, mean task time was $V_i$7 seconds, mean environment time was 5.075 seconds, and explicit waits accounted for 4.739 seconds. Environment time excluding waits was $V_i$8 seconds.

The result establishes that reducing transport latency is not sufficient. Agents must also learn synchronization, waiting, and state-confirmation policies. Infrastructure optimization can expose previously hidden temporal assumptions in the agent’s policy.

Harness effects

The harness determines how model outputs are converted into actions, how screenshots and histories are managed, and whether actions are batched. GPT-5.6 Luna on OSWorld illustrates this dependence:

Reasoning effort	Direct API	Codex
Low	61.8% / 74 s	75.6% / 145 s
Medium	75.6% / 119 s	67.6% / 189 s
High	81.6% / 161 s	79.6% / 325 s

At low effort, Codex improved performance but nearly doubled task time. At medium and high effort, the direct API was both more accurate and faster.

Harness errors include reversed scroll direction, key names typed as text rather than pressed, discarded middle-clicks, premature modifier release, omitted later tool calls, and incorrect modifier or Unicode handling. The original SSH/PyAutoGUI pathway passed 83 of 94 supported test cases over five repetitions; after correction, the CUA-Speedrun QEMU runtime passed all $V_i$9 tests.

Batching

Batching can reduce screenshot requests, model calls, and environment round trips. For Astra xhigh, batched actions achieved 91.62% in 126.82 seconds, while one action per screenshot achieved the same score in 182.73 seconds. The single-action restriction increased action batches from 7.14 to 16.50 and generated tokens from 2,060 to 2,559.

Batching is not universally beneficial. Kimi’s results varied with reasoning effort: at low effort, batching reduced time and improved score; at high effort, batching increased time and reduced score; at maximum effort, batching reduced time and improved score. The value of batching therefore depends on whether the model can reliably plan multiple actions without requiring intermediate observations.

6. Empirical findings, limitations, and significance

Model and frontier results

On OSWorld, representative high-performing points on the performance–time frontier included:

• Claude Opus 5, low: 87.6% in 85.9 seconds.
• GPT-6 Astra, low: 90.8% in 89.5 seconds.
• GPT-6 Astra, xhigh: 91.6% in 126.8 seconds.
• Gemini 3.8 Flash, low: 91.6% in 127.1 seconds.

Gemini 3.8 Flash at low effort matched the highest reported score while costing approximately $\pi$00.708 for Astra xhigh and $0.531 for Astra low.

GPT-5.6 Luna through its direct API occupied the low-cost region:

• low effort: 61.8% in 73.6 seconds at $0.011 per task;
• medium effort: 75.6% in 118.9 seconds at $0.019;
• high effort: 81.6% in 160.5 seconds at $0.029.

On OSWorld2, Astra formed the entire time frontier. Its scores and times were 68.2% in 904.2 seconds at low effort, 67.2% in 828.1 seconds at medium effort, 75.0% in 914.0 seconds at high effort, and 76.9% in 1,237.3 seconds at xhigh effort.

No evaluated open-weight system occupied the performance–time or performance–cost frontier on OSWorld or OSWorld2. Kimi K3 at maximum reasoning approximately matched Astra at high performance but required 443.7 seconds per OSWorld task compared with 126.8 seconds for Astra xhigh.

Reasoning effort

Reasoning effort has a non-monotonic relationship with performance, time, and cost. Gemini 3 Flash Preview on OSWorld achieved 33.6% in 492 seconds at low effort, 57.6% in 268 seconds at medium effort, and 59.6% in 337 seconds at high effort. Medium effort generated more tokens but made fewer model calls and action steps, reducing total time.

Additional reasoning can also be wasteful. Gemini 3.8 Flash achieved 91.6% in 127 seconds at low effort, the same 91.6% in 250 seconds at medium effort, and 87.6% in 348 seconds at high effort. On OSWorld2, however, moving from low to medium effort substantially improved performance, while moving from medium to high added only 0.2 percentage points and increased time and cost.

The findings support workload-specific reasoning calibration rather than a universal preference for either minimal or maximal reasoning.

Tokens and interaction efficiency

The fastest configurations are not necessarily those with the highest token-generation rate. They tend to generate fewer total tokens and execute fewer unnecessary interactions. Claude Opus 5 low generated 1,795 tokens per task and achieved 87.6% in 85.9 seconds; Astra low generated 1,579 tokens and achieved 90.8% in 89.5 seconds.

Claude Sonnet 5 high generated 3.5 times more tokens than Astra medium while taking 141 seconds compared with 90 seconds. Gemini 3.8 Flash high generated 5.6 times more tokens than its low-effort configuration and took 2.7 times longer.

On the same OSWorld2 tasks completed successfully at every effort, Gemini 3.8 Flash high required 3,024.5 seconds compared with 1,291.0 seconds at low effort, generated 73,578 tokens compared with 29,596, and had a lower effective token rate: 42.18 compared with 47.57 tokens per second. These results indicate that reducing unnecessary model calls, screenshots, actions, and retries can be more consequential than increasing raw token throughput.

Reproducibility and run variation

The platform measures run-to-run variation. GPT-6 Astra low through Codex, evaluated five times on the same 50 OSWorld tasks with different task seeds, produced a score of $\pi$1, task time of $\pi$2 seconds, and 5.86 action batches per task with standard deviation 0.06.

Partial-credit scores were identical across all five runs on 48 of 50 tasks. Under standard input-output, 42 tasks received full credit in all five runs, two in four runs, and six in none. Standard-I/O Pass@1 was 87.2%, while Pass@5 was 88.0%; the best-of-five partial score was 91.62%.

Fast input-output produced somewhat greater action-sequence variability: median task-level standard deviation was 0.99 batches and median range was two batches, compared with 0.55 and one batch under standard input-output. Repeated runs are therefore relevant for high-stakes comparisons, although the tested mean results were relatively stable.

Relationship to other CUA evaluation principles

The platform’s emphasis on standardized timing complements work on benchmark validity. A fixed-state benchmark can be solved by memorizing and replaying successful action sequences, even without screen observation; a 1 MB replay script outperformed its source frontier model on OSWorld and MobileWorld in one such evaluation (D'Oro et al., 7 May 2026). This motivates varying task parameters, data profiles, visual themes, and initial UI states, together with privileged verification and sandboxed execution.

Reliable completion verification is also essential. VLM judges frequently over-accept incomplete trajectories, particularly when an agent’s final narration claims success. On OSReward, over-accepting failed runs was the dominant judge error, and performance fell sharply on deceptive hard cases (Sun et al., 30 Jul 2026). A speed benchmark must therefore distinguish rapid verified completion from rapid production of a plausible completion claim.

CUADebug further indicates that failure diagnosis can improve re-execution success, although it does not directly establish lower latency or fewer actions (Zhang et al., 31 Jul 2026). These results suggest that speed evaluation should record recovery behavior, restart costs, and the distinction between first-pass speed and time-to-verified-success.

Limitations

The platform has several limitations:

1. Time- and price-dependence: provider latency, throughput, and API prices can change.
2. Subset dependence: representative task sets require revalidation as model capabilities and failure patterns change.
3. Incomplete configuration coverage: not every combination of model, harness, reasoning effort, benchmark, hardware, and serving configuration can be evaluated.
4. Partial cost accounting: reported model cost excludes environment hosting, GPU rental, infrastructure, setup, initialization, and verification.
5. Judge dependence: CUA-World and MyPCBench rely on vision-language-model judges, which can introduce evaluator noise and bias.
6. Fast-I/O realism: extremely low latency may produce stale screenshots and may not represent ordinary desktop deployments.
7. Selection overfitting: task-selection methods can overfit the calibration agents used to construct them.
8. Limited ecological validity: controlled virtual desktops do not capture every production condition, including authentication challenges, network failures, interruptions, and application-specific side effects.

cua-speedrun establishes a measurement framework rather than a single universal leaderboard score. Its central methodological claim is that CUA progress must be evaluated jointly through verified performance, wall-clock time, interaction efficiency, model cost, infrastructure latency, and reproducibility. A fast agent is meaningful only when its success is valid, its actions are correctly executed, its timing boundary is explicit, and its measured advantage survives standardized re-evaluation.