MacAgentBench: Benchmark for macOS Agents

• MacAgentBench is a benchmark that evaluates computer use agents on macOS desktops using deterministic rule-based scoring across 676 tasks spanning 25 applications.
• It features a fine-grained multi-checkpoint scoring system for multi-application workflows, highlighting partial progress beyond simple pass/fail metrics.
• Empirical results indicate that the inclusion of pre-built skill libraries significantly boosts success rates compared to improvements from framework design alone.

Searching arXiv for the target paper and closely related benchmark work to ground the article in current literature. MacAgentBench is a benchmark for evaluating computer use agents (CUAs) on the real-world macOS desktop. It was introduced to address two limitations attributed to existing benchmarks, including those for macOS: evaluation without framework augmentation and reliance on binary evaluation. The benchmark comprises 676 distinct tasks across 25 macOS applications, with nearly 60% involving both GUI and CLI interaction, and it adopts deterministic rule-based evaluation together with fine-grained multi-checkpoint scoring for multi-application tasks. In reported experiments across three frameworks and 16 models, the best configuration—Claude Opus 4.6 on OpenClaw—attains 73.7% Pass@1, while the reported analysis attributes most of this advantage to the skill library rather than to framework design (Fu et al., 21 Jun 2026).

1. Design motivation and benchmark scope

MacAgentBench is positioned in the context of rapidly advancing desktop automation by CUAs, with the paper noting that a growing number of users deploy CUAs such as OpenClaw on Mac Mini for always-on automation. The benchmark is intended to capture capabilities used by modern CUAs that are not reflected by evaluations that omit framework augmentation, and to expose partial progress on long-horizon, multi-application tasks that binary pass/fail judgments do not reveal (Fu et al., 21 Jun 2026).

The benchmark contains 676 tasks spanning 25 macOS applications. Its task inventory is distributed across six application groupings: Productivity with 224 tasks (33.1%), System with 44 tasks (6.5%), Internet with 108 tasks (16.0%), Development with 44 tasks (6.5%), Multimedia with 116 tasks (17.2%), and Multi-App workflows with 140 tasks (20.7%). The application examples explicitly listed in the benchmark summary include Notes, Pages, and Keynote for Productivity; Terminal, Settings, and Clock for System; Safari, GitHub, and Email for Internet; VS Code and Tmux for Development; and ASR, TTS, and Video Frames for Multimedia (Fu et al., 21 Jun 2026).

Category	Example applications	Tasks
Productivity	Notes, Pages, Keynote	224 (33.1%)
System	Terminal, Settings, Clock	44 (6.5%)
Internet	Safari, GitHub, Email	108 (16.0%)
Development	VS Code, Tmux	44 (6.5%)
Multimedia	ASR, TTS, Video Frames	116 (17.2%)
Multi-App workflows	Cross-application	140 (20.7%)

This composition suggests that MacAgentBench is not restricted to pure GUI manipulation or to a narrow application vertical. A plausible implication is that the benchmark is intended to approximate macOS desktop work as a heterogeneous control problem involving application state, files, content, and system configuration rather than isolated interface navigation.

2. Interaction modalities and task decomposition

A defining feature of MacAgentBench is the distribution of interaction modes. Of the 676 tasks, 124 are GUI only (18.3%), 148 are CLI only (21.9%), and 404 require combined GUI + CLI interaction (59.8%) (Fu et al., 21 Jun 2026). The predominance of combined GUI + CLI tasks is technically significant because it means evaluation is centered on hybrid control rather than on single-modality operation.

The benchmark’s most structurally elaborate component is its set of 140 multi-application tasks. Each such task is decomposed into $m_t$ checkpoints, with mean $m_t=4.1$ and range 2–7. Every checkpoint is tagged with exactly one of five capability dimensions:

1. Research (external knowledge lookup)
2. App State (verifying or manipulating application state)
3. Content (content-correctness checks)
4. File Ops (file-system operations)
5. Sys Config (system-setting modifications)

These annotations provide a fixed vocabulary for describing where a multi-application workflow succeeds or fails. The paper’s interpretation is that capability annotations can reveal imbalances across dimensions such as Research versus File Ops versus Sys Config, which binary outcome metrics would mask (Fu et al., 21 Jun 2026).

3. Evaluation methodology and scoring formalism

MacAgentBench uses a deterministic, rule-based evaluation pipeline. The procedure has five steps: launch a fresh Docker-QEMU container from a shared copy-on-write base image with approximately 30 seconds startup and 1 GB extra disk; run a task-specific setup script to configure the initial state; deliver the natural-language instruction to the agent and let it act for up to 50 steps; invoke one or more deterministic getter+metric evaluators; and then destroy the container to enforce full isolation (Fu et al., 21 Jun 2026).

The deterministic evaluators are implemented through shell commands, AppleScript, and Python scripts. The reported counts are 88 shell-command functions, 48 AppleScript functions, and 20 Python scripts. Each evaluator returns pass/fail on a specific state predicate. This design eliminates trajectory matching and instead checks whether relevant desktop state predicates hold after execution (Fu et al., 21 Jun 2026).

For multi-application tasks, the benchmark introduces fine-grained multi-checkpoint scoring. For task $t$ with $m_t$ checkpoints, indicator variables are defined as

$$x_{t,i}=\begin{cases} 1 & \text{if checkpoint }i\text{ is passed,}\ 0 & \text{otherwise.} \end{cases}$$

The task’s partial-credit score is

$S_t=\frac{1}{m_t}\sum_{i=1}^{m_t}x_{t,i}\in[0,1].$

A task is considered fully passed only if $S_t=1$ (Fu et al., 21 Jun 2026).

The benchmark also defines Pass@k metrics. Let $\mathrm{succ}_{t,r}\in\{0,1\}$ indicate whether run $r$ on task $t$ succeeds fully. With $m_t=4.1$0 tasks,

$m_t=4.1$1

and

$m_t=4.1$2

The combination of deterministic state checking and partial-credit scoring is central to the benchmark’s methodological identity. The paper’s stated rationale is that agents frequently make substantial partial progress on long-horizon workflows before failing, so binary pass/fail alone underestimates capability (Fu et al., 21 Jun 2026).

4. Frameworks and model coverage

MacAgentBench evaluates three agent frameworks. The first is “/”, described as a baseline GUI-only agent with a screenshot$m_t=4.1$3pyautogui actions loop. The second is AgentS3, a multi-agent architecture with separate planner, grounder (GUI and code), coder, and reflector, and with support for GUI+AppleScript+code. The third is OpenClaw, an in-container harness exposing shell, AppleScript, web search, memory, and a library of 53 pre-built skills, with examples including apple-notes, blogwatcher, weather, and gifgrep (Fu et al., 21 Jun 2026).

The model set comprises 16 models. The proprietary general VLMs accessed via API are Claude Opus 4.6, GPT-5.4, and Gemini 3.1 Pro. The open-source general VLMs are Qwen3VL in 235B, 32B, and 8B variants, and InternVL3.5 in 14B and 8B variants. Native GUI agents, evaluated only on the baseline framework, are UI-TARS in 72B-DPO and 1.5-7B variants, ScaleCUA in 32B and 7B variants, GUI-Owl in 1.5-32B and 1.5-8B variants, and OpenCUA in 32B and 7B variants (Fu et al., 21 Jun 2026).

This experimental design permits comparison along at least three axes: model family, execution framework, and the presence or absence of explicit skill augmentation. The benchmark’s own analysis places particular emphasis on separating framework-level gains from gains attributable to OpenClaw’s pre-built skill library.

5. Empirical results and analytical findings

The headline Pass@1 results reported in Table 2 identify different best models for different frameworks. On the baseline, GPT-5.4 is best at 58.4%, while Claude Opus 4.6 scores 39.2%. On AgentS3, Claude Opus 4.6 reaches 66.9%. On OpenClaw, Claude Opus 4.6 reaches 73.7%, which is the best reported configuration overall (Fu et al., 21 Jun 2026).

The paper further isolates the contribution of OpenClaw’s skills from the contribution of the framework itself. On tasks covered by OpenClaw skills, Claude 4.6 achieves 89.4% on OpenClaw versus 84.3% on AgentS3 and 55.9% on the baseline. On tasks without skill coverage, OpenClaw falls to 45.6% versus 48.8% for AgentS3 and 16.2% for the baseline. The reported conclusion is explicit: the bulk of OpenClaw’s lead derives from its pre-built skill library, and the underlying framework itself gives little or no advantage on uncovered tasks (Fu et al., 21 Jun 2026).

MacAgentBench also uses its checkpoint-based scoring to expose sub-goal completion patterns that binary pass metrics obscure. On the 140 multi-app tasks, GPT-5.4 on the baseline records Pass@1 = 48.6% but average checkpoint Score = 64.1%, which the paper interprets as extensive partial progress. A second example compares UI-TARS-72B and UI-TARS-1.5-7B: both obtain 1.4% Pass@1 on these tasks, yet their Scores are 12.9% and 8.9%, respectively, indicating that the larger model completes more sub-goals even when neither model often fully solves the task (Fu et al., 21 Jun 2026).

Two common misconceptions are directly countered by these findings. One is that the best overall framework result necessarily demonstrates superior framework design; the skill-coverage analysis indicates otherwise. The second is that similar Pass@1 values imply similar agent competence; the multi-checkpoint results show that sub-goal completion can differ substantially even when final binary success does not.

6. Robustness, implications, and relation to adjacent benchmarks

The paper identifies robustness as an unresolved challenge, stating that large gaps between Pass@1, Pass@4, and Pass⁴ demonstrate that consistency, or low variance across runs, remains a major issue (Fu et al., 21 Jun 2026). This observation connects MacAgentBench’s single-trial and multi-trial metrics to a broader concern in agent evaluation: reproducible performance under repeated execution.

The benchmark’s recommendations for future agent and benchmark design are concrete. It recommends open environments that allow hybrid GUI+CLI+code interactions so that frameworks and models can be fairly compared; deterministic, rule-based multi-checkpoint evaluation to surface partial progress and pinpoint failure points; capability-dimension annotations to identify weakest areas such as Research queries; robustness metrics such as Pass@4 versus Pass@1 versus Pass⁴ in addition to single-trial success rates; and expanded coverage beyond the current seed applications together with regular evaluator updates for new macOS versions (Fu et al., 21 Jun 2026).

Within the broader benchmark landscape, MacAgentBench can be contrasted with MCP-AgentBench, which evaluates language agents in MCP-mediated tool interactions using 33 operational servers, 188 distinct tools, and 600 queries across six interaction categories, with outcome-oriented evaluation rather than desktop-state predicates (Guo et al., 10 Sep 2025). The contrast is instructive. MacAgentBench centers on a desktop operating environment with real macOS applications, hybrid GUI and CLI control, and deterministic state-based checking, whereas MCP-AgentBench centers on protocol-mediated tool interoperability. This suggests that the two benchmarks stress different strata of agent competence: operating-system-level action and application-state manipulation in one case, and protocol-level tool orchestration in the other.

MacAgentBench’s code and data are publicly available at https://github.com/JetAstra/MacAgentBench (Fu et al., 21 Jun 2026). Its principal contribution is therefore not only a large task suite but a structured evaluation paradigm that separates framework design, skill augmentation, and core model capability while making partial task completion measurable rather than invisible.