MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop

Abstract: Computer use agents (CUAs) have advanced rapidly in desktop automation, and a growing number of users deploy CUAs such as OpenClaw on Mac Mini for always-on automation. However, existing benchmarks, including those for macOS, evaluate agents without framework augmentation and rely on binary evaluation. As a result, they fail to capture both the framework capabilities leveraged by modern CUAs and the partial progress on long-horizon, multi-application tasks. We present MacAgentBench, a comprehensive macOS agent benchmark comprising 676 tasks across 25 applications, with nearly 60% involving both GUI and CLI interaction. The benchmark adopts deterministic rule-based evaluation and introduces fine-grained multi-checkpoint scoring with capability annotations for multi-application tasks. Experiments across three frameworks and 16 models show that the best configuration, Claude Opus 4.6 on OpenClaw, attains 73.7% Pass@1, while this advantage is primarily driven by the skill library rather than by framework design. Fine-grained metrics further reveal that models with similar Pass@1 can differ substantially in sub-goal completion. Our code and data are publicly available at https://github.com/JetAstra/MacAgentBench.

• The paper introduces MacAgentBench, a comprehensive benchmark using multi-modal interactions and checkpoint-based scoring to evaluate macOS agents.
• It demonstrates that integrating skill libraries within agent frameworks can boost performance from 39.2% to 73.7% Pass@1, highlighting critical methodological improvements.
• It shows that proprietary models outperform open-source ones, underscoring the need for advanced framework engineering to address agent instability in long-horizon tasks.

MacAgentBench: A Comprehensive Benchmark for macOS Computer Use Agents

Motivation and Limitations of Prior Work

Benchmarking computer use agents (CUAs) in realistic OS environments is central to measuring progress in agentic automation. Yet, existing macOS evaluation suites suffer fundamental limitations. Prior macOS benchmarks [macOSWorld, macOSArena] are constrained by fixed action spaces dominated by screenshot/GUI protocols, neglecting the hybrid action spaces enabled by contemporary frameworks (e.g., shell commands, scripting, pre-built skills). Furthermore, nearly all benchmarks rely predominantly on binary pass/fail metrics, which are inherently coarse-grained and mask model and framework-level differences, particularly for long-horizon or multi-application tasks. MacAgentBench is proposed to address these limitations by providing a unified, extensible, and open evaluation platform supporting realistic agent frameworks, multi-modal task interaction (GUI, CLI, skills), and fine-grained, checkpoint-based evaluation protocols.

Benchmark Design and Infrastructure

MacAgentBench consists of 676 tasks spanning 25 macOS-native applications with task diversity reflecting real-world workflows: 33.1% productivity (Notes, Pages, Keynote), 16% internet (GitHub, Email), 17.2% multimedia (ASR, TTS), 6.5% each for system and development, and 20.7% multi-application workflows. Notably, nearly 60% of tasks require mixed GUI+CLI interaction. Critically, the environment is open at the agent-framework interface; no restrictions are imposed on the observation/action space, allowing both remote (e.g., SSH+VNC) and in-container deployments. In practice, frameworks like OpenClaw are pre-installed inside the container, orchestrating GUI, tool, skill, AppleScript, and shell-based actions on the native OS substrate.

Figure 1: Overview of MacAgentBench, showing the spectrum from pure GUI agents to hybrid and harness-based paradigms, and the fine-grained checkpoint-based scoring protocol.

MacAgentBench is implemented atop a Docker-QEMU stack, achieving rapid (≈30s) container startup using copy-on-write overlays (≈1GB per-instance disk) versus the 15-minute EC2 snapshotting in macOSWorld and 5-minute start times with a full disk copy in macOSArena. This yields high-throughput, parallel, and clean-state task execution. Each task is specified by a triplet: a natural language instruction, a deterministic environment setup script, and a rule-based evaluator script. Multi-application workflows have annotated checkpoints for scoring and analysis.

Figure 2: Distribution of tasks across categories and interaction modalities, illustrating substantial representation across all major macOS workflow types.

Task and Evaluation Protocol

To ensure reproducibility and control, MacAgentBench eschews LLM-as-a-judge in favor of fully deterministic, script-based evaluation. Evaluation scripts (Python, shell, AppleScript) are composed of getter functions extracting environmental facts, and metric functions returning binary outcomes. For multi-step/multi-app tasks, fine-grained checkpoint scoring tracks completion rate and partial credit at the sub-goal level. Checkpoints are further annotated according to underlying capability type: Research (external knowledge), App State (application status), Content (content match), File Ops, and System Config.

Seed tasks are constructed based on both prior benchmarks (substantially revised to accommodate UI and automation changes in macOS Tahoe~26) and new workflows (e.g., iWork, cross-app automations). Each seed task is expanded lexically and parametrically via LLM rewriting and variable substitution, with manual cross-validation.

Agent Paradigms and Experimental Setup

MacAgentBench supports three archetypal paradigms:

• Pure GUI agents (e.g., pyautogui-based), returning code actions corresponding solely to mouse/keyboard events.
• Hybrid frameworks (e.g., AgentS3), extending the base action space via AppleScript or embedded code-execution agents.
• Harness-based agents (e.g., OpenClaw), which leverage full tool/skill orchestration, scripting, and sand-boxed skill library calls.

Experiments span 16 models (including proprietary VLMs like Claude Opus 4.6, GPT-5.4, Gemini 3.1 Pro, and open-source agents such as Qwen3VL and GUI-Owl families) across all three frameworks. Each model is given a budget of 50 interaction steps per task.

Main Results

Framework Impact: With the same model, framework selection is crucial. For instance, Claude Opus 4.6 reaches 39.2% Pass@1 as a pure GUI agent but jumps to 66.9% (AgentS3) and 73.7% (OpenClaw). The improved performance in OpenClaw is largely attributable to its skill library; on tasks without skill coverage, OpenClaw can underperform the hybrid AgentS3, and even the baseline for some models.

Model Impact: Proprietary VLMs consistently outperform open source. On OpenClaw, Claude Opus 4.6 achieves 73.7% Pass@1; on AgentS3, GPT-5.4 leads. Open-source models generally trail, indicating ongoing limitations in sample/parameter efficiency and tool-use reasoning.

Robustness: There exist substantial gaps between Pass@1 and Pass@4: while Claude Opus 4.6 can solve 85.2% of tasks at least once, its per-trial consistency is only 73.7%. This variability suggests a large performance ceiling limited by non-deterministic or unstable planning—an under-explored aspect in CUA evaluation.

Fine-grained Capability and Skill Library Effects

Checkpoint-based metrics reveal meaningful capability imbalances; agents achieving similar binary Pass@1 can diverge substantially in sub-goal accomplishment. For example, file operations are generally robust, but research (external knowledge lookup) and content correctness checkpoints systematically depress overall scores, especially for open-source and non-specialist models.

Figure 3: Example GPT-5.4 task trajectory on a multi-application workflow, achieving all checkpoints and demonstrating seamless CLI and GUI tool integration.

The study isolates the effect of skill augmentation. On tasks covered by a predefined skill, OpenClaw enables a ≳30% absolute improvement over pure GUI or hybrid frameworks; when skills are unavailable, AgentS3 (with multi-agent planning but no external skills) performs more consistently than OpenClaw for most models. This dictates that framework engineering and orthogonal skill libraries must be considered as confounding, not additive, performance factors.

Theoretical and Practical Implications

The results substantiate several theoretical findings:

• Evaluation Protocols: Pass/fail metrics are insufficient for nuanced progress evaluation; robust benchmarking requires multi-dimensional metrics, transparent checkpointing, and deterministic ground truth.
• Agent Robustness: Ceiling performance and per-attempt consistency are not aligned; agent instability in long-horizon tasks remains a bottleneck, particularly when offloading to skills is restricted.
• Skill Library Generalization: Real-world applicability of agent frameworks is heavily modulated by the breadth and specificity of the underlying tool set.
• Open Architecture: Flexible, open agent-environment loops are critical; enforcing a monolithic GUI or CLI paradigm fundamentally hobbles both research and practical deployment.

Practically, MacAgentBench delivers a scalable, extensible, and open infrastructure suitable for future research in agentic orchestration, prompting, and memory/in-context learning. The containerized protocol with task-level isolation also offers robust experimental methodology for controlled ablation and privacy-compliant deployment.

Future Developments

Potential avenues include expanding the skill/task space to cover more domains (e.g., creative tools, developer IDEs, advanced automation stacks), incorporating evolving macOS APIs, and refining checkpoint taxonomies to disambiguate subtle reasoning failures. Further, exploring agent robustness under policy sampling, safety constraints, or adversarial task construction will be essential for practical deployment scenarios.

Conclusion

MacAgentBench establishes a new performance and reproducibility standard for evaluation of CUAs on macOS, addressing prior limitations in action space, agent-framework definition, and evaluation granularity. The open, containerized environment, fine-grained checkpoint protocol, and empirical evidence on framework/model interplay provide a foundation for systematic agentic research, both in fundamental AI reasoning and applied system integration. The pivotal insight is that skill library composition dominates raw framework improvements; thus, benchmarking agent intelligence independently from system augmentation remains a central—and challenging—open problem.

---

References

Please see (2606.22557) for the full citation and extended results.