AutomationBench: AI Automation Benchmark

• AutomationBench is a benchmark platform that evaluates AI agents on realistic, cross-application business workflow automation with autonomous API discovery and strict policy adherence.
• It leverages over 600 real workflow tasks across six business domains and integrates more than 500 REST endpoints to simulate enterprise environments with realistic noise and adversarial data.
• Its evaluation protocol employs deterministic end-state correctness, providing fine-grained performance metrics that highlight significant capability gaps in current state-of-the-art models.

AutomationBench is a benchmark platform designed to evaluate AI agents on realistic, cross-application business workflow automation. It addresses deficiencies in prior benchmarks by combining autonomous multi-application orchestration via REST APIs, self-directed API endpoint discovery, and strict organizational policy adherence. AutomationBench is grounded in business workflows derived from real-world patterns and simulates realistic environments containing both relevant and adversarial data. Its evaluation protocol focuses exclusively on deterministic end-state correctness, providing a rigorous, fine-grained metric for automation capabilities relevant to enterprise contexts (Shepard et al., 21 Apr 2026).

1. Scope, Motivation, and Core Contributions

AutomationBench was motivated by practical demands in enterprise automation where tasks span multiple SaaS applications (e.g., CRM, email, calendar, spreadsheets, messaging). These tasks require agents to determine which APIs to call, sequence actions across interdependent systems, interpret layered business policies, and filter noisy or misleading data. Previous benchmarks were limited to single-application tool use, web browsing without layered policy, or fixed tool corpora, lacking the cross-app coordination and realistic policy constraints of business processes.

Key contributions include:

• Definition of a large-scale, cross-application API orchestration benchmark using over 600 real workflow tasks from Zapier’s platform.
• Task coverage spanning six business domains (Sales, Marketing, Operations, Support, Finance, HR) and 47 simulated applications.
• Incorporation of more than 500 REST endpoints and tool interfaces (Search, Execute), implemented in a simulated environment with Pydantic-backed state modeling.
• Layered business policies embedded in spreadsheets and structured documents that may override naïve action defaults.
• Exclusive end-state, programmatic grading via deterministic invariants—precluding partial credit or LLM-in-the-loop judging.
• Comprehensive model evaluation with a public leaderboard, cost-aware metrics, and detailed failure analysis.

Even state-of-the-art models (e.g., GPT-5.4, Opus 4.7, Gemini 3.1 Pro) achieve pass rates under 10% under the AutomationBench regime, highlighting a significant capability gap (Shepard et al., 21 Apr 2026).

2. Formal Task and Environment Specification

AutomationBench models the evaluation scenario as a Markov Decision Process (MDP):

• $S$: the state space, representing all simulated application databases.
• $A$: the action space, i.e., possible API calls (including method, URL, and payload), and tool invocations.
• $O$: the observation space, consisting of API schema metadata, call responses, and policy documents.

At each timestep $t$, the agent observes $o_t \in O$, chooses an action $a_t \in A$, and the environment advances as $s_{t+1} = T(s_t, a_t)$ and $o_{t+1} = \Omega(s_{t+1})$. The objective is maximization of end-state task success across a task distribution:

$$\mathrm{SuccessRate} = \frac{1}{N}\sum_{i=1}^N \mathbb{I}(\text{task}_i \text{ is correct})$$

This objective is computed after a deterministic, at-most-50-step trajectory per task, using programmatic invariants that check for exact data and side effects in the target systems.

3. Environment and Workflow Design

Tasks in AutomationBench are constructed by seeding realistic state across six domains, using data schema and workflows patterned after real enterprise processes from Zapier’s customer base. Each task incorporates:

• Up to 47 simulated applications (e.g., Zoom, Google Calendar, Gmail, Google Sheets, Slack, Asana, CRMs).
• Realistic noise: extra calendar events, spam, “DRAFT” spreadsheet entries, misnamed or similar IDs, timezone pitfalls, or decoy emails—to ensure agents cannot simply rely on heuristics, but must apply persistent search and careful reasoning.
• Representative example tasks include coordinating meeting resolutions across calendars with priority rules, automating Asana task creation from processed emails, triaging Zendesk support tickets through CRM and Slack, automating HR onboarding workflows, and performing multi-step finance reconciliations.

Agents are required to search the environment for correct records, applying business and policy logic rather than following surface cues. The design challenges hard-coded search strategies and mandates robust, context-integrated reasoning.

4. Autonomous API Orchestration

Agents in AutomationBench receive neither explicit endpoint lists nor REST API toolnames a priori. Instead, two primary interfaces are available:

• Search(schema, keyword): retrieves top-k matches via BM25 over OpenAPI-style schema descriptions.
• Execute(method, url, body): simulates an HTTP request, returning a Pydantic-validated response and standard error codes.

The API surface is specified as OpenAPI-style JSON schemas, covering endpoint names, payload types, and parameters. Agents must parse these schemas, explore their structure, correctly populate request parameters, handle pagination, error codes (e.g., 404, 400), and construct correct sequences of interdependent calls without advance knowledge of endpoint topology. Up to 50 actions are permitted per task; parallelization is allowed.

Errors in API discovery often dominate failure analysis, with agents defaulting to incorrect applications or endpoints due to assumptions instead of schema-driven exploration.

5. Policy Handling and Rule Adherence

A distinguishing feature of AutomationBench is its embedding of layered business rules—represented as documents or spreadsheets accessible via Google Sheets APIs or configuration files—directly into the environment. Rules may involve:

• Explicit prioritizations ("C-level attendee → Rule 1", "PRIORITY flag → Rule 2"), sometimes overriding direct user requests.
• Constraints that require agents to fetch, parse, and apply policies before any environment mutations.
• Rejection of task completions when rules are violated (e.g., acting on a "DRAFT" entry, honoring an unauthorized external request).

Rules have hierarchical priority and may explicitly stipulate conditions under which external or lower-priority inputs must be ignored, necessitating that agents programmatically resolve and apply policy layers correctly.

6. Evaluation Protocol and Metrics

AutomationBench employs a strictly programmatic and deterministic end-state grading. For each evaluated task, assertions are coded to verify both positive invariants (e.g., correct record presence, state fields, message content) and negative invariants (e.g., no spurious Slack postings, no incorrect field updates). There is no partial credit—all "must-pass" checks must succeed and no "must-not" conditions can be violated for success. Run-to-run variance is tightly controlled (≤1%).

Key metrics:

• SuccessRate as defined above
• Cost per run (USD), calculated as token usage multiplied by model API rates
• Detailed run statistics (step count, tool call count, failure diagnostics)
• Leaderboard with public and private task sets

Models are benchmarked with multiple seeds for statistical robustness. As of the most recent leaderboard:

• Opus 4.7 achieves 9.9% pass, Gemini 3.1 Pro 9.6%, GPT 5.4 7.6%, Sonnet 4.6 5.3%, Haiku 4.5 1.5%
• Jaccard similarity between Opus and Gemini pass sets is 0.17, indicating diverse solution strategies and non-overlapping failure modes
• Observed average steps: Opus 12.6, Gemini 21.8, GPT 5.4 15.4; tool calls: Opus 29.8, Gemini 35.4, GPT 5.4 43.9

7. Analysis of Model Failures

Identified bottlenecks in current systems include:

• False positives: 72% of Opus failures involve missing required items despite being marked as success
• Incomplete list processing: Agents fail to loop over all necessary items (e.g., emails, leads), leading to partial or zero task completion
• Policy misapplication: Errors in rule priority interpretation and override handling
• API/endpoint discovery: Incorrect inferences regarding application or data location, typically from heuristic or fragile exploration patterns

Many errors are uncoverable only in rigorous, multi-app, high-noise contexts, thus demonstrating the necessity of comprehensive, end-state benchmarks over input/output-only or single-step evaluation.

8. Limitations, Open Problems, and Future Directions

Notable open challenges are:

• Subjectivity: Current grading schemes minimize judgement components; future work may require nuanced evaluation of subjective or free-form behaviors.
• Dynamic workflows: Extending AutomationBench to support interactive, multi-turn dialogues and clarifications during task execution.
• Real-world unpredictability: Simulating rate limits, partial failures, and dynamic schema changes to reflect production realities.
• Generalization: Mitigating agent overfitting to known hardening patterns in the private task set.

Potential directions include:

• Integration of dynamic policy languages (logic rules, decision trees)
• Combining REST API orchestration with UI-level actions (enabling hybrid “cross-modal” agents)
• Incorporation of reinforcement learning protocols with verifiable reward signals to incentivize exhaustive search behavior
• Enhanced tool and endpoint discovery via semantic embedding and code-aware schema analysis
• Partial-credit grading schemes for incremental progress in long-horizon workflows

AutomationBench delineates a rigorously realistic and incrementally extensible testbed for agentic AI, establishing a high bar for automation benchmarks that require robust, policy-aware, cross-application coordination, thereby mirroring the operational demands of contemporary business environments (Shepard et al., 21 Apr 2026).