---
title: 'Agents'' Last Exam: AI Workflow Benchmark'
url: https://www.emergentmind.com/topics/agents-last-exam-ale
type: topic
---

# Agents' Last Exam: AI Workflow Benchmark

Agents' Last Exam (ALE) is a benchmark for evaluating AI agents on long-horizon, end-to-end digital workflows drawn from professional practice and selected for verifiable outcomes [2606.05405]. It was developed in collaboration with over 250 industry experts, is anchored in O*NET / SOC 2018, and organizes more than 1,000 task instances into 55 subdomains within 13 industry clusters [2606.05405]. ALE is intended to measure whether a Generalist Computer-Use Agent can carry out sustained, economically meaningful work rather than merely answer short questions, while remaining a living benchmark through continuous task-pool growth and rotating public/private splits [2606.05405].

## 1. Definition, scope, and design objective

ALE is defined as a benchmark of real-world, multi-step computer-use tasks drawn from professional practice [2606.05405]. Its primary aim is explicitly two-fold. First, it serves as a competence threshold: an agent that “passes” ALE demonstrates the ability to carry out sustained, economically meaningful workflows. Second, it serves as a difficulty frontier: by embedding authentic software tools, GUI and CLI interactions, and long-horizon planning, it is positioned at the boundary of what current systems can accomplish [2606.05405].

The benchmark evaluates a Generalist Computer-Use Agent (GCUA), defined as an AI system integrating five functional layers: Brain (LLM reasoning), Eyes (GUI perception), Body (control flow), Hands (tool invocation), and Feet (code execution / file work) [2606.05405]. The target workflows are professionally sourced and end in deliverables such as CAD files, workbooks, rendered scenes, code repositories, and reports, with outcomes assessed by deterministic or narrowly scoped rubric-based verification [2606.05405].

This design choice places ALE in a different evaluative regime from short-form QA or stylized tool-use tests. A plausible implication is that ALE treats workflow completion, rather than isolated inference quality, as the core unit of capability.

## 2. Taxonomy, coverage, and task construction

ALE’s coverage is grounded in the U.S. federal occupational taxonomy, specifically O*NET and SOC 2018 [2606.05405]. The construction process screened all 1,016 O*NET occupations for in-scope digitally mediated workflows, distilled 117 candidate SOC base codes, and then, after expert grouping, LLM-assisted clustering, and a small “frontier” supplement for emerging digital fields, produced the released taxonomy [2606.05405].

The resulting structure is hierarchical.

| Level | Scale | Examples from the benchmark |
|---|---:|---|
| Top level | 13 industry clusters | Manufacturing & Industrial Operations; Biomolecular Design; Visual Media Arts; Autonomous Systems |
| Second level | 55 subdomains | CAM programming, protein design, 3D compositing, robot simulation |
| Tasks | over 1,000 unique task instances | CAD files, workbooks, rendered scenes, code repositories, reports |

Tasks enter ALE through a five-gate pipeline: expert sourcing via dedicated advisory committees; submission and AI-assisted specification refinement; first-pass conference-style review; engineering implementation through containerization, VM provisioning, and evaluator scripts; and final peer QC by domain experts [2606.05405]. Only approximately 10% of instances are public at any time, with the remainder kept private to avoid contamination; public/private splits rotate over time to maintain an uncontaminated “fresh” evaluation surface [2606.05405].

The benchmark’s industry collaboration is central rather than ornamental. Experts contributed actual day-to-day projects drawn from their own professional practice, which makes the taxonomy operationally tied to digitally mediated labor rather than to synthetic benchmark authoring [2606.05405].

## 3. Formal scoring and verification framework

Each ALE task instance $i$ produces a normalized score $s_i \in [0,1]$ via a deterministic evaluator [2606.05405]. The benchmark defines the full-pass indicator as

$$
P_i = \begin{cases}
1, & s_i = 1, \\
0, & s_i < 1.
\end{cases}
$$

and defines Full Pass Rate (FPR) over a set of $N$ tasks as

$$
\mathrm{FPR} = \frac{1}{N}\sum_{i=1}^N P_i.
$$

Mean Score (MS) is defined as

$$
\mathrm{MS} = \frac{1}{N}\sum_{i=1}^N s_i.
$$

ALE organizes tasks into three nested difficulty tiers—Near-Term, Full-Spectrum, and Last-Exam—so that evaluation cost can be traded against remaining headroom [2606.05405]. For a tier $T \in \{\mathrm{Near}, \mathrm{Full}, \mathrm{Last}\}$ with $N_T$ tasks, the tier-specific full pass rate is

$$
\mathrm{FPR}_T = \frac{1}{N_T}\sum_{i \in T} P_i.
$$

Verification is implemented through one or more of six artifact modes: exact/hash match, numeric/tolerance tables, 3D-geometry distance, behavioral state replay, free-text rubrics, or “vision-LLM” gates [2606.05405]. These can be composed through “gate-and-score,” weighted rubrics, checklist averaging, or per-file means [2606.05405]. By default, judges are code-based; only approximately 7% of tasks use narrowly scoped LLM-as-a-judge probes [2606.05405].

This scoring design formalizes a strict distinction between partial completion and end-to-end completion. In ALE, the headline statistic is not merely whether an agent made progress, but whether it fully completed the professional deliverable under the benchmark’s evaluator.

## 4. Experimental setup and baseline performance

The reported experiments evaluate mainstream harness and backbone configurations in GCUA mode, each extended with a GUI-as-Tool bridge exposing 14 desktop actions, including mouse, keyboard, and screenshot operations [2606.05405]. The harnesses listed are OpenClaw, ALE-Claw, Claude Code, Cursor, and Droid; the backbones listed are GPT-5.5, GPT-5.4, Opus 4.7, Sonnet 4.6, Gemini 3.1 Pro, DeepSeek V4, and GLM 5.1, among others [2606.05405].

The public evaluation split contains 150 tasks: 59 Near-Term, 55 Full-Spectrum, and 36 Last-Exam [2606.05405]. Reported headline results are as follows.

| Tier | Public task count | Reported result |
|---|---:|---|
| Near-Term | 59 | top FPR ≈42% |
| Full-Spectrum | 55 | top FPR ≈20% |
| Last-Exam | 36 | average FPR ≈2.6% across frontier configs |

The Near-Term top full pass rate is reported as approximately 42% for Codex + GPT-5.5, while Full-Spectrum reaches approximately 20% and Last-Exam remains at an average full pass rate of approximately 2.6% across frontier configurations [2606.05405]. The paper further notes that, despite some models reaching approximately 80% on prior CLI-only or GUI-only benchmarks, their Last-Exam FPR on ALE remains in the single-digit range [2606.05405].

These results position ALE as a headroom-rich benchmark. The hardest tier is not merely unsolved; it is far from saturated under the reported configurations.

## 5. Error sources, unsaturation, and research implications

A detailed failure-cause analysis for Claude Code + Opus 4.7 attributes approximately 75% of failures to high-level errors rather than to mechanical execution bugs [2606.05405]. The two largest categories are database gaps in domain knowledge, at approximately 31%, and flawed strategy or premature abandonment, at approximately 44% [2606.05405].

The paper interprets these findings as evidence that Last-Exam tasks require deep professional expertise, specialized tool chains, or non-trivial domain judgment, and that current agents lack the integrated domain understanding and long-horizon planning needed to close such workflows end-to-end [2606.05405]. This suggests that the dominant failure mode in ALE is not simply brittle GUI control or action-level unreliability. A plausible implication is that progress on the benchmark will depend at least as much on domain grounding, workflow decomposition, and tool-chain semantics as on low-level computer-use dexterity.

The future directions explicitly highlighted are richer domain-knowledge retrieval, hybrid neuro-symbolic planning over professional APIs, and tighter tool-chain integration [2606.05405]. These directions follow directly from the benchmark’s diagnosis that unsaturation persists primarily at the level of expertise and strategy.

## 6. Living-benchmark properties, economic orientation, and terminological disambiguation

ALE is designed as a living benchmark [2606.05405]. Its task pool grows as new workflows and subdomains are onboarded through the same five-gate expert pipeline; public tasks are periodically retired to private status and replaced by fresh private instances; and the taxonomy can evolve through extension of the frontier supplement as new digital professions emerge [2606.05405]. This structure is intended to preserve an uncontaminated challenge for each new model generation.

The benchmark’s broader purpose is to close what the paper describes as the gap between benchmark success and GDP-relevant impact [2606.05405]. Its workflows are grounded in productive labor, including legal document filings, engineering simulations, and creative-media production, and the stated premise is that saturation of ALE would signal readiness for substantive professional responsibilities rather than performance on stylized exams alone [2606.05405].

The acronym “ALE” has prior and parallel uses in the literature. In reinforcement learning, ALE commonly denotes the Arcade Learning Environment, introduced as a unified interface to Atari 2600 games and described as a “final exam” for general-purpose reinforcement learning and planning agents [1207.4708]. Later work on that benchmark emphasized evaluation protocols such as sticky actions, multi-mode support, reporting at fixed frame milestones, and human-normalized metrics [1709.06009]. Separately, agentified assessment for logical reasoning agents uses an assessor agent to issue tasks, enforce execution budgets, parse outputs, and record structured failure types under a standardized agent-to-agent interface; that framework benchmarks an auto-formalization agent on a cleaned split of FOLIO and is methodologically distinct from the industry-workflow benchmark introduced as Agents' Last Exam [2603.02788].

For this reason, “ALE” is not a unique label across AI evaluation. In current usage, Agents' Last Exam refers specifically to the 2026 benchmark of long-horizon, economically valuable, real-world digital workflows with verifiable outcomes [2606.05405].

Source: https://www.emergentmind.com/topics/agents-last-exam-ale