Agents' Last Exam

Abstract: Recent AI systems have achieved strong results on a wide range of benchmarks, yet these gains have not translated into economically meaningful deployment across many professional domains. We argue that this gap is largely an evaluation problem: widely used benchmarks lack sustained performance measurement on real and economically valuable workflows. This paper introduces Agents' Last Exam (ALE), a benchmark designed to evaluate AI agents on long-horizon, economically valuable, real-world tasks with verifiable outcomes. Developed in collaboration with 250+ industry experts, ALE covers non-physical industries defined with reference to O*NET / SOC 2018 (the U.S. federal occupational taxonomy). It is organized around a task taxonomy with 55 subfields grouped into 13 industry clusters covering 1K+ tasks. Current results show that the hardest tier remains far from saturated: across mainstream harness and backbone configurations, the average full pass rate is 2.6%. ALE is designed as a living benchmark: its task pool grows continuously as new workflows and industries are onboarded. More broadly, ALE is intended not merely as another leaderboard, but as an instrument for closing the gap between benchmark success and GDP-relevant impact.

• The paper introduces ALE, a benchmark that evaluates economic value realism in generalist AI agents through authentic professional workflows.
• It outlines a multi-stage evaluation pipeline with deterministic scoring and strict rubric checks to assess agents over diverse, complex tasks.
• Empirical results reveal low pass rates on high-difficulty tasks, highlighting significant performance gaps in current agent architectures.

Agents' Last Exam: Benchmarking Economic Value Realism in Generalist AI Agents

Introduction

"Agents' Last Exam" (ALE) (2606.05405) introduces a benchmark targeting the persistent disconnect between rapid progress on AI leaderboards and the lack of palpable economic deployment across professional domains. Existing benchmarks emphasize short-horizon, synthetic, or answer-based tasks, insufficient for measuring the long-horizon, compositional workflows characteristic of real professional practice. ALE responds by defining an evaluation platform constructed from authentic domain-expert workflows, scoring generalist agents on tasks whose successful completion would directly correspond to economically valuable outputs.

ALE comprises 1,000+ task instances spanning 55 subfields within 13 industry clusters, referencing SOC 2018 and O*NET occupational taxonomies, and is supported by an advisory committee of over 250 industry experts. Tasks are derived from completed real-world projects and undergo a stringent validation and review pipeline. The benchmark is intentionally "living," supporting ongoing expansion as new workflows and domains are incorporated.

Figure 1: ALE encompasses a comprehensive taxonomy of real professional tasks and software-mediated workflows, ensuring relevance to genuine economic activity.

Benchmark Scope and Task Construction

ALE fundamentally distinguishes itself by three selection criteria for task inclusion:

1. Representativeness: Each workflow mirrors actual professional practice, utilizing toolchains professionals currently employ. Tasks are drawn from previous domain-expert projects with no synthetic simplification.
2. Complexity: Tasks must be end-to-end deliverables requiring sustained operation (e.g., days to weeks of expert labor), as opposed to atomic UI or scripting actions.
3. Verifiability: Each deliverable must be objectively checkable via deterministic scripts or tight rubrics, minimizing the role of subjective judgment.

Tasks traverse a multi-stage construction pipeline that includes expert proposal, conference-style peer review, engineering for executable assets, and final committee-level quality control. Only a fraction of tasks are public at any time to mitigate contamination and preserve long-term challenge validity; the rest remain private and are periodically rotated into the public evaluation set.

Figure 2: Distribution of task instances across ALE’s 55 subdomains, showing nonzero coverage throughout the taxonomy.

Task Taxonomy and Industry Grounding

ALE's taxonomy is grounded in the SOC 2018 and O*NET systems, consolidating digitally mediated professional workflows into 13 sectors (e.g., finance, engineering, health, visual media), each subdivided into workflow-level subdomains. The taxonomy is expanded to cover emerging fields absent from SOC/O*NET but relevant to contemporary professional and technical practice.

The benchmark substantially enlarges previous coverage: even the union of 16 leading agentic and QA benchmarks leaves 13/55 subdomains unrepresented, whereas ALE ensures nonzero instantiation in each.

Evaluation Pipeline and Agent Architecture

The ALE pipeline separates the task specification, agent, and environment. Tasks are defined as Python scripts encapsulating all instructions, input/output artifacts, software requirements, and scoring routines, which are then instantiated in cloud-hosted virtual machines. The environment enforces modularity and reproducibility via a standard filesystem layout: input/, software/, output/, and reference/ directories.

Agents in ALE must operate as Generalist Computer-Use Agents (GCUA), with the following compositional capabilities:

• Brain: LLM-based reasoning and planning.
• Eyes: Multi-modal (screenshot) perception for GUI workflows.
• Body: Orchestration and session management.
• Hands: Tool invocation (shell, scripts, API, mouse/keyboard macros).
• Feet: Native runtime and OS-level interaction.

Most existing benchmarks are restricted to CLI or GUI interaction, but not both. ALE’s agent substrate and harnesses execute the full surface, requiring, for example, seamless alternation between code execution, GUI application manipulation, file system operations, and web research.

Figure 3: Decomposition of functional capability layers for evaluated agents, distinguishing GCUAs from more primitive CLI- or GUI-only agents.

Scoring, Evaluation Modes, and Automation

Scoring in ALE is almost entirely deterministic and artifact-based. Evaluations include exact/hashed value comparison, structured tabular/numeric tolerances, spatial similarity for 3D assets, behavioral outcome comparisons, and, when necessary, targeted LLM vision-model probes limited to gates or atomic checklist items (e.g., visual confirmation of a rendered artifact). Only in cases where code-based checks are impossible (<7%) is an LLM-as-judge deployed, and always as a narrow binary probe rather than holistic assessment.

Aggregation follows robust schemes (gated scoring, weighted rubrics, checklists) to ensure interpretability and resistance to superficial reward hacking.

Empirical Results

Evaluation on the current public set (three difficulty tiers: Near-Term, Full-Spectrum, Last-Exam) reveals substantial unsolved challenge:

• Across mainstream agent harnesses and foundation models, the average pass rate on the hardest tier (Last-Exam) is only 2.6%.
• Even the strongest observed configuration (Codex harness + GPT-5.5 backbone) achieves 42% pass on Near-Term but drops below 9% pass on Last-Exam.
• For comparison, the same configuration easily saturates prior leaderboards such as Terminal-Bench (82% pass).

This steep drop confirms that ALE's workflows are properly out of distribution relative to both pretraining data and prior synthetic/curated benchmarks.

Performance bottlenecks are further analyzed via domain-level scores, tool-usage patterns, and failure taxonomy:

• Best results are observed in computational math and environment/agriculture, weakest in visual media and education.
• GUI tool use is lower than task demand: agents systematically attempt GUI-centered workflows using CLI surrogates, highlighting gaps in embodied visual interaction.
• Root-cause analysis classifies ~75% of failures as arising from "Understanding" or "Approach" problems: i.e., fundamental knowledge or strategy gaps, not routine execution bugs.

Figure 4: Analysis of model performance by domain, tool utilization, and failure cause; approach/strategy and domain-understanding failures dominate.

Figure 5: Variation in pass rate is controlled predominantly by foundation-model choice, with a threefold effect size over agent harness engineering.

Figure 6: Resource usage (tokens, time, cost) varies non-monotonically with performance, indicating that efficiency and capability are not tightly coupled in current systems.

Comparative Coverage and Software Ecosystem

ALE spans a unique application/software ecosystem, supporting workflows realized in over 50 distinct desktop and domain-specific applications, covering everything from standard office and engineering suites to VFX and animation packages. This ensures that agents are evaluated on domains with genuine economic significance, not just academic or hobbyist tasks.

Figure 7: Venn-style overview of the software ecosystem covered by ALE tasks, illustrating the intersection among professional tools and ALE taxonomy domains.

Public Subset Representativeness

The public task subset is empirically shown to be representative of the full private pool, both in domain coverage and in pass-rate distribution, with strong correlation ($r=0.89$) across taxonomy clusters.

Figure 8: Cross-cluster pass rates for the public subset vs. the full task pool confirm representativeness—critical for ongoing leaderboard validity.

Implications and Future Directions

ALE demonstrates that contemporary GCUAs—including those based on recent frontier LLMs—remain far from achieving robust, tool-intensive, economically valuable automation over long-horizon workflows. The main implication is that prior "benchmark saturation" is not even close for high-value economic tasks; passing rates below 10% signal a large unsolved gap.

Given the tight linkage between benchmark design and AI research focus, ALE is positioned to recalibrate incentives toward genuinely impactful agentic systems. The modular, living design allows for continuous expansion in both task diversity and difficulty, promising to drive progress on fidelity, generalization, and integrated multi-modal/compositional workflows.

The predominance of knowledge/reasoning errors over execution/format failures indicates that foundation model improvements—particularly in vision-grounding, domain modeling, and integrated planning—are likely to yield the greatest near-term advances. Harness engineering and context/window management, while not insignificant, deliver smaller returns at current system scales.

Conclusion

ALE offers a comprehensive, rigorously validated platform for closing the gap between synthetic benchmark completion and economically consequential agent deployment. Through expansive professional domain coverage, rigorous task construction, and strict artifact-based evaluation, ALE forces AI agents to confront the actual demands of digital professional labor. With pass rates on authentic long-horizon tasks still near zero for most leading agents, ALE sets the agenda for AI system design and evaluation for the foreseeable future, moving the community toward practical, GDP-relevant autonomy.