APEX: AI Productivity Benchmark

• APEX is a benchmark that measures AI model performance on economically valuable, real-world tasks in domains such as law, consulting, investment banking, and medicine.
• It employs an expert-driven methodology with domain-specific prompts and detailed rubrics to objectively assess complex knowledge work.
• Current findings show models achieving around 64% of expert performance, highlighting a gap that underscores the need for domain-specific AI advancements.

The AI Productivity Index (APEX) is a benchmark specifically designed to evaluate whether advanced AI models can perform economically valuable knowledge work. Distinct from most existing benchmarks—which tend to measure abstract reasoning or factual recall—APEX operationalizes the assessment of AI model performance on real-world tasks generated and reviewed by experts from high-value professional domains. The introduction of APEX addresses the gap between economic relevance and traditional AI evaluation, prioritizing tasks that reflect the substantive knowledge work found in sectors like finance, consulting, law, and medicine (Vidgen et al., 30 Sep 2025).

1. Conceptual Motivation and Scope

APEX emerges in response to a recognized shortcoming in AI evaluation, namely the lack of benchmarks that directly test capabilities relevant to real-world, economically significant knowledge work. Most prominent existing suites fail to capture advanced reasoning, synthesis, and communication as required by highly skilled professionals outside the context of software development. APEX explicitly focuses on four vital domains: investment banking, management consulting, law, and primary medical care. These sectors are characterized by tasks that typically take experienced practitioners several hours to complete and have clear economic and societal implications. Unlike academic question-answering or synthetic challenge sets, APEX tasks are constructed to mirror the complexity, nuance, and regulatory requirements that define high-value expert output.

2. Methodology and Benchmark Construction

The construction of APEX follows a three-stage, expert-driven protocol:

1. Expert Sourcing: Experts were selected for their firsthand experience in elite firms or institutions within each domain—for example, investment bankers from Goldman Sachs. Selection was based on interviews and trial evaluation to ensure domain mastery.
2. Prompt and Scenario Generation: Each expert was tasked with authoring prompts that reflect challenging, day-to-day work assignments (e.g., writing detailed legal memoranda, conducting management consulting analysis, preparing complex investment casework, or synthesizing clinical plans in medical care).
3. Rubric Design and Objective Evaluation: For every prompt, experts authored detailed rubrics specifying objective, pass/fail evaluative criteria. Each rubric is analogous to a suite of unit tests, operationalizing requirements such as correct citations (e.g., referencing 17 U.S.C. § 101 in law), regulatory compliance, structured reasoning, and complete technical deliverables.

A sample scoring formula for an individual prompt is: $\text{Score} = \frac{m}{N} \times 100\%$ where $m$ is the number of criteria passed and $N$ is the total number of rubric criteria. This approach ensures fine-grained, interpretable measurement of model outputs with minimal ambiguity.

3. Evaluation Protocol: Automatic Judging and Domains

Each model’s output is scored by a committee of three large model (LM) judges, each with calibrated “thinking” settings. Judgments are made by majority vote on pass/fail basis for each rubric item. Internal consistency among LM judges exceeds 99%, and agreement with human expert re-grading stands at approximately 89%. This level of reliability supports the use of automated evaluation at scale.

The domains covered and their characteristics are:

Domain	Example Task	Avg. Rubric Criteria
Investment Banking	Drafting DCF models, market analysis	~29
Management Consulting	Strategic reporting, business insight	~29
Law	Drafting memoranda, compliance reviews	~29+ (detailed legal)
Primary Medical Care	Synthesizing diagnosis/plans	>29 (complex, regulated)

Each prompt is designed to require between 1 and 8 hours of expert time, with a mean of 3.5 hours, ensuring the tasks mirror those with meaningful economic impact in professional contexts.

4. Model Performance and Comparative Analysis

APEX-v1.0 evaluated 23 frontier AI models, encompassing both proprietary and open-source systems. Notable performance metrics:

• Top models: GPT 5 (Thinking = High) scored 64.2% on average; Grok 4 scored 61.3%; Gemini 2.5 Flash (Thinking = On) scored 60.4%.
• Open-source: Qwen 3 235B achieved best-in-class among open models, ranking seventh overall.
• Domain variation: Performance is highest in law (~70.5% for the best models) and lowest in medicine (~47.5%), likely due to the greater complexity and regulatory stringency of medical prompts.
• Pairwise win rate analysis: GPT 5 wins 77.5% of head-to-head prompt comparisons, while Phi 4 wins only 4.3%.
• Statistical validation: Kruskal–Wallis significance test ($\alpha = 0.00001$) confirms substantial score differentiation among models.

Together, these results highlight both the current state-of-the-art and the substantial gap remaining between model and expert human performance.

5. Criteria Design and Economic Significance

A critical feature of APEX is the detailed, objective decomposition of complex quality notions into tractable, discrete criteria. Example criteria from the law domain include the correct identification of legal issues, citation of statutory language, and adherence to strict stylistic and word-limit protocols. This decomposition enables robust, scalable, and transparent scoring.

The economic significance of APEX lies in its unique alignment with actual workflows in high-value domains. Rather than generic reasoning, it evaluates practical, revenue- and outcome-centric capabilities. As such, APEX scores more closely track the model’s ability to augment or substitute for expert workforces in sectors with high wage premiums and stringent quality demands.

6. Implications for Benchmarking, Policy, and Research

APEX exposes a material performance gap between frontier AI models and human experts. Even the most advanced models achieve only about 64% of the possible score, with substantial variability across domains and tasks. This result implies that advancements in LLM “thinking” modes still yield at most moderate performance improvements, and that further research must focus on domain-specific reasoning, evidence synthesis, and regulatory compliance.

For the research community, APEX provides a measurable, economically grounded target for model optimization. For product development, it offers a direct link between AI research advances and real-world automation value. Policy and industry actors may use APEX to assess readiness for AI integration in critical knowledge domains.

7. Future Directions and Outstanding Needs

The gap observed on APEX benchmarks between models and human experts underlines several challenges:

• Progress in domain adaptation, long-horizon planning, multimodal evidence processing, and nuanced synthesis are necessary to close the gap.
• There is a need for further expansion of domain coverage, rubric complexity, and source document diversity to keep pace with evolving economic trends.
• Improved benchmarking frameworks that refine the mapping between quantitative performance on APEX and actual labor force impact will further connect AI research to productivity policy.

APEX, by design, sets a new standard in AI benchmarking—one grounded in the demands of knowledge-intensive economic activity. Its rigorous evaluation protocol, expert-authored prompts and rubrics, and transparent scoring formula enable precise, reproducible measurement of the economic value generated by frontier AI systems (Vidgen et al., 30 Sep 2025).