BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows

Abstract: Existing AI benchmarks lack the fidelity to assess economically meaningful progress on professional workflows. To evaluate frontier AI agents in a high-value, labor-intensive profession, we introduce BankerToolBench (BTB): an open-source benchmark of end-to-end analytical workflows routinely performed by junior investment bankers. To develop an ecologically valid benchmark grounded in representative work environments, we collaborated with 502 investment bankers from leading firms. BTB requires agents to execute senior banker requests by navigating data rooms, using industry tools (market data platform, SEC filings database), and generating multi-file deliverables--including Excel financial models, PowerPoint pitch decks, and PDF/Word reports. Completing a BTB task takes bankers up to 21 hours, underscoring the economic stakes of successfully delegating this work to AI. BTB enables automated evaluation of any LLM or agent, scoring deliverables against 100+ rubric criteria defined by veteran investment bankers to capture stakeholder utility. Testing 9 frontier models, we find that even the best-performing model (GPT-5.4) fails nearly half of the rubric criteria and bankers rate 0% of its outputs as client-ready. Our failure analysis reveals key obstacles (such as breakdowns in cross-artifact consistency) and improvement directions for agentic AI in high-stakes professional workflows.

• The paper introduces BankerToolBench as a novel benchmark that assesses AI agents across full-cycle investment banking workflows with expert-curated rubrics.
• Methodologically, it employs multi-artifact deliverables and an agent-as-a-judge evaluation to simulate real-world tasks like DCF analysis and pitchbook creation.
• Results show frontier models, including GPT-5.4, display low client-readiness due to errors in code generation, inconsistent reasoning, and challenges in long-horizon planning.

Evaluating LLM Agents in Investment Banking: An Analysis of BankerToolBench

Motivation and Benchmark Overview

BankerToolBench (BTB) addresses an acute deficit in high-fidelity benchmarks for evaluating the real-world utility of AI agents in professional, economically consequential domains. Focusing on junior investment banking workflows, BTB is the first open-source benchmark to require agents to complete end-to-end analytical tasks encompassing unstructured requests, authentic data rooms, industry tool access, and the generation of multi-artifact deliverables. Constructed with direct input from 502 investment bankers, BTB is firmly grounded in an ecological validity paradigm, spanning realistic tasks such as discounted cash flow analyses, leveraged buyout models, and pitchbook preparation—each requiring heterogeneous deliverables processed by RL agents in a sandboxed environment.

Figure 1: BTB benchmark structure spans request intake, tool use, deliverable generation, and agentic rubric-driven evaluation.

Each BTB task is evaluated via fine-grained rubrics curated by veteran practitioners, typically featuring around 150 pass/fail criteria weighted by criticality. Automated evaluation is performed using an agent-as-a-judge verifier that leverages LLM-powered code execution to inspect deliverables—a necessity given the complexity of artifacts such as Excel workbooks or PowerPoint presentations, which cannot be reliably scored by serialization-based approaches or simplistic text-based verifiers.

Benchmark Construction and Fidelity

BTB explicitly mitigates the limitations observed in previous financial or economic utility benchmarks (e.g., GDPval, APEX-Agents, Finance Agent Benchmark) by:

• Focusing on depth within a single high-impact domain, rather than breadth across multiple occupations.
• Requiring multi-file, interdependent deliverables typical of IB, with rigorous human-in-the-loop task and rubric review.
• Using stakeholder-derived rubric design, supported by a job task analysis (JTA) survey and interviews with actual bankers to calibrate the blueprint for representative work.
• Including edge-case complexity, e.g., ambiguous or underspecified prompts, judgment-heavy requirements, and data imperfections, yielding tasks that demand synthesis, precision, and cross-artifact consistency.

A multi-phase data validation pipeline ensures that each BTB task, from prompt to final scoring, undergoes stringent curation and review for realism, completeness, rubric coverage, and grading fidelity.

Figure 2: The BTB data pipeline, encompassing task authorship, rubric creation, expert review, and rigorous acceptance criteria.

Experimental Setup and Evaluation

Nine frontier models, including GPT-5.2/5.4, Claude Opus 4.5/4.6, Gemini 2.5/3.1 Pro, Grok 4, Qwen 3.5 397B, and GLM 5, are systematically evaluated under a standardized agentic harness. Deliverables are automatically scored using BTB's rubric-driven, agentic verifier. The principal metrics are: (1) Score (weighted rubric pass percentage), (2) Pass@1 and Pass@3 (fraction of tasks above stakeholder-aligned acceptability thresholds), and (3) more stringent criticality-based thresholds corresponding to client-readiness.

Results: Performance and Failure Analysis

Frontier models demonstrate nontrivial but inadequate proficiency:

• GPT-5.4 achieves the highest Score, passing roughly 16% of all tasks at an "acceptable" standard, but fails nearly half of rubric criteria and does not produce any outputs rated client-ready by bankers.
• No model achieves full reliability; even at highly relaxed thresholds, Pass rates rarely exceed 20%.

  Figure 3: Pass rates (tasks deemed "acceptable") for all models under single and best-of-three rollouts, reflecting low reliability.

  

  Figure 4: Left: Mean Score across all BTB tasks. Right: Pairwise win-rate matrix across model pairs, showing consistent superiority of GPT-5.4.

Failure mode taxonomy indicates the majority of failures result from code or formula generation errors (e.g., use of non-existent APIs, hard-coding vs. formulaic logic in spreadsheets), flawed reasoning/judgment (incorrect application of IB concepts or analytical frameworks), ineffective retrieval/persistence strategies, and unprincipled data fabrication when retrieval fails.

Models reveal marked variation across rubric categories:

Figure 5: Model-wise performance breakdown by rubric category, with GPT models generally excelling at technical correctness but underperforming on presentation attributes.

• GPT-5.4 leads in technical and internal consistency, but lags Opus 4.6 in client readiness and compliance.
• All models exhibit significant brittleness in chained, long-horizon workflows—e.g., inconsistent propagation of assumptions, incoherent reconciliation between artifacts, and premature halting.

Agentic harness implementation has a modest impact on Score (within ±2 points), but significant effect on runtime and cost, with OpenCode being more efficient than OpenHands or Goose.

Task Difficulty, Domain Knowledge, and Sensitivity

Task characteristics strongly modulate model performance:

Figure 6: Score stratified by expected deliverable type, input file type, and IB product group. Model performance systematically varies with task complexity and artifact requirements.

• PowerPoint-centric and ECM tasks are easier for current agents than quantitative, model-heavy analyses (e.g., DCF/LBO/merger models), particularly those demanding formula integrity and scenario toggling.
• Providing richer IB-specific context (domain and formatting conventions) to prompts increases agent performance, suggesting acute sensitivity to prompt engineering and model pretraining/curriculum.

Post-training, Reward Signal Quality, and Human Alignment

BTB’s rubric Score serves as an effective objective for RLHF-based post-training. Applying Dr. GRPO and DPO to Qwen 3 4B/32B yields substantial ($4–13\times$) relative improvements in holdout scores. The verifier demonstrates 88.2% agreement with human grades, comparable to inter-human agreement, and cost-effective configuration (Gemini 3 Flash Preview).

Theoretical and Practical Implications

BTB repositions the debate on LLM/agent economic utility: current models are not ready for full delegation in high-consequence professional domains. Their observed failure regimes (local errors triggering global deliverable invalidity) highlight that current progress on isolated capabilities—e.g., tool use, chain-of-thought reasoning—does not propagate reliably to end-to-end workflows. Task decomposition and human-in-the-loop paradigms are thus necessary (but ultimately inefficient for true labor substitution in IB or similar domains).

The benchmark's design principles—domain immersion, expert-aligned rubrics, multi-artifact delivery, and agentic verification—establish a template for evaluating LLM agents where “unit tests” are inadequate surrogates for economic or stakeholder value.

Conclusion

BankerToolBench constitutes a new standard for evaluating AI agents in high-stakes, knowledge-intensive workflows. Despite improvements on fine-grained capability metrics, frontier models remain brittle on holistic, cross-artifact investment banking work. The benchmark’s results underscore the need for advances in long-horizon planning, complex artifact generation, and the integration of domain-specific judgment. Wide adoption of environments constructed with BTB's principles will be crucial for aligning AI evaluation—and ultimately training—with genuine economic value and deployment readiness.