---
title: 'FinAgentBench: Agentic Retrieval in Finance'
url: https://www.emergentmind.com/topics/finagentbench
type: topic
---

# FinAgentBench: Agentic Retrieval in Finance

FinAgentBench is a benchmarking suite designed to rigorously evaluate information retrieval and agentic reasoning capabilities of large language models (LLMs) and LLM-based agents in specialized financial workflows. It introduces a modular, interpretable evaluation protocol centered on multi-step retrieval over high-value financial disclosures, with an annotation framework and metric suite tailored to the complexity and domain-specificity of professional financial research. As the first large-scale benchmark for agentic evidence retrieval in finance, FinAgentBench has fostered extensions across enterprise accounting, wealth management, and execution-grounded safety assessment.

## 1. Problem Motivation and Agentic Retrieval Paradigm

Financial research demands extraction of precise, context-dependent information from extensive, heterogeneous filings (10-K, 10-Q, 8-K, earnings transcripts, DEF-14A). Traditional retrieval pipelines—both sparse (e.g., BM25, TF–IDF) and dense neural embeddings—are frequently inadequate for two central reasons:

- **Semantic fine-grainedness**: Financial concepts (e.g., “proxy statement” vs. “DEF-14A”; GAAP vs. non-GAAP metrics) require domain alignment and structural understanding, not simple keyword overlap or sentence embeddings.
- **Document scale and latency**: Indexing and querying thousands of pages or tables per firm result in prohibitive context windows or information loss due to truncation.

FinAgentBench formalizes *agentic retrieval* as a modular, two-step process:
1. **Document Type Selection**: Given a query $q$, select the most relevant filing type $t^* \in T$ via $s_1(t|q)$.
2. **Chunk Pinpointing**: Within $d_{t^*}$, score and rank paragraph- or table-level “chunks” $c_i$ using $s_2(c_i|q, d_{t^*})$.

This decomposition mirrors analyst workflows and yields interpretable stages for error attribution and model analysis [2508.14052, 2511.14130].

## 2. Dataset Construction and Annotation Protocol

FinAgentBench provides 3,429 expert-annotated query-document pairs for S&P-100 firms, spanning five SEC filing types. The construction protocol comprises:

- **Query Generation**: Two finance professionals independently draft ten queries for each company, cubered over ten categories (e.g., Analyst Q&A, Risks, Operating Metrics), with cross-validation to final consensus. Each query is designed to invoke nuanced, context-driven searching.
- **Document and Chunk Preprocessing**: >15,000 filings are split into paragraph-level chunks, with each table treated as a single unit to preserve semantic coherence.
- **Dual-Phase Annotation**: (1) Filing types are ranked per query using a drag-and-drop interface; (2) every chunk within the top-ranked document is graded on a TREC-scale (0=irrelevant, 1=partially relevant, 2=directly relevant), with adjudicated dual annotation for label consistency.

Key dataset statistics:
- Average relevant document types per query: 2.92
- Average directly relevant chunks: 6.41
- Average partially relevant chunks: 9.24 [2508.14052].

## 3. Evaluation Protocol and Metrics

FinAgentBench adheres to a rigorous information retrieval metric regime at both document and chunk levels over the top-5 predictions per stage:

- **Precision@k (P@k)**: $\frac{1}{k} \sum_{i=1}^k \mathbb{1}[rel_i > 0]$
- **Recall@k (R@k)**: $\frac{1}{|G|} \sum_{i=1}^k \mathbb{1}[rel_i > 0]$, $|G|$ relevant items
- **Mean Reciprocal Rank (MRR@k)**: $\frac{1}{|Q|} \sum_q 1/\text{rank}_q$
- **Mean Average Precision (MAP@k)**: $\mathrm{MAP@k} = \frac{1}{|Q|} \sum_q \frac{1}{|G_q|} \sum_{i=1}^k P@i \cdot \mathbb{1}[rel_{q,i}>0]$
- **nDCG@k**: $DCG@k = \sum_{i=1}^k \frac{2^{rel_i}-1}{\log_2(i+1)}$, normalized by the ideal DCG [2508.14052, 2511.14130].

This layered metric suite enables direct attribution of bottlenecks—whether in document scoping or intra-document localization—with all relevance labels and permutation ground truths derived from expert annotation.

## 4. Agent Baselines, Fine-Tuning, and Performance Analysis

FinAgentBench benchmarks the retrieval-centric performance of leading LLMs in strict zero-shot, retrieval-only settings. Representative results include [2508.14052, 2511.14130]:

| Model            | Doc nDCG@5 | Doc MAP@5 | Doc MRR@5 | Chunk nDCG@5 | Chunk MRR@5 |
|------------------|-----------:|----------:|----------:|-------------:|------------:|
| Claude-Sonnet-4  | 0.783      | 0.849     | 0.892     |   0.419      |   0.567     |
| GPT-o3           | 0.770      | 0.829     | 0.875     |   —          |   —         |

Chunk-level retrieval is consistently less accurate, highlighting the intrinsic challenge of pinpointing sparse, heterogeneously distributed relevant evidence under tight window constraints. Targeted reinforcement fine-tuning on GPT-o4-mini (90% of training split) yields robust stage-wise improvements: Doc nDCG@5 rises from 0.758 to 0.808, and MRR@5 from 0.872 to 0.933; chunk nDCG@5 from 0.345 to 0.371, and MRR@5 from 0.526 to 0.587. These results empirically establish that even strong LLMs benefit from domain-specific, task-aligned supervision.

## 5. Extensions Across Financial Workflows and Agentic Reasoning

The agentic retrieval paradigm of FinAgentBench has catalyzed a spectrum of dataset extensions covering enterprise finance and accounting [2512.13168], wealth management [2512.02230], and execution-grounded agent security [2601.07853].

- **Enterprise Financial Workflows (Finch)**: Tasks span multi-modal artifacts in Enron-style real-world spreadsheet and email corpora, interleaving structuring, cross-file retrieval, calculation, validation, and translation. Workflow Pass Rate and Task Success Rate are primary metrics, with GPT-5.1 Pro achieving a 38.4% pass rate. Steep performance decrements are observed on workflows exceeding two compositional subtasks [2512.13168].
- **Wealth Management (Closed-World Evaluation)**: Benchmarks apply paired high- and low-autonomy variants across retrieval, analytic, and communication workflows, with deterministic grading via multi-checkpoint validators. Metrics include overall accuracy, reliability, and API cost per task, facilitating Pareto analysis of autonomy vs. economic efficiency [2512.02230].
- **Security in Execution-Grounded Environments**: The FinAgentBench extension within FinVault formalizes 31 auditable scenarios with mutable state, cross-domain tool call privileges, 107 regulatory vulnerabilities, and attack taxonomies including prompt injection, jailbreaking, and authority impersonation. Metrics such as Attack Success Rate and Vulnerability Compromise Rate highlight persistent agentic safety gaps (ASR up to 50%); even the most robust models manifest a VulnRate >20% [2601.07853].

## 6. Principal Data-Driven Insights and Open Problems

Evaluations on FinAgentBench and its extensions reveal:

- LLMs exhibit strong reporting priors at the document-type selection stage but degrade sharply at fine-grained chunking due to context fragmentation and evidence sparsity [2508.14052, 2511.14130].
- Domain-specific tuning and structured prompts with explicit reasoning scaffolds yield significant returns, while uninformed extension of in-context learning (ICL) to all retrieval stages can be detrimental [2511.14130].
- Real-world, multi-modal enterprise workflows expose persistent limitations in layout-awareness, formula interpretation, cross-file reasoning, and multimodal fusion, which current agent architectures struggle to surmount [2512.13168].
- For security and compliance, traditional safety alignments fail to transfer, with role-playing and authority-impersonation attacks dominating even in tightly sandboxed, auditable environments [2601.07853].

Open challenges include scaling to full S&P 500 coverage, enriching for temporal and comparative queries, simulating collaborative and adversarial workflows, quantifying scenario severity, and jointly integrating retrieval with answer-generation in “truly agentic” assistants.

## 7. Implications and Future Directions

FinAgentBench establishes a foundation for research into compositional reasoning, interpretable error analysis, and rigorous, domain-grounded evaluation of LLM agents in financial contexts. Forthcoming releases will extend to the S&P 500, introduce longitudinal analyses across reporting periods, and investigate unified agentic protocols combining retrieval, answer selection, and compliance validation [2508.14052].

Key recommendations for future systems include explicit modeling of domain policies, introduction of tool-augmented actions (e.g., for formula computation and chart parsing), robust prompt isolation for safety, and granular measurement of cost and fidelity in high-stakes financial workflows. These directions aim to bridge persistent capability gaps in evidence-centered reasoning, agentic orchestration, and operational security in applied financial AI.

Source: https://www.emergentmind.com/topics/finagentbench