---
title: 'FinanceBench: Financial QA Benchmark'
url: https://www.emergentmind.com/topics/financebench
type: topic
---

# FinanceBench: Financial QA Benchmark

FinanceBench is a large-scale, ecologically valid benchmark for evaluating the question-answering capabilities of large language models and retrieval-augmented generation (RAG) systems over real-world financial documents. Introduced by Islam et al. (2023), FinanceBench comprises expert-annotated questions, gold-standard answers, and evidence strings linked to U.S. SEC filings. It has emerged as a principal testbed for both open-domain and domain-adapted financial QA models, driving methodological advances in information retrieval, LLM instruction tuning, neurosymbolic hybrid architectures, and vision-enhanced document intelligence systems.

## 1. Dataset Definition, Construction, and Scope

FinanceBench was designed to provide a rigorous, transparent, and challenging “open-book” QA standard for financial documents [2311.11944]. Its construction involved the curation and annotation of 10,231 question/answer/evidence triplets spanning 361 public company filings (10-Ks, 10-Qs, 8-Ks, earnings releases) from 2015–2023, covering 40 major U.S.-listed firms. Each entry includes:

- A human-generated question (e.g., “What is Boeing’s FY2022 cost of goods sold (in USD millions)?”)
- A gold-standard answer string
- An evidence span—typically a precise sentence or two from the filing, with an associated page number
- (Optionally) a free-text justification for chain-of-thought or multi-step calculation

Question types are drawn from authentic financial analysis workflows:
- Information extraction (lookup)
- Numerical reasoning (calculating financial/reporting metrics)
- Logical inference (fact synthesis, trend detection, or compliance assessment)

The question taxonomy includes: domain-relevant hand-crafted questions, realistic analyst-generated items, and metrics-generated queries reflecting both extractive and multi-step arithmetic patterns. Approximately 28% of questions demand only simple data extraction, 66% require numerical calculation, and 6% involve explicit logical reasoning. Around 85% of numerical questions are directly answerable from a single statement, while the remainder require aggregation across multiple report sections.

A commonly used open-source mini-benchmark is the 150-case subset, which is stratified by question type (50 each: domain-relevant, novel generated, metrics-generated) and is heavily referenced in contemporary QA research [2311.11944, 2411.07142, 2510.24402].

## 2. Evaluation Protocols, Metrics, and Judging Paradigms

FinanceBench adopts a rigorous evaluation protocol. For most settings, the QA system is provided either with the full relevant document, retrieved page or chunk, or, in the oracle condition, the gold evidence page [2311.11944]. The output is compared against the gold-standard answer using deterministic or LLM-graded criteria.

**Primary metrics:**
- **Accuracy:** For N_total questions, 
  $$
  \mathrm{Accuracy} = \frac{N_{\text{correct}}}{N_{\text{total}}}
  $$
  A response is “correct” only if it matches the gold answer including units and values, with minor tolerance for rounding.

- **For IR/RAG settings:**
  - **Precision@k** and **Recall@k** over retrieved chunks/pages.
  - **MRR@k:** Mean reciprocal rank of the first relevant chunk.
  - **DCG@k, NDCG@k:** Discounted/Normalized Cumulative Gain for ranking retrieval candidates.

**Advanced Judging:**
- Several studies (notably those with LLM-based judges) employ a 1–10 scale, where three independent judge scores per answer are averaged and reported as percentages [2409.13749].

**Other metrics in research practice:**
- **Consistency** (frequency of repeatable/correct outputs) [2410.02823]
- **Faithfulness** and **Hallucination** rates (claims supported by retrieved evidence) [2510.24402]
- **ROUGE-L** and **cosine similarity** (semantic matching, especially in fine-tuning studies or RAG-enhanced QA) [2402.01722]

## 3. Retrieval-Augmented Generation (RAG) and Embedding Evaluations

FinanceBench is a core benchmark in the evaluation of RAG pipelines and finance-adapted embedding models. Its challenging nature stems from the sparsity and cross-referencing of supporting evidence in lengthy, multi-modal filings:

- **Vector store evaluations:** Systems are benchmarked under "shared vector store" (single global index for all filings) and "single vector store" (per-document index) modalities [2311.11944, 2411.07142].
- **Embedding models:** OpenAI ada-002, text-embedding-3-small/large, BGE, Multilingual-E5, and domain-finetuned bi-encoders are commonly evaluated [2411.07142, 2512.08088].
- **Domain-adapted embeddings:** BAM (finetuned on 14.3M finance query-passage pairs), LLM-distilled bi-encoders, and hybrid retrieval (dense + sparse or metadata-augmented) approaches all show substantial accuracy or recall boosts—e.g., BAM yields an 8 percentage-point lift over ada-002 (55% vs. 47%) [2411.07142], and LLM-distilled retrievers increase NDCG in three of four FinanceBench document classes [2512.08088].
- **Pre-/post-retrieval enhancements:** Query rewriting, chunk metadata augmentation, pre-filtering using LLM-generated file summaries, and post-retrieval re-ranking (including cross-encoder and metadata rerankers) are shown to improve both retrieval fidelity and answer support [2510.24402].

A comparison table of representative FinanceBench IR/RAG results:

| Model/Method                       | Accuracy (%) | NDCG | Comments                         |
|-------------------------------------|--------------|------|----------------------------------|
| OpenAI ada-002 (shared store)       | 47           | —    | Baseline dual-encoder            |
| BAM Embeddings (domain-adapted)     | 55           | —    | +8 pp over ada-002               |
| Metadata-RAG, best rerank+metadata  | 44.4 (F1)    | —    | +35% relative over naive         |
| LLM-distilled retriever (NDCG)      | —            | 0.60 | best on 10-Q; +0.08 over base    |

## 4. Advanced Architectures: LLM Fine-tuning, Instruction Tuning, and Hybrid Methods

Recent work demonstrates that end-to-end QA accuracy on FinanceBench is most significantly advanced by targeted fine-tuning of both the retrieval mechanism and the LLM generator [2404.11792]:

- **Supervised fine-tuning:** Even with 10–50 example pairs from FinanceBench, both LLM and embedding model fine-tuning boosts ROUGE-L and semantic similarity substantially (e.g., LLaMA-2 ROUGE-L: 0.35 → 0.68, cosine: 0.52 → 0.80) [2402.01722].
- **Instruction tuning vs. full model fine-tuning:** Domain-specific, instruction-tuned LLMs (e.g., KodeX-70Bv0.1) now exceed GPT-4 on FinanceBench in LLM-as-a-judge scoring (79.7% vs. 77.8%) [2409.13749].
- **Neurosymbolic/hybrid approaches:** DANA [2410.02823] uses domain-aware neurosymbolic task plans, deterministic symbolic operators, and a Knowledge Store to achieve 94.3% overall accuracy and near-perfect consistency across all FinanceBench question types. DANA constructs explicit hierarchical task plans for extraction, computation, and judgment, eliminating hallucinations and arithmetic slip-ups endemic to pure neural approaches.
- **Iterative reasoning modules (e.g., OODA):** When layered on top of fully fine-tuned RAG, iterative reasoning yields dramatic accuracy gains—85% vs. 37–59% for best non-iterative setups [2404.11792].

## 5. Document Processing Pipelines: Metadata, Multimodal, and Vision-Enhanced Methods

State-of-the-art document QA systems for FinanceBench increasingly exploit rich pre-processing, metadata fusion, and direct vision-based indexing:

- **Metadata-augmented chunking:** Embedding not just text, but also LLM-generated metadata (entities, clusters, nuggets) into chunk representations improves retrieval F1 (e.g., 44.4% vs. 32.9% baseline) [2510.24402].
- **Hybrid and multi-path retrieval:** Combining BM25, dense (BGE-M3), metadata, and hypothetical-answer-based paths achieves 92.51% recall at top-K on comparable expert QA [2504.14493].
- **Vision-enhanced retrieval:** VisionRAG [2511.21121] indexes whole page images using a pyramid of semantic vectors—page-level, section header, atomic fact, and visual hotspot artifacts—constructed via a vision-language model (e.g., GPT-4o). Fusing retrieval signals via Reciprocal Rank Fusion, VisionRAG achieves 80.5% Accuracy@10, surpassing text-based vector store baselines by 30+ percentage points.

| System                       | Accuracy@10 (%) | Comments                                    |
|------------------------------|-----------------|---------------------------------------------|
| GPT-4 Turbo (shared store)   | 19.0            | Text vectorstore                            |
| GPT-4 Turbo (single global)  | 50.0            | Text vectorstore, per-filing index          |
| Claude 2 (long context)      | 76.0            | Full filing in prompt                       |
| GPT-4 Turbo (oracle pages)   | 85.0            | Human-annotated pages                       |
| VisionRAG (pyramid, OCR-less)| 80.5            | Lean index, vision artifacts, 10 pages/K=10 |

## 6. Impact, Limitations, and Future Research Directions

FinanceBench is now established as the definitive QA benchmark for retrieval and question answering on SEC filings. Studies reveal:

- **Domain-adapted retrieval is critical:** Most performance gains in FinanceBench QA are due to specialized retriever/embedding models [2411.07142, 2404.11792], with further boosts from metadata/contextual enrichment [2510.24402].
- **Long-context LLMs vs. vector/RAG hybrids:** Feeding entire filings enables higher accuracy (e.g., GPT-4 79%), but with impractical latency and token costs for enterprise scale; hybrid RAG architectures are preferred for real-world deployment [2311.11944].
- **Hallucinations and error propagation remain central challenges:** Even SOTA LLMs exhibit logical, arithmetic, and retrieval slip-ups, with rare multi-hop or cross-file evidence needs still causing brittleness [2410.02823, 2504.14493].
- **Open, domain-anchored evaluation encourages methodological rigor:** The availability of transparent annotation, precise evaluation, and challenge in evidence-centered retrieval fosters community progress and reproducibility [2311.11944, 2510.24402].

Current limitations include:
- Sensitivity to chunking and retrieval granularity
- Need for continual retriever adaptation as new jargon/reporting standards emerge
- Multi-modal and table/figure evidence support remains incomplete in most text-centric systems

Active research addresses:
- Improved cross-modal and co-reference handling
- Online (incremental) retriever adaptation for continually updated filings
- More efficient cross-encoder and distillation methods for retrieval relevance
- Advanced multi-turn and agentic reasoning scaffolds integrated with retrieval and verification

FinanceBench thus remains central in driving both methodological and practical advances in production-grade financial QA, retrieval, and document intelligence [2311.11944, 2404.11792, 2410.02823, 2411.07142, 2510.24402, 2511.21121, 2504.14493].

Source: https://www.emergentmind.com/topics/financebench