---
title: 'XFinBench: Advanced Financial Benchmark'
url: https://www.emergentmind.com/topics/xfinbench
type: topic
---

# XFinBench: Advanced Financial Benchmark

XFinBench is a benchmark for evaluating large language models on **complex financial problem solving and reasoning** rather than on narrow finance QA or simple table arithmetic. It contains **4,235 examples** drawn from **graduate-level finance** source material, includes both text-only and multimodal instances, and is explicitly organized around five core capabilities: **terminology understanding, temporal reasoning, future forecasting, scenario planning, and numerical modelling**. The benchmark was introduced to address a gap in prior financial LLM evaluation, where many datasets emphasized quantity extraction, basic numerical reasoning from financial reports, or finance knowledge questions without sufficiently testing advanced reasoning, planning, forecasting, and multimodal interpretation [2508.15861].

## 1. Origin, scope, and source material

XFinBench is positioned as a **graduate-level, knowledge-intensive benchmark built from finance textbooks** and designed to test realistic, technically demanding financial reasoning. Its source material comes from three well-known graduate-level textbooks: *Fundamentals of Corporate Finance* by Stephen A. Ross, **8th edition, 22 chapters**; *Options, Futures, and Other Derivatives* by John C. Hull, **9th edition, 32 chapters**; and *The Economics of Money, Banking, and Financial Markets* by Frederic S. Mishkin, **9th edition, 25 chapters**. This source choice gives the benchmark broad coverage across core finance, derivatives, corporate finance, and monetary economics while making the questions substantially more difficult than basic retail-finance or accounting trivia [2508.15861].

The benchmark’s motivation is tied to a specific diagnosis of prior evaluation practice. Existing finance benchmarks such as **TAT-QA, FinQA, MultiHiertt, PACIFIC, and ConvFinQA** mainly test quantity extraction and relatively basic numerical reasoning over financial reports, while other resources focus on finance knowledge questions without sufficiently testing planning, forecasting, or multimodal interpretation. XFinBench instead targets problems that require integrating domain knowledge with time-dependent reasoning, prediction of future outcomes under theoretical models, analysis of alternative scenarios, and exact or near-exact numerical calculation, sometimes from multimodal evidence such as tables, diagrams, and financial charts.

Its intended use is evaluation of models meant to support **financially sophisticated assistant behavior**, especially in settings where the system must reason through textbook-style but realistic decision problems. The benchmark spans **28 finance topics**, although the exact percentages for each topic are not enumerated in the provided text.

## 2. Dataset composition and construction pipeline

XFinBench contains **4,235 examples**. These are divided into **1,795 statement-judging questions (42.4%)**, **761 multiple-choice questions (18.0%)**, and **1,679 financial calculation questions (39.6%)**. Among the multiple-choice questions, **146 include image-based visual context**. Among financial calculation questions, **330 include tabular context**. The **median question length is 244 tokens**, the **average is 273.7**, and each question is annotated with a **median of 1 relevant finance term** and an **average of 1.3 terms**. The benchmark is split into **1,000 validation examples** and **3,235 test examples**; the split is created by random sampling over the original after-class textbook questions, and test answers are not publicly released to reduce contamination [2508.15861].

| Task type | Count | Share |
|---|---:|---:|
| Statement judging | 1,795 | 42.4% |
| Multiple choice | 761 | 18.0% |
| Financial calculation | 1,679 | 39.6% |

The construction pipeline starts from **2,018 after-class questions** extracted from the textbooks and solution manuals. **Three annotators** used OCR with `pdfplumber` to extract textbook text and also captured screenshots whenever a question had visual or tabular context. Tables were converted into LaTeX format with **GPT-4o assistance**. Of the **2,018** original source questions, **343** contained visual or tabular context. The initial pool was then classified into the three task types, with **813** source questions assigned to statement judging, **624** to multiple-choice QA, and **858** to financial calculation; some source questions belonged to more than one task.

Because textbook end-of-chapter questions are often open-ended or contain multiple subparts, the authors used **GPT-4o in a generate-then-verify framework**. In the generation stage, GPT-4o rewrote source questions into examples with clear final answers. For statement judging, it extracted both true and false statements from the original problem/solution pair using separate prompt templates to balance labels. For multiple-choice questions, it generated three answer options under **STARC-style constraints**: one correct answer and two plausible but misleading distractors with similar wording and length. For financial calculation, it decomposed multi-part textbook questions into independent questions each with a single numerical answer. This stage produced **6,227 candidate questions**.

The verification stage filtered these candidates for correctness and completeness, including whether the generated question contained all information necessary to answer it and whether the answer matched the original textbook solution. Additional task-specific checks were applied: independence constraints in statement judging, distractor validity and stylistic similarity in multiple choice, and a purely numerical final answer in financial calculation. Overall, **35.2%** of generated questions were discarded during verification.

Quality control involved **three evaluators** reviewing each example for **fluency, completeness, answer correctness, and helpfulness of the annotated finance terms** on a **1–5 scale**. According to the appendix values cited in the paper, approximately **97.1%** of examples had average fluency score at least 4, **97.7%** had completeness at least 4, **98.0%** had answer correctness at least 4, and **96.8%** had knowledge helpfulness at least 4. The annotators were finance-trained, had completed relevant finance courses, and were on track to finish finance master’s degrees.

## 3. Capability taxonomy, multimodality, and task structure

The conceptual backbone of XFinBench is its five-capability taxonomy. **Terminology understanding** is defined as the ability to accurately understand finance concepts, including standard financial terms, acronyms, accounting principles, financial instruments, regulatory terminology, and economic indicators. **Temporal reasoning** is defined as understanding temporal relations in time-based data and making time-sensitive decisions involving cross-period data such as quarterly earnings, historical performance, and future cash flows. **Future forecasting** is defined as predicting future values or trends of financial indicators such as output, price level, and inflation using economic theory and quantitative methods. **Scenario planning** is defined as generating and analyzing different possible future scenarios to assess impacts on decisions and strategies. **Numerical modelling** is defined as creating structured representations of a company’s or product’s financial performance, often using income statements, balance sheets, and cash-flow statements [2508.15861].

The overall capability proportions are **56.1%** for terminology understanding, **21.7%** for temporal reasoning, **5.0%** for future forecasting, **7.6%** for scenario planning, and **17.2%** for numerical modelling. These percentages sum to more than 100 because questions can receive more than one capability label. Capability labels were assigned by **three annotators**, with a label retained if at least **two annotators** agreed. On the test/validation split, the appendix reports counts of **1,814/582** for terminology understanding, **703/222** for temporal reasoning, **162/44** for future forecasting, **246/69** for scenario planning, and **557/188** for numerical modelling.

The benchmark also makes explicit how capabilities map to task formats. Terminology understanding is mostly in **statement judging (74.7%)** and **multiple choice (24.3%)**, with only **1.0%** in financial calculation. Temporal reasoning is overwhelmingly in **financial calculation (89.5%)**. Future forecasting appears across all tasks: **22.8%** in statement judging, **45.6%** in multiple choice, and **31.6%** in financial calculation. Scenario planning is concentrated in **financial calculation (88.6%)**, with smaller fractions in statement judging (**3.2%**) and multiple choice (**8.3%**). Numerical modelling is almost entirely financial calculation (**98.8%**).

Multimodality is central to the benchmark’s design. The inputs include screenshots of textbook figures, option payoff diagrams, reserve market supply-demand diagrams, recession-shaded macroeconomic charts, and financial tables from statements. Multiple-choice questions can depend on image interpretation, while financial calculation can depend on tabular content. The paper emphasizes that these questions require more than reading values off a chart: models must connect visual structure to financial reasoning, such as identifying which payoff line corresponds to an option strategy or inferring how a reserve-market equilibrium shifts after a macroeconomic change. For evaluation, multimodal inputs are denoted as **\(Q,I\)** for question plus image and **\(Q,T\)** for question plus table.

## 4. Evaluation protocol and scoring methodology

As a benchmark, XFinBench uses three close-ended formats suitable for automatic scoring. **Statement judging** asks whether a statement is true or false. **Multiple-choice QA** presents exactly three candidate answers, one correct and two distractors. **Financial calculation** requires a numerical answer. The main metrics are **Accuracy** for statement judging and multiple choice, and two metrics for financial calculation: **exact-match accuracy** and a tolerance-based metric denoted **\(Acc_{ERR@5}\)**, defined as accuracy within a **0.5% error margin** of the correct answer. Unless otherwise specified, financial-calculation results are reported using **\(Acc_{ERR@5}\)** [2508.15861].

The prompting protocol was designed for fairness and informed by a prompt sensitivity study. The authors compared direct answering and chain-of-thought, with and without role-play and explicit output requirements, and found that the template combining **chain-of-thought**, **role-play** (“You are a financial expert”), and **explicit output formatting** gave the best or near-best results consistently while preserving ranking stability. In text tasks, the template asks the model to think step by step and conclude with a standardized phrase such as **“Therefore, my answer is …”**.

For financial calculation, the paper also evaluates **Program-of-Thought (PoT)**, prompting models to write a Python function in the form:

```python
def solution():
```

The maximum token length is set to **1024**, and default API hyperparameters are otherwise used. For **Llama-3.2 Vision** models in text-only tasks, the evaluation feeds a **blank image** because those models require visual input.

The evaluated systems are grouped into **multimodal models** and **text-only models**. The multimodal group includes **GPT-4o, GPT-4o-mini, Claude-3.5-Sonnet, Claude-3-Opus, Claude-3-Haiku, Gemini-1.5-Flash, Gemini-1.5-Pro, Llama-3.2-90B-Vision, and Llama-3.2-11B-Vision**. The text-only group includes **o1, o1-mini, Llama-3.1-405B, DeepSeek-Chat, Llama-3.1-70B, Llama-3-70B, Llama-3.1-8B, Llama-3-8B, and Mixtral-\(8\times7\)B**. The paper repeatedly states that it evaluates **18 leading models**, while the appendix hyperparameter table shows **17 automated models** and the main table includes a **human row** separately. The human baseline was measured on a **random 1,000-example subset** of the test set in a **close-book setting** with **three finance master’s degree holders** who were not involved in dataset construction.

## 5. Empirical performance and benchmark difficulty

The principal leaderboard results show that, on the full benchmark including visual-context questions, the best multimodal model is **Claude-3.5-Sonnet** with **64.1% overall accuracy**, narrowly ahead of **GPT-4o** at **63.6%**. On visual-context multiple-choice questions specifically, **GPT-4o** is best at **65.3%**, slightly above **Claude-3.5-Sonnet’s 63.7%**. In the text-only setting, **o1** is the top model with **67.3% overall accuracy**. The human baseline is **79.8% overall**, yielding a **12.5 percentage-point gap** between humans and o1 [2508.15861].

Task-level results clarify where the benchmark is hardest. In **statement judging**, o1 scores **87.6%**, Claude-3.5-Sonnet **84.3%**, GPT-4o **84.0%**, and humans **90.9%**. In **text-only multiple choice**, o1 reaches **94.0%**, while humans score **92.1%**. The harder separation appears in multimodal multiple choice and calculation. On **visual-context multiple choice**, GPT-4o gets **65.3%**, Claude-3.5-Sonnet **63.7%**, and humans **81.1%**. On **financial calculation**, o1 reaches **34.2 / 62.0** for exact match / \(Acc_{ERR@5}\) on plain question input and **42.2 / 66.4** when tabular input is involved, whereas humans reach **63.8 / 77.6** for plain calculation and **74.6 / 83.6** for calculation with tables.

Capability-level results reinforce the same pattern. Among reported models, o1 is best across the five capabilities with **88.9** in terminology understanding, **59.1** in temporal reasoning, **74.7** in future forecasting, **60.1** in scenario planning, and **66.5** in numerical modelling. Human performance on the same capabilities is **91.0**, **79.5**, **86.2**, **75.8**, and **78.0**, respectively. The central claim supported by these numbers is that LLMs are approaching humans in **terminology understanding**, but still lag substantially in **temporal reasoning** and **scenario planning**.

The benchmark also exposes heterogeneity within open-source and smaller models. **Llama-3.1-405B** reaches **61.9** overall, close to **o1-mini’s 62.0** and above **GPT-4o-mini’s 57.4** in text-only tasks. **DeepSeek-Chat** reaches **59.6**. Smaller systems degrade sharply: **Llama-3.1-8B** scores **45.5**, **Llama-3-8B** scores **42.9**, and **Mixtral-\(8\times 7\)B** scores **16.6**.

A separate result concerns **Program-of-Thought**. It mostly does not help: most models perform worse under PoT than under CoT. The paper attributes this to low execution rates of generated Python programs, with many models producing non-executable code. The reported scatterplot shows a relationship between execution rate and **\(Acc_{ERR@5}\)**. **o1** is an exception: despite lower execution rate than some closed models, it still achieves the highest **\(Acc_{ERR@5}\)**, which the authors interpret as better reasoning efficiency when its generated programs do execute.

## 6. Knowledge bank, retrieval augmentation, and error analysis

A notable component of XFinBench is its **finance knowledge bank**. The authors built a bank of **3,032 finance terms** with **1,766 unique definitions**, collected from the subject indexes of the three textbooks and manually defined from the corresponding pages. The definitions are long, with **median length 830** and **average length 1,249**; about **34.3%** include mathematical formulas. Annotators linked each question to **1 to 3 highly relevant terms from the same chapter**, attaching a term only when at least **two annotators** agreed. This produced **1.3 annotated terms per question on average** [2508.15861].

The knowledge bank supports retrieval augmentation experiments. At inference time, retrieved knowledge is prepended to the prompt. The paper evaluates three retrieval settings: **BM25**, **Ada-Embed-based dense retrieval**, and **Oracle retrieval** using the human-annotated ground-truth relevant terms. The setup uses **\(n=3\)**, retrieving the **top 3 question-relevant terms**, where each term is represented by the concatenation of its name and definition. Retriever accuracy is defined as whether the retriever locates at least one gold annotated term for the question. **Ada Embed** consistently outperforms **BM25**: for example, in the **All** setting, BM25 achieves **30.6** retriever accuracy, Ada Embed **39.8**, and Oracle **100.0**.

Model-level augmentation results are selective rather than uniform. Overall accuracy changes from no knowledge to Oracle as follows: **GPT-4o: 63.6 → 65.2**, **GPT-4o-mini: 57.4 → 60.2**, **Llama-3.1-405B: 61.9 → 64.0**, and **Llama-3.1-8B: 45.5 → 48.5**. The paper’s qualitative conclusion is more specific than a general endorsement of retrieval-augmented generation: **relevant knowledge consistently improves only the smaller open-source models**. The largest improvements appear in **terminology understanding**, while **future forecasting** sees limited gains and can even worsen, as in **GPT-4o dropping from 63.6 to 61.1 under Oracle on future forecasting**. The authors interpret this as evidence that external knowledge helps when the main limitation is missing domain vocabulary or definitions, but does less when the bottleneck is deeper reasoning or computation.

The error analysis is unusually concrete. For **financial calculation**, annotators examined **400 o1 responses** and identified two dominant error types: **rounding error** in intermediate steps and **knowledge misuse**, meaning application of the wrong or incomplete finance formula. The paper reports that **55.2%** of analyzed o1 responses had a correct reasoning path without intermediate rounding error or knowledge misuse. For **visual-context questions**, the authors analyzed **100 GPT-4o responses** and found two main failure types: **blindness**, meaning difficulty identifying the position and/or intersection of curves, and **knowledge misuse**, meaning introduction of irrelevant financial knowledge that derails reasoning. **35%** of sampled responses contain blindness, and **24%** had otherwise correct reasoning except for blindness.

The paper also analyzes augmentation failures under the Oracle setting using **100 GPT-4o wrong answers**. It identifies three error types: **reasoning error**, where failure is unrelated to the added knowledge; **over thinking**, where the augmented knowledge already contains direct evidence but the model reasons further than necessary and goes wrong; and **over reliance**, where the model follows the injected knowledge too literally and ignores a simpler or more appropriate path. The reported pattern is that wrong answers on calculation questions are mostly reasoning errors unrelated to augmentation, over thinking is common in multiple-choice future-forecasting questions, and over reliance often appears in scenario planning.

## 7. Position within the finance-benchmark landscape

Within the finance benchmark ecosystem, XFinBench occupies the niche of **graduate-level, multimodal, capability-annotated financial problem solving**. It differs from earlier benchmarks such as **TAT-QA, FinQA, PACIFIC, ConvFinQA, and MultiHiertt**, which primarily test extraction and basic numerical reasoning over financial tables and reports. It also differs from **FinEval**, described in the paper as text-only and concept-oriented; from **BizBench**, which covers business and finance reasoning and quantitative tasks but not the same level of complex multimodal financial problem solving; and from **FinanceMATH**, which emphasizes financial mathematics and coding but lacks the broader planning, forecasting, and multimodal dimension [2508.15861].

Later and related benchmarks emphasize different evaluation objects. **FinBen** broadens coverage across **35 datasets spanning 23 tasks** and seven aspects including information extraction, textual analysis, question answering, text generation, risk management, forecasting, and decision-making, with a distinctive stock-trading agent component [2402.12659]. **BigFinanceBench** shifts the target from final-answer correctness to **workflow-grounded** evaluation of **auditable derivation** in open-ended financial-research tasks, using **928 items** with point-weighted rubrics [2606.03829]. **FINESSE-Bench** introduces an explicit **hierarchy of professional difficulty** across certification-style and trading-oriented finance tasks, including technical analysis and Russian-language olympiad problems [2605.15482]. **FrontierFinance** and **Finch** move further toward long-horizon artifact construction and spreadsheet-centric enterprise workflows rather than close-ended benchmark items [2604.05912] [2512.13168]. **FinChart-Bench** isolates the problem of **financial chart comprehension** in vision-language models [2507.14823]. **BizFinBench** focuses on **real-world financial applications in Chinese**, including reasoning, numerical calculation, information extraction, prediction recognition, and knowledge-based QA [2505.19457]. **FinMaster** emphasizes simulation-driven accounting, auditing, and consulting workflows [2505.13533], while **FinForge** presents a semi-synthetic pipeline for generating finance-specific multiple-choice benchmarks from authoritative financial corpora [2601.06747].

XFinBench’s limitations are stated explicitly. Its evaluation relies on a **specific prompt design** that includes role-play, CoT or PoT, and strict output formatting, which can affect model performance. Although the benchmark is high-quality and difficult, it is **smaller than simpler finance datasets** focused on extraction-style tasks. Its source material is limited to **three textbooks**, which may induce **textbook-style bias** in topic framing and solution style. There is also an implicit multimodal comparability limitation: **text-only models cannot be directly compared on image questions**, which is why the paper distinguishes best text-only performance from best multimodal performance on the full benchmark. The framing of the work suggests further directions in more complex multimodal finance settings, stronger evaluation of forecasting and planning, and systems that better combine domain knowledge, computation, and visual reasoning.

Source: https://www.emergentmind.com/topics/xfinbench