XFinBench: Advanced Financial Benchmark

• XFinBench is a benchmark for complex financial reasoning, assessing LLMs on tasks such as terminology understanding, forecasting, and scenario planning.
• It employs a rigorous methodology that includes OCR extraction, GPT-4o assisted generation, and multi-stage verification to ensure high-quality questions.
• The benchmark integrates multimodal inputs from images, tables, and text, providing a robust evaluation of advanced financial reasoning capabilities.

XFinBench is a benchmark for evaluating LLMs on complex financial problem solving and reasoning rather than on narrow finance QA or simple table arithmetic. It contains 4,235 examples drawn from graduate-level finance source material, includes both text-only and multimodal instances, and is explicitly organized around five core capabilities: terminology understanding, temporal reasoning, future forecasting, scenario planning, and numerical modelling. The benchmark was introduced to address a gap in prior financial LLM evaluation, where many datasets emphasized quantity extraction, basic numerical reasoning from financial reports, or finance knowledge questions without sufficiently testing advanced reasoning, planning, forecasting, and multimodal interpretation (Zhang et al., 20 Aug 2025).

1. Origin, scope, and source material

XFinBench is positioned as a graduate-level, knowledge-intensive benchmark built from finance textbooks and designed to test realistic, technically demanding financial reasoning. Its source material comes from three well-known graduate-level textbooks: Fundamentals of Corporate Finance by Stephen A. Ross, 8th edition, 22 chapters; Options, Futures, and Other Derivatives by John C. Hull, 9th edition, 32 chapters; and The Economics of Money, Banking, and Financial Markets by Frederic S. Mishkin, 9th edition, 25 chapters. This source choice gives the benchmark broad coverage across core finance, derivatives, corporate finance, and monetary economics while making the questions substantially more difficult than basic retail-finance or accounting trivia (Zhang et al., 20 Aug 2025).

The benchmark’s motivation is tied to a specific diagnosis of prior evaluation practice. Existing finance benchmarks such as TAT-QA, FinQA, MultiHiertt, PACIFIC, and ConvFinQA mainly test quantity extraction and relatively basic numerical reasoning over financial reports, while other resources focus on finance knowledge questions without sufficiently testing planning, forecasting, or multimodal interpretation. XFinBench instead targets problems that require integrating domain knowledge with time-dependent reasoning, prediction of future outcomes under theoretical models, analysis of alternative scenarios, and exact or near-exact numerical calculation, sometimes from multimodal evidence such as tables, diagrams, and financial charts.

Its intended use is evaluation of models meant to support financially sophisticated assistant behavior, especially in settings where the system must reason through textbook-style but realistic decision problems. The benchmark spans 28 finance topics, although the exact percentages for each topic are not enumerated in the provided text.

2. Dataset composition and construction pipeline

XFinBench contains 4,235 examples. These are divided into 1,795 statement-judging questions (42.4%), 761 multiple-choice questions (18.0%), and 1,679 financial calculation questions (39.6%). Among the multiple-choice questions, 146 include image-based visual context. Among financial calculation questions, 330 include tabular context. The median question length is 244 tokens, the average is 273.7, and each question is annotated with a median of 1 relevant finance term and an average of 1.3 terms. The benchmark is split into 1,000 validation examples and 3,235 test examples; the split is created by random sampling over the original after-class textbook questions, and test answers are not publicly released to reduce contamination (Zhang et al., 20 Aug 2025).

Task type	Count	Share
Statement judging	1,795	42.4%
Multiple choice	761	18.0%
Financial calculation	1,679	39.6%

The construction pipeline starts from 2,018 after-class questions extracted from the textbooks and solution manuals. Three annotators used OCR with pdfplumber to extract textbook text and also captured screenshots whenever a question had visual or tabular context. Tables were converted into LaTeX format with GPT-4o assistance. Of the 2,018 original source questions, 343 contained visual or tabular context. The initial pool was then classified into the three task types, with 813 source questions assigned to statement judging, 624 to multiple-choice QA, and 858 to financial calculation; some source questions belonged to more than one task.

Because textbook end-of-chapter questions are often open-ended or contain multiple subparts, the authors used GPT-4o in a generate-then-verify framework. In the generation stage, GPT-4o rewrote source questions into examples with clear final answers. For statement judging, it extracted both true and false statements from the original problem/solution pair using separate prompt templates to balance labels. For multiple-choice questions, it generated three answer options under STARC-style constraints: one correct answer and two plausible but misleading distractors with similar wording and length. For financial calculation, it decomposed multi-part textbook questions into independent questions each with a single numerical answer. This stage produced 6,227 candidate questions.

The verification stage filtered these candidates for correctness and completeness, including whether the generated question contained all information necessary to answer it and whether the answer matched the original textbook solution. Additional task-specific checks were applied: independence constraints in statement judging, distractor validity and stylistic similarity in multiple choice, and a purely numerical final answer in financial calculation. Overall, 35.2% of generated questions were discarded during verification.

Quality control involved three evaluators reviewing each example for fluency, completeness, answer correctness, and helpfulness of the annotated finance terms on a 1–5 scale. According to the appendix values cited in the paper, approximately 97.1% of examples had average fluency score at least 4, 97.7% had completeness at least 4, 98.0% had answer correctness at least 4, and 96.8% had knowledge helpfulness at least 4. The annotators were finance-trained, had completed relevant finance courses, and were on track to finish finance master’s degrees.

3. Capability taxonomy, multimodality, and task structure

The conceptual backbone of XFinBench is its five-capability taxonomy. Terminology understanding is defined as the ability to accurately understand finance concepts, including standard financial terms, acronyms, accounting principles, financial instruments, regulatory terminology, and economic indicators. Temporal reasoning is defined as understanding temporal relations in time-based data and making time-sensitive decisions involving cross-period data such as quarterly earnings, historical performance, and future cash flows. Future forecasting is defined as predicting future values or trends of financial indicators such as output, price level, and inflation using economic theory and quantitative methods. Scenario planning is defined as generating and analyzing different possible future scenarios to assess impacts on decisions and strategies. Numerical modelling is defined as creating structured representations of a company’s or product’s financial performance, often using income statements, balance sheets, and cash-flow statements (Zhang et al., 20 Aug 2025).

The overall capability proportions are 56.1% for terminology understanding, 21.7% for temporal reasoning, 5.0% for future forecasting, 7.6% for scenario planning, and 17.2% for numerical modelling. These percentages sum to more than 100 because questions can receive more than one capability label. Capability labels were assigned by three annotators, with a label retained if at least two annotators agreed. On the test/validation split, the appendix reports counts of 1,814/582 for terminology understanding, 703/222 for temporal reasoning, 162/44 for future forecasting, 246/69 for scenario planning, and 557/188 for numerical modelling.

The benchmark also makes explicit how capabilities map to task formats. Terminology understanding is mostly in statement judging (74.7%) and multiple choice (24.3%), with only 1.0% in financial calculation. Temporal reasoning is overwhelmingly in financial calculation (89.5%). Future forecasting appears across all tasks: 22.8% in statement judging, 45.6% in multiple choice, and 31.6% in financial calculation. Scenario planning is concentrated in financial calculation (88.6%), with smaller fractions in statement judging (3.2%) and multiple choice (8.3%). Numerical modelling is almost entirely financial calculation (98.8%).

Multimodality is central to the benchmark’s design. The inputs include screenshots of textbook figures, option payoff diagrams, reserve market supply-demand diagrams, recession-shaded macroeconomic charts, and financial tables from statements. Multiple-choice questions can depend on image interpretation, while financial calculation can depend on tabular content. The paper emphasizes that these questions require more than reading values off a chart: models must connect visual structure to financial reasoning, such as identifying which payoff line corresponds to an option strategy or inferring how a reserve-market equilibrium shifts after a macroeconomic change. For evaluation, multimodal inputs are denoted as $Q,I$ for question plus image and $Q,T$ for question plus table.

4. Evaluation protocol and scoring methodology

As a benchmark, XFinBench uses three close-ended formats suitable for automatic scoring. Statement judging asks whether a statement is true or false. Multiple-choice QA presents exactly three candidate answers, one correct and two distractors. Financial calculation requires a numerical answer. The main metrics are Accuracy for statement judging and multiple choice, and two metrics for financial calculation: exact-match accuracy and a tolerance-based metric denoted $Acc_{ERR@5}$, defined as accuracy within a 0.5% error margin of the correct answer. Unless otherwise specified, financial-calculation results are reported using $Acc_{ERR@5}$ (Zhang et al., 20 Aug 2025).

The prompting protocol was designed for fairness and informed by a prompt sensitivity study. The authors compared direct answering and chain-of-thought, with and without role-play and explicit output requirements, and found that the template combining chain-of-thought, role-play (“You are a financial expert”), and explicit output formatting gave the best or near-best results consistently while preserving ranking stability. In text tasks, the template asks the model to think step by step and conclude with a standardized phrase such as “Therefore, my answer is …”.

For financial calculation, the paper also evaluates Program-of-Thought (PoT), prompting models to write a Python function in the form:

$Q,T$0

The maximum token length is set to 1024, and default API hyperparameters are otherwise used. For Llama-3.2 Vision models in text-only tasks, the evaluation feeds a blank image because those models require visual input.

The evaluated systems are grouped into multimodal models and text-only models. The multimodal group includes GPT-4o, GPT-4o-mini, Claude-3.5-Sonnet, Claude-3-Opus, Claude-3-Haiku, Gemini-1.5-Flash, Gemini-1.5-Pro, Llama-3.2-90B-Vision, and Llama-3.2-11B-Vision. The text-only group includes o1, o1-mini, Llama-3.1-405B, DeepSeek-Chat, Llama-3.1-70B, Llama-3-70B, Llama-3.1-8B, Llama-3-8B, and Mixtral-$8\times7$B. The paper repeatedly states that it evaluates 18 leading models, while the appendix hyperparameter table shows 17 automated models and the main table includes a human row separately. The human baseline was measured on a random 1,000-example subset of the test set in a close-book setting with three finance master’s degree holders who were not involved in dataset construction.

5. Empirical performance and benchmark difficulty

The principal leaderboard results show that, on the full benchmark including visual-context questions, the best multimodal model is Claude-3.5-Sonnet with 64.1% overall accuracy, narrowly ahead of GPT-4o at 63.6%. On visual-context multiple-choice questions specifically, GPT-4o is best at 65.3%, slightly above Claude-3.5-Sonnet’s 63.7%. In the text-only setting, o1 is the top model with 67.3% overall accuracy. The human baseline is 79.8% overall, yielding a 12.5 percentage-point gap between humans and o1 (Zhang et al., 20 Aug 2025).

Task-level results clarify where the benchmark is hardest. In statement judging, o1 scores 87.6%, Claude-3.5-Sonnet 84.3%, GPT-4o 84.0%, and humans 90.9%. In text-only multiple choice, o1 reaches 94.0%, while humans score 92.1%. The harder separation appears in multimodal multiple choice and calculation. On visual-context multiple choice, GPT-4o gets 65.3%, Claude-3.5-Sonnet 63.7%, and humans 81.1%. On financial calculation, o1 reaches 34.2 / 62.0 for exact match / $Acc_{ERR@5}$ on plain question input and 42.2 / 66.4 when tabular input is involved, whereas humans reach 63.8 / 77.6 for plain calculation and 74.6 / 83.6 for calculation with tables.

Capability-level results reinforce the same pattern. Among reported models, o1 is best across the five capabilities with 88.9 in terminology understanding, 59.1 in temporal reasoning, 74.7 in future forecasting, 60.1 in scenario planning, and 66.5 in numerical modelling. Human performance on the same capabilities is 91.0, 79.5, 86.2, 75.8, and 78.0, respectively. The central claim supported by these numbers is that LLMs are approaching humans in terminology understanding, but still lag substantially in temporal reasoning and scenario planning.

The benchmark also exposes heterogeneity within open-source and smaller models. Llama-3.1-405B reaches 61.9 overall, close to o1-mini’s 62.0 and above GPT-4o-mini’s 57.4 in text-only tasks. DeepSeek-Chat reaches 59.6. Smaller systems degrade sharply: Llama-3.1-8B scores 45.5, Llama-3-8B scores 42.9, and Mixtral-$8\times 7$B scores 16.6.

A separate result concerns Program-of-Thought. It mostly does not help: most models perform worse under PoT than under CoT. The paper attributes this to low execution rates of generated Python programs, with many models producing non-executable code. The reported scatterplot shows a relationship between execution rate and $Acc_{ERR@5}$. o1 is an exception: despite lower execution rate than some closed models, it still achieves the highest $Acc_{ERR@5}$, which the authors interpret as better reasoning efficiency when its generated programs do execute.

6. Knowledge bank, retrieval augmentation, and error analysis

A notable component of XFinBench is its finance knowledge bank. The authors built a bank of 3,032 finance terms with 1,766 unique definitions, collected from the subject indexes of the three textbooks and manually defined from the corresponding pages. The definitions are long, with median length 830 and average length 1,249; about 34.3% include mathematical formulas. Annotators linked each question to 1 to 3 highly relevant terms from the same chapter, attaching a term only when at least two annotators agreed. This produced 1.3 annotated terms per question on average (Zhang et al., 20 Aug 2025).

The knowledge bank supports retrieval augmentation experiments. At inference time, retrieved knowledge is prepended to the prompt. The paper evaluates three retrieval settings: BM25, Ada-Embed-based dense retrieval, and Oracle retrieval using the human-annotated ground-truth relevant terms. The setup uses $n=3$, retrieving the top 3 question-relevant terms, where each term is represented by the concatenation of its name and definition. Retriever accuracy is defined as whether the retriever locates at least one gold annotated term for the question. Ada Embed consistently outperforms BM25: for example, in the All setting, BM25 achieves 30.6 retriever accuracy, Ada Embed 39.8, and Oracle 100.0.

Model-level augmentation results are selective rather than uniform. Overall accuracy changes from no knowledge to Oracle as follows: GPT-4o: 63.6 → 65.2, GPT-4o-mini: 57.4 → 60.2, Llama-3.1-405B: 61.9 → 64.0, and Llama-3.1-8B: 45.5 → 48.5. The paper’s qualitative conclusion is more specific than a general endorsement of retrieval-augmented generation: relevant knowledge consistently improves only the smaller open-source models. The largest improvements appear in terminology understanding, while future forecasting sees limited gains and can even worsen, as in GPT-4o dropping from 63.6 to 61.1 under Oracle on future forecasting. The authors interpret this as evidence that external knowledge helps when the main limitation is missing domain vocabulary or definitions, but does less when the bottleneck is deeper reasoning or computation.

The error analysis is unusually concrete. For financial calculation, annotators examined 400 o1 responses and identified two dominant error types: rounding error in intermediate steps and knowledge misuse, meaning application of the wrong or incomplete finance formula. The paper reports that 55.2% of analyzed o1 responses had a correct reasoning path without intermediate rounding error or knowledge misuse. For visual-context questions, the authors analyzed 100 GPT-4o responses and found two main failure types: blindness, meaning difficulty identifying the position and/or intersection of curves, and knowledge misuse, meaning introduction of irrelevant financial knowledge that derails reasoning. 35% of sampled responses contain blindness, and 24% had otherwise correct reasoning except for blindness.

The paper also analyzes augmentation failures under the Oracle setting using 100 GPT-4o wrong answers. It identifies three error types: reasoning error, where failure is unrelated to the added knowledge; over thinking, where the augmented knowledge already contains direct evidence but the model reasons further than necessary and goes wrong; and over reliance, where the model follows the injected knowledge too literally and ignores a simpler or more appropriate path. The reported pattern is that wrong answers on calculation questions are mostly reasoning errors unrelated to augmentation, over thinking is common in multiple-choice future-forecasting questions, and over reliance often appears in scenario planning.

7. Position within the finance-benchmark landscape

Within the finance benchmark ecosystem, XFinBench occupies the niche of graduate-level, multimodal, capability-annotated financial problem solving. It differs from earlier benchmarks such as TAT-QA, FinQA, PACIFIC, ConvFinQA, and MultiHiertt, which primarily test extraction and basic numerical reasoning over financial tables and reports. It also differs from FinEval, described in the paper as text-only and concept-oriented; from BizBench, which covers business and finance reasoning and quantitative tasks but not the same level of complex multimodal financial problem solving; and from FinanceMATH, which emphasizes financial mathematics and coding but lacks the broader planning, forecasting, and multimodal dimension (Zhang et al., 20 Aug 2025).

Later and related benchmarks emphasize different evaluation objects. FinBen broadens coverage across 35 datasets spanning 23 tasks and seven aspects including information extraction, textual analysis, question answering, text generation, risk management, forecasting, and decision-making, with a distinctive stock-trading agent component (Xie et al., 2024). BigFinanceBench shifts the target from final-answer correctness to workflow-grounded evaluation of auditable derivation in open-ended financial-research tasks, using 928 items with point-weighted rubrics (Wang et al., 2 Jun 2026). FINESSE-Bench introduces an explicit hierarchy of professional difficulty across certification-style and trading-oriented finance tasks, including technical analysis and Russian-language olympiad problems (Stanishevskii et al., 14 May 2026). FrontierFinance and Finch move further toward long-horizon artifact construction and spreadsheet-centric enterprise workflows rather than close-ended benchmark items (Krumdick et al., 7 Apr 2026, Dong et al., 15 Dec 2025). FinChart-Bench isolates the problem of financial chart comprehension in vision-LLMs (Shu et al., 20 Jul 2025). BizFinBench focuses on real-world financial applications in Chinese, including reasoning, numerical calculation, information extraction, prediction recognition, and knowledge-based QA (Lu et al., 26 May 2025). FinMaster emphasizes simulation-driven accounting, auditing, and consulting workflows (Jiang et al., 18 May 2025), while FinForge presents a semi-synthetic pipeline for generating finance-specific multiple-choice benchmarks from authoritative financial corpora (Matlin et al., 11 Jan 2026).

XFinBench’s limitations are stated explicitly. Its evaluation relies on a specific prompt design that includes role-play, CoT or PoT, and strict output formatting, which can affect model performance. Although the benchmark is high-quality and difficult, it is smaller than simpler finance datasets focused on extraction-style tasks. Its source material is limited to three textbooks, which may induce textbook-style bias in topic framing and solution style. There is also an implicit multimodal comparability limitation: text-only models cannot be directly compared on image questions, which is why the paper distinguishes best text-only performance from best multimodal performance on the full benchmark. The framing of the work suggests further directions in more complex multimodal finance settings, stronger evaluation of forecasting and planning, and systems that better combine domain knowledge, computation, and visual reasoning.