FinMMR: Multimodal Financial Reasoning

• FinMMR is a bilingual benchmark that evaluates multimodal LLMs on complex financial documents using integrated visual and textual analysis.
• It comprises 4,300 questions and 8,700 images across 14 financial subdomains, requiring multi-step reasoning and OCR capabilities.
• The benchmark emphasizes precision with a strict 0.2% error margin, categorizing tasks into easy, medium, and hard based on difficulty scores.

FinMMR (“Financial Multimodal Numerical Reasoning”) is a bilingual benchmark specifically developed to evaluate and advance Multimodal LLMs (MLLMs) for expert-level numerical reasoning tasks based on real-world financial documents. It is distinguished by its multimodality, comprehensive domain coverage, and rigorous demands on multi-step numerical reasoning that integrates financial knowledge with deep visual-textual understanding. FinMMR sets a new standard for realism and challenge in benchmarking the financial reasoning capabilities of AI models (Tang et al., 6 Aug 2025).

1. Motivation and Scope

FinMMR addresses key limitations of prior financial question answering (QA) benchmarks, which were primarily text-based or multiple choice, and therefore insufficient in both domain breadth and the depth of multimodal reasoning required by professional financial analysis. The benchmark aims to replicate the practical conditions faced by financial analysts, who must interpret complex layouts of tables, charts, and ownership structures within financial reports and perform precise, multi-step calculations with specialized domain knowledge (e.g., corporate finance, derivatives pricing). The core objectives are: (a) realistic multimodal integration by visually parsing complex financial images; (b) application of advanced financial expertise; and (c) execution of exact computations under strict tolerance constraints.

2. Dataset Composition and Domain Coverage

The FinMMR dataset contains 4,300 questions paired with 8,700 images and is distributed evenly between Chinese and English (2,150 each). The questions span 14 major financial subdomains:

• Corporate Finance
• Banking
• Asset Management
• Financial Markets
• Industry Analysis
• Macroeconomic Analysis
• Mergers & Acquisitions
• Equity Valuation
• Fixed Income
• Derivatives & Option Pricing
• Risk Management
• Alternative Investments
• FinTech & Digital Finance
• Strategy Research

Each question package includes one or more ground-truth images (plus two distractor images for the Chinese set), a numerical query, an executable Python solution script, and an exact numeric answer. This bilingual approach and breadth of domain coverage distinguish FinMMR from previous benchmarks by enabling evaluation across both languages and a wide variety of financial analyses.

3. Image Typology and Data Structure

FinMMR’s 8,700 images are systematically categorized across 14 types, emphasizing the variety of financial visual formats encountered in real reports. The distribution is shown below:

Image Category	Number of Images	Representative Context
Tables	2,100	Financial statements, key figures
Bar charts	900	Comparative analytics, growth
Line charts	800	Trends, time-series
Pie charts	700	Proportional breakdowns
Candlestick charts	650	Price movement analysis
Ownership-structure charts	700	Shareholding analysis
Flow diagrams	600	Cash flow or process visualizations
Cash-flow diagrams	550	Capital movements
Organizational charts	450	Hierarchy structures
Stacked-bar charts	450	Category aggregations
Area charts	350	Cumulative trends
Scatter plots	250	Correlation exploration
Histograms	150	Distributions
Network diagrams	100	Relationship mapping

A salient feature is that all tabular data are rendered as images, and original text tables are removed, requiring models to perform OCR and visual parsing as a prerequisite to reasoning.

4. Data Curation and Complexity Stratification

FinMMR construction involved two primary data curation strategies. First, 1,419 questions were adapted from extant text-only financial QA datasets (MMMU, MMMU-Pro, FinanceMath, CodeTAT-QA, CodeFinQA, DocMath-Eval); all tables were visually rendered to images. Second, 90 licensed Chinese financial research reports were processed using 360LayoutAnalysis to extract relevant images. Qwen-VL-Max was employed to generate multi-step reasoning questions, which were refined by a team of 16 finance graduates and 2 CFAs, who also provided Python solution code and final numerical answers. For greater realism, Chinese questions include two distractor images from adjacent pages.

Problem difficulty is stratified using a “reasoning complexity score”:

$$rc = \ln\bigl(\max(o,1)\bigr) + \ln\bigl(\max(l + p,1)\bigr)$$

where $o$ is the number of operators, $l$ is the solution length (lines of code), and $p$ is the number of parentheses pairs. Based on this score, questions are partitioned into Easy (1,300), Medium (1,500), and Hard (1,500) categories.

5. Problem Types and Reasoning Requirements

FinMMR’s reasoning scenarios are characterized as follows:

• Easy (rc ≲ 2.0): Single-step lookup and arithmetic from images (e.g., extracting revenue and cost, simple subtraction).
• Medium (2.0 ≲ rc ≲ 3.5): Two-step chaining, typically involving extraction from charts and formula application (e.g., computing year-on-year growth rates).
• Hard (rc ≳ 3.5): Multi-step calculations requiring advanced financial formulas (e.g., implementing the Black–Scholes inverse using Newton–Raphson for implied volatility). This level often mandates both precise extraction of multiple parameters and correct multi-line programmatic computation.

Each problem type explicitly blends image understanding, domain expertise, and strict numeric answer conformance.

6. Evaluation Metrics and Baseline Performance

Evaluation in FinMMR employs rigorous numerical accuracy: a model’s answer $\hat{y}$ is correct if

$|\hat y - y^*| / y^* \leq 0.2\%$

where $y^*$ is the ground-truth answer. Overall accuracy per subset follows

$$\text{Accuracy} = \frac{\text{Number of Correct Answers}}{\text{Total Questions}} \times 100\%$$

Fifteen leading proprietary and open-source MLLMs were evaluated in zero-shot settings using: IO (direct prompting), Chain-of-Thought (CoT), and Program-of-Thought (PoT) strategies. On the Hard test subset, the top performing MLLM (Claude 3.7 Sonnet, 64K extended thinking, PoT) achieved 53.0% accuracy. Performance across splits for this model is:

Difficulty	Accuracy (%)
Easy	78.5
Medium	62.5
Hard	53.0
Overall	64.0

Error analysis on 100 sampled Hard failures indicated breakdowns as follows: 30% from visual perception (misreading images), 38% from knowledge reasoning (incorrect formula/concept), and 32% from numerical computation (precision errors, calculation mistakes).

7. Contributions and Research Significance

FinMMR introduces three core advancements:

• Multimodality: All tabular/text data are rasterized as images; distractor images contribute to real-world document complexity and “noise.”
• Comprehensiveness: Coverage of 14 financial subdomains, bilingual design, and 14 distinct image types collectively deliver an unprecedented breadth and realism for benchmarking.
• Challenge: Multi-step expert reasoning, strict domain-specific numerical computations, and a precision threshold at the 0.2% error margin underscore the benchmark’s rigor.

Findings underscore the current gap between MLLM performance and expert human-level reasoning in integrated visual-financial tasks, even under advanced prompting and extended context regimes. FinMMR is openly available with an online evaluation platform and is expected to facilitate further advances in multimodal reasoning, visual-textual integration, and financial AI method development (Tang et al., 6 Aug 2025).