--- title: 'FinMMR: Multimodal Financial Reasoning' url: https://www.emergentmind.com/topics/finmmr type: topic --- # FinMMR: Multimodal Financial Reasoning FinMMR (“Financial Multimodal Numerical Reasoning”) is a bilingual benchmark specifically developed to evaluate and advance Multimodal Large Language Models (MLLMs) for expert-level numerical reasoning tasks based on real-world financial documents. It is distinguished by its multimodality, comprehensive domain coverage, and rigorous demands on multi-step numerical reasoning that integrates financial knowledge with deep visual-textual understanding. FinMMR sets a new standard for realism and challenge in benchmarking the financial reasoning capabilities of AI models [2508.04625]. ## 1. Motivation and Scope FinMMR addresses key limitations of prior financial question answering (QA) benchmarks, which were primarily text-based or multiple choice, and therefore insufficient in both domain breadth and the depth of multimodal reasoning required by professional financial analysis. The benchmark aims to replicate the practical conditions faced by financial analysts, who must interpret complex layouts of tables, charts, and ownership structures within financial reports and perform precise, multi-step calculations with specialized domain knowledge (e.g., corporate finance, derivatives pricing). The core objectives are: (a) realistic multimodal integration by visually parsing complex financial images; (b) application of advanced financial expertise; and (c) execution of exact computations under strict tolerance constraints. ## 2. Dataset Composition and Domain Coverage The FinMMR dataset contains 4,300 questions paired with 8,700 images and is distributed evenly between Chinese and English (2,150 each). The questions span 14 major financial subdomains: - Corporate Finance - Banking - Asset Management - Financial Markets - Industry Analysis - Macroeconomic Analysis - Mergers & Acquisitions - Equity Valuation - Fixed Income - Derivatives & Option Pricing - Risk Management - Alternative Investments - FinTech & Digital Finance - Strategy Research Each question package includes one or more ground-truth images (plus two distractor images for the Chinese set), a numerical query, an executable Python solution script, and an exact numeric answer. This bilingual approach and breadth of domain coverage distinguish FinMMR from previous benchmarks by enabling evaluation across both languages and a wide variety of financial analyses. ## 3. Image Typology and Data Structure FinMMR’s 8,700 images are systematically categorized across 14 types, emphasizing the variety of financial visual formats encountered in real reports. The distribution is shown below: | Image Category | Number of Images | Representative Context | |------------------------------|------------------|------------------------------------| | Tables | 2,100 | Financial statements, key figures | | Bar charts | 900 | Comparative analytics, growth | | Line charts | 800 | Trends, time-series | | Pie charts | 700 | Proportional breakdowns | | Candlestick charts | 650 | Price movement analysis | | Ownership-structure charts | 700 | Shareholding analysis | | Flow diagrams | 600 | Cash flow or process visualizations| | Cash-flow diagrams | 550 | Capital movements | | Organizational charts | 450 | Hierarchy structures | | Stacked-bar charts | 450 | Category aggregations | | Area charts | 350 | Cumulative trends | | Scatter plots | 250 | Correlation exploration | | Histograms | 150 | Distributions | | Network diagrams | 100 | Relationship mapping | A salient feature is that all tabular data are rendered as images, and original text tables are removed, requiring models to perform OCR and visual parsing as a prerequisite to reasoning. ## 4. Data Curation and Complexity Stratification FinMMR construction involved two primary data curation strategies. First, 1,419 questions were adapted from extant text-only financial QA datasets (MMMU, MMMU-Pro, FinanceMath, CodeTAT-QA, CodeFinQA, DocMath-Eval); all tables were visually rendered to images. Second, 90 licensed Chinese financial research reports were processed using 360LayoutAnalysis to extract relevant images. Qwen-VL-Max was employed to generate multi-step reasoning questions, which were refined by a team of 16 finance graduates and 2 CFAs, who also provided Python solution code and final numerical answers. For greater realism, Chinese questions include two distractor images from adjacent pages. Problem difficulty is stratified using a “reasoning complexity score”: $$ rc = \ln\bigl(\max(o,1)\bigr) + \ln\bigl(\max(l + p,1)\bigr) $$ where \(o\) is the number of operators, \(l\) is the solution length (lines of code), and \(p\) is the number of parentheses pairs. Based on this score, questions are partitioned into Easy (1,300), Medium (1,500), and Hard (1,500) categories. ## 5. Problem Types and Reasoning Requirements FinMMR’s reasoning scenarios are characterized as follows: - **Easy (rc ≲ 2.0):** Single-step lookup and arithmetic from images (e.g., extracting revenue and cost, simple subtraction). - **Medium (2.0 ≲ rc ≲ 3.5):** Two-step chaining, typically involving extraction from charts and formula application (e.g., computing year-on-year growth rates). - **Hard (rc ≳ 3.5):** Multi-step calculations requiring advanced financial formulas (e.g., implementing the Black–Scholes inverse using Newton–Raphson for implied volatility). This level often mandates both precise extraction of multiple parameters and correct multi-line programmatic computation. Each problem type explicitly blends image understanding, domain expertise, and strict numeric answer conformance. ## 6. Evaluation Metrics and Baseline Performance Evaluation in FinMMR employs rigorous numerical accuracy: a model’s answer \(\hat{y}\) is correct if $$ |\hat y - y^*| / y^* \leq 0.2\% $$ where \(y^*\) is the ground-truth answer. Overall accuracy per subset follows $$ \text{Accuracy} = \frac{\text{Number of Correct Answers}}{\text{Total Questions}} \times 100\% $$ Fifteen leading proprietary and open-source MLLMs were evaluated in zero-shot settings using: IO (direct prompting), Chain-of-Thought (CoT), and Program-of-Thought (PoT) strategies. On the Hard test subset, the top performing MLLM (Claude 3.7 Sonnet, 64K extended thinking, PoT) achieved 53.0% accuracy. Performance across splits for this model is: | Difficulty | Accuracy (%) | |------------|-------------| | Easy | 78.5 | | Medium | 62.5 | | Hard | 53.0 | | Overall | 64.0 | Error analysis on 100 sampled Hard failures indicated breakdowns as follows: 30% from visual perception (misreading images), 38% from knowledge reasoning (incorrect formula/concept), and 32% from numerical computation (precision errors, calculation mistakes). ## 7. Contributions and Research Significance FinMMR introduces three core advancements: - **Multimodality:** All tabular/text data are rasterized as images; distractor images contribute to real-world document complexity and “noise.” - **Comprehensiveness:** Coverage of 14 financial subdomains, bilingual design, and 14 distinct image types collectively deliver an unprecedented breadth and realism for benchmarking. - **Challenge:** Multi-step expert reasoning, strict domain-specific numerical computations, and a precision threshold at the 0.2% error margin underscore the benchmark’s rigor. Findings underscore the current gap between MLLM performance and expert human-level reasoning in integrated visual-financial tasks, even under advanced prompting and extended context regimes. FinMMR is openly available with an online evaluation platform and is expected to facilitate further advances in multimodal reasoning, visual-textual integration, and financial AI method development [2508.04625]. Source: https://www.emergentmind.com/topics/finmmr