VisFinEval: Chinese Multimodal Financial Benchmark
- VisFinEval is a large-scale Chinese multimodal benchmark designed to evaluate MLLMs' financial reasoning across front-, middle-, and back-office workflows.
- It integrates diverse financial image modalities and process-aware, scenario-driven QA tasks that simulate real-world financial operations.
- Benchmark findings reveal strong front-office performance while highlighting challenges in expert-level back-office risk and asset optimization tasks.
VisFinEval is a large-scale, Chinese multimodal benchmark purpose-built to measure how well multimodal LLMs (MLLMs) understand and reason across the full front–middle–back office lifecycle of real financial operations. It was introduced as “the first large-scale Chinese benchmark that spans the full front-middle-back office lifecycle of financial tasks,” and comprises 15,848 annotated question-answer pairs drawn from eight common financial image modalities, organized into three hierarchical scenario depths: Financial Knowledge & Data Analysis, Financial Analysis & Decision Support, and Financial Risk Control & Asset Optimization (Liu et al., 13 Aug 2025). The benchmark is scenario-driven and process-aware, targets Chinese-language financial content and visuals that practitioners actually encounter, and simulates realistic environments via multi-turn interactions, multi-image reasoning, counterfactuals, and controlled perturbations. In this design, VisFinEval functions not merely as a visual question answering dataset, but as a process-aligned evaluation framework for holistic financial understanding by MLLMs.
1. Position within multimodal financial evaluation
VisFinEval was motivated by gaps in prior benchmarks that either focus on text-only tasks or cover limited visual modalities without mapping tasks to actual business workflows (Liu et al., 13 Aug 2025). Text-only financial benchmarks such as FinEval, CFBenchmark, SuperCLUE-Fin, and FinDABench test language understanding but miss the charts, seals, and complex documents that drive actual decisions. Document and tabular QA datasets such as TAT-QA, DocFinQA, DocTabQA, and FinQA/ConvFinQA/FinTextQA emphasize structured and long-context financial text and tables, but do not span a comprehensive visual modality set and do not explicitly align tasks to front–middle–back office workflows. Existing multimodal finance benchmarks, including FinVQA, FIN-FACT, FAMMA, MME-Finance, and FinTMMBench, include visual tasks but tend to be smaller scale, narrower in business scope, less stratified by operational depth, or limited in modality diversity.
The benchmark therefore advances coverage breadth, task depth, business fidelity, and realistic noise simulation. Its organizing principle is not only modality diversity but operational fidelity: tasks are structured to mirror how real finance teams work end-to-end. This distinguishes it from multimodal RAG benchmarks such as FinRAGBench-V, which emphasize retrieval over visually rich PDF pages, answer generation, and visual citation in a bilingual finance setting (Zhao et al., 23 May 2025), and from visualization-generation benchmarks such as VisEval, which evaluate static charts generated from natural language in Python under validity, legality, and readability criteria (Chen et al., 2024). A plausible implication is that VisFinEval occupies a complementary position: it evaluates financial multimodal understanding in workflow-grounded scenarios rather than retrieval-grounded citation or NL2VIS generation.
2. Scenario hierarchy and modality design
VisFinEval defines three layers that map to front-, middle-, and back-office workflows (Liu et al., 13 Aug 2025). The front-office layer, Financial Knowledge & Data Analysis, contains 8,700 items across 7 sub-scenarios. The mid-office layer, Financial Analysis & Decision Support, contains 4,650 items across 4 sub-scenarios. The back-office layer, Financial Risk Control & Asset Optimization, contains 2,498 items across 4 sub-scenarios. The benchmark documentation also reports individual counts for the front-office sub-scenarios: Financial Data Statistics 3,655, Candlestick Chart Analysis 1,124, Financial Indicator Assessment 1,160, Financial Entity Relationships Interpretation 919, Stock Selection Strategies Backtesting 719, Financial Information Extraction 924, and Financial Seal Recognition 199.
The front-office scenarios emphasize perception and basic analysis. Financial Data Statistics involves organizing and comparing enterprise or market data, such as determining whether multiple indicators showed negative growth in September and October by reading a time series. Candlestick Chart Analysis requires interpreting K-line patterns and technical indicators, including MACD crossovers and trend direction. Financial Indicator Assessment focuses on locating specific items and years in statements and matching options. Financial Entity Relationships Interpretation requires parsing ownership structures and aggregating indirect holdings. Stock Selection Strategies Backtesting uses candlestick features such as long lower shadows to infer backtest rule implications. Financial Information Extraction emphasizes precise localization in tables and extraction of a target value. Financial Seal Recognition requires reading text and hierarchy in seals to match candidate institutions.
The mid-office scenarios center on analytical decision support. Financial Scenario Analysis includes identifying typical events and contextual modeling, with counterfactual adjustments such as decreasing market share by 5% and checking whether a category remains largest. Industry Analysis & Inference uses sector-trend charts and transmission logic to futures or stock impact. Investment Analysis requires reading multiple rate curves and assessing liquidity easing, OBFR volatility, and short-term money market risk. Financial Market Sentiment Analysis relates sentiment indexes and trend co-movements for predictive reasoning.
The back-office scenarios emphasize risk control and asset optimization. Financial Strategy Optimization involves optimizing pricing or cost margins under constraints and checking multimodal consistency between a visual table and its Markdown text. Financial Risk & Policy Analysis requires extracting percentage fluctuations and benchmarking against risk thresholds. Financial Data Reasoning & Interpretation requires multi-step extrapolation of overhead ratios across periods from statement data. Asset Allocation Analysis evaluates shareholding thresholds and classification from equity maps.
The eight image modalities were selected for high frequency in real workflows. They are Financial Relationship Graph, Line Chart, Histogram, Candlestick Chart, Pie Chart, Official Seal, Financial Statement, and Supporting Data Table (Liu et al., 13 Aug 2025). Each modality is associated with specific financial operations: Financial Relationship Graph supports ownership and control path reasoning; Line Chart supports trend analysis and temporal comparisons; Histogram supports dispersion and comparative statistics; Candlestick Chart supports price action and technical signals; Pie Chart supports composition analysis; Official Seal supports document compliance, authenticity, and organizational alignment; Financial Statement supports line-item extraction and indicator computation; Supporting Data Table supports precise localization, extraction, and cross-row or cross-column computations.
3. Dataset construction, annotation, and realism mechanisms
The sources used to construct VisFinEval include financial PDF research reports and annual reports for line charts, histograms, pie charts, financial relationship graphs, financial statements, and supporting data tables; Chinese CPA and actuary professional exams for advanced calculations and reasoning; public financial websites for K-line charts; and an open-source seal dataset, TrOCR-Seal-Recognition (Liu et al., 13 Aug 2025). The paper states that images were verified as free of copyright restrictions, while also noting that it does not detail anonymization or de-identification beyond source curation.
Scenario-specific prompts designed with financial experts guided Qwen-VL-Plus to generate QA items from images, and Qwen-max was then used to classify QA pairs into the scenario taxonomy. The answer formats include multiple-choice with 3 or 4 options, multiple-answer, true/false, open-ended, and numerical reasoning. Answers can be numerical, categorical, or textual. This construction pipeline indicates that VisFinEval is both model-assisted and expert-constrained.
Quality control proceeds in three stages. First, Qwen-VL-Plus-latest performs automated filtering with prompt-driven criteria covering image information density, semantic validity, diversity, objectivity, and computational complexity; the paper reports that heatmaps show strong alignment with human judgments. Second, six trained undergraduate finance students manually annotate for correctness, domain specificity, answer verifiability, visual completeness, scenario alignment, and logical coherence. Third, three financial experts with more than 10 years experience perform a final cyclical review, and unanimous approval is required for each QA item. The paper does not report inter-annotator agreement such as Cohen’s kappa, and it does not report train, validation, or test splits, because VisFinEval is designed primarily as an evaluation benchmark.
A defining feature of the benchmark is realistic environment simulation. The dataset includes multi-turn dialogues of up to six turns, multi-image tasks with 2–4 charts per question, simple and complex perturbations such as occlusion, image replacement, irrelevant text, and missing information, multimodal consistency checks between image content and Markdown text, and long-instruction reasoning with varying text length and image counts (Liu et al., 13 Aug 2025). This suggests that benchmark difficulty is not reducible to static chart reading alone; it also targets context retention, cross-image integration, counterfactual reasoning, and robustness under noise.
4. Evaluation protocol, scoring, and benchmark results
The evaluation protocol is zero-shot, and 21 state-of-the-art MLLMs were tested: 9 closed-source models and 12 open-source models (Liu et al., 13 Aug 2025). Open-source models were run on NVIDIA A800 GPUs. Some models faced context window or multi-image input limits, and those items were excluded for such models. Because instruction-following varies across models, the authors used a judge model, Qwen-max-latest with tailored prompts, to extract answers and score outputs; manual audits found judge accuracy exceeded 98%.
The reported metrics are accuracy per sub-scenario and a Weighted Average score aggregated across sub-scenarios and layers. The formal definitions are:
where is the number of evaluated items, the predicted answer, and the ground truth, and
for a weighted average across categories with known weights summing to $1$. The exact weights follow task distributions. No precision, recall, , or loss functions are reported.
The headline result is that Qwen-VL-max ranks first overall with 76.3% WA accuracy across all scenarios and ranks first in 10 of 15 sub-scenarios (Liu et al., 13 Aug 2025). It performs particularly strongly on front-office tasks, including Financial Information Extraction at 90.6% and Candlestick Chart Analysis at 90.5%. InternVL3-78B is the strongest open-source model with 72.5% WA overall and 76.4% WA in mid-office, within 3.8 percentage points of Qwen-VL-max. GPT-4o-2024-11-20 achieves 68.5% WA overall. Qwen2.5-VL-72B reaches approximately 71.0% WA, and Doubao-1.5-vision-pro-32k approximately 71.7% WA. Step-1o-vision-32k reaches 98.0% accuracy in Financial Seal Recognition but is weaker elsewhere, while Claude-3-7-Sonnet-20250219 scores 34.7% in Financial Seal Recognition and often misaligns semantics. LLaVA variants are constrained by context window limits and multi-image handling; LLaVA-NeXT-34B achieves near 88% in mid-office Industry Analysis & Inference and Investment Analysis but 12.7% in back-office WA.
Performance declines with scenario depth: models do well on front-office perception and basic analytics, fare moderately in mid-office decision support, and drop substantially in back-office risk and optimization. Human baselines were measured on a sampled 2% of items, approximately 300 questions, under closed-book conditions with participants uninvolved in annotation. Non-experts averaged 56.4%, the financial expert averaged 88.0%, and Qwen-VL-max averaged 73.9% overall, with 85.8 on Financial Knowledge & Data Analysis, 76.9 on Financial Analysis & Decision Support, and 59.1 on Financial Risk Control & Asset Optimization. InternVL3-78B averaged 70.4%, GPT-4o-2024-11-20 averaged 66.1%, and Qwen2.5-VL-72B averaged 68.7%. The benchmark’s stated headline conclusion is that Qwen-VL-max surpasses non-experts overall but trails financial experts by over 14 percentage points, indicating that expert-level back-office reasoning remains elusive for current MLLMs (Liu et al., 13 Aug 2025).
5. Error taxonomy and interpretive significance
The authors sampled 10% of incorrect outputs across models and identified six recurring failure modes (Liu et al., 13 Aug 2025). Cross-modal information misalignment refers to failure to align visual and textual representations of the same data, such as giving a correct answer from a Markdown table but an incorrect answer from the visually identical image version. Market sentiment and semantic tendency misjudgment refers to over- or mis-interpretation of sentiment signals and macro indicators, such as focusing on a local peak while ignoring broader downtrend pressure. Bias in understanding financial terms and indicators refers to confusion among technical patterns or indicator definitions, for example misclassifying candlestick patterns while correctly extracting time points and shapes. Perceived barriers to financial business processes refers to inability to follow multi-step business logic and dependencies, so that early mistakes cascade through later turns in multi-round QA. Hallucination generation and irrational reasoning refers to inventing facts or relying on irrelevant attributes when evidence is missing, particularly in noisy or ambiguous inputs. Financial subject identification and causation confusion refers to failure to distinguish financial subjects or to separate correlation from causation in entity relations.
These failure modes correlate with modality and scenario difficulty. Back-office tasks in risk and policy analysis, asset allocation, and strategy optimization require precise multi-step computation and causal reasoning under noise, and the benchmark reports that models falter most strongly there (Liu et al., 13 Aug 2025). Seals and charts in Chinese require robust OCR, lexicon knowledge, and domain calibration. This suggests that current performance ceilings are not explained by OCR or chart parsing alone; they also involve business-process reasoning, dependency tracking, and domain-specific semantic discrimination.
The benchmark’s reported strengths and gaps follow directly from this taxonomy. Top models already show strong front-office perception in chart reading, table extraction, statement line-item localization, and in some cases seal recognition. Mid-office decision support is improving, and the small gap between InternVL3-78B and Qwen-VL-max suggests convergence among top-tier models. By contrast, back-office reasoning remains challenging because it combines multi-step numeric reasoning, cross-modal consistency, and business-process logic dependencies (Liu et al., 13 Aug 2025). A plausible implication is that future model improvements in finance will depend less on generic multimodal scale alone and more on domain calibration, explicit workflow modeling, and robustness to controlled perturbations.
6. Uses, reproducibility, and limitations
VisFinEval is intended as a practical evaluation benchmark for multimodal financial intelligence (Liu et al., 13 Aug 2025). The recommended evaluation protocol is the zero-shot setup with standardized prompts provided in the repository and appendices. The authors recommend judge-model scoring for answer extraction and evaluation, specifically Qwen-max-latest, and recommend reporting per-sub-scenario accuracy together with WA per layer and overall. They also recommend robustness checks under the benchmark’s perturbation settings: key information occlusion, redundant image perturbation with irrelevant charts or images, missing relevant information, and irrelevant information perturbation as semantic noise. Multi-turn dialogues and multi-image tasks are also recommended for probing instruction following, context retention, and cross-chart integration.
For model development, the benchmark is proposed as a diagnostic instrument for domain-specific fine-tuning, including Chinese OCR, financial lexicon, and chart-reading modules. The paper also recommends process-aware reasoning strategies such as tool-use for calculations, explicit step-checking, and causal templates for policy and risk analysis, as well as multi-representation consistency constraints between image and text tables to improve cross-modal alignment (Liu et al., 13 Aug 2025). These recommendations are framed as directions rather than reported benchmark components.
The repository hosts dataset files, prompts for generation, verification, and classification, documentation of scenario taxonomy, and evaluation scripts. The appendices include prompt templates for constructing multiple-choice questions, counterfactuals, multi-turn dialogues, and true/false judgments; quality verification prompts; scenario classification prompts; model lists and evaluation details; and dataset distributions per sub-scenario. The benchmark is available at the SUFE-AIFLM-Lab repository.
Several limitations are explicitly identified. While VisFinEval includes trend analysis and multi-turn tasks, it has limited coverage of more dynamic, time-sensitive micro and macro market scenarios. Current evaluation focuses on zero-shot rather than few-shot or fine-tuning. Real-world importance of sub-scenarios varies by institution and role, so alternative business scenario weights could better reflect practical performance. The benchmark is Chinese-only, and multilingual extensions are identified as future work. The paper verifies copyright status but does not detail de-identification or bias audits, and future releases are suggested to include bias analyses, explicit PII handling, and governance (Liu et al., 13 Aug 2025). Taken together, these limitations indicate that VisFinEval is broad in process coverage but still bounded by language scope, evaluation regime, and incomplete treatment of data governance.
In sum, VisFinEval demonstrates that leading MLLMs can already exceed non-expert human performance on multimodal financial tasks, especially in front-office perception, but remain far from expert-level back-office reasoning (Liu et al., 13 Aug 2025). Its scenario-stratified design, realistic perturbations, and broad modality coverage make it a benchmark for measuring how well multimodal systems integrate textual and visual financial information across actual business workflows.