FinChart-Bench: Financial Chart Benchmark

• FinChart-Bench is a dedicated benchmark for evaluating multimodal financial chart comprehension using 1,200 real corporate charts and 7,016 QA pairs.
• It employs a five-stage pipeline with extensive manual filtering and dual QA review to ensure high-quality, reproducible evaluation across True/False, Multiple Choice, and numerical tasks.
• Empirical results reveal that while LVLMs excel in simple chart tasks, they struggle with spatial reasoning and numerical QA, underscoring complex challenges in financial chart understanding.

FinChart-Bench is a benchmark for evaluating large vision-LLMs on real-world financial chart understanding. Introduced as the first benchmark specifically focused on financial charts, it comprises 1,200 financial chart images collected from 2015 to 2024 and 7,016 questions spanning True/False, Multiple Choice, and numerical Question Answering formats. Its defining design choices are the use of real corporate finance presentation charts, single-token answers for Exact Match scoring, and two full rounds of manual review over the dataset, making it a finance-domain evaluation resource for multimodal chart comprehension rather than a general chart corpus or a text-only finance benchmark (Shu et al., 20 Jul 2025).

1. Scope and position in the benchmark landscape

FinChart-Bench was introduced to address three gaps in prior chart-understanding evaluation. The first is the absence of a dedicated benchmark for real-world financial charts. The second is the ambiguity of evaluation in many existing chart benchmarks, especially when free-form answers require normalization or judge models. The third is the authors’ skepticism that current LVLMs are reliable enough to serve as automated judges during benchmark construction. In the benchmark’s framing, finance remains “notably underexplored” despite its practical importance and its distinctive visual and semantic demands (Shu et al., 20 Jul 2025).

The benchmark treats financial charts as a special case of chart understanding rather than as a simple domain adaptation of generic chart QA. Financial charts are described as involving complex temporal patterns, market indicators, domain-specific terminology, dense chart annotations, trend and price/volume interpretation, and complex multi-element layouts. This makes transfer from open-domain, synthetic, or scientific chart benchmarks nontrivial. The benchmark therefore targets the intersection of real-world charts, finance, reproducible evaluation, and human-verified quality.

This domain emphasis also distinguishes FinChart-Bench from broad chart datasets that prioritize taxonomy coverage over finance specificity. For example, ChartComplete broadens chart coverage to 30 chart types across 12 categories, but it is a classified image collection without a learning signal and is explicitly not focused on finance (Mustapha et al., 15 Jan 2026). FinChart-Bench instead concentrates on a narrower but domain-grounded slice of the chart landscape: investor and corporate financial communication.

2. Corpus sources and benchmark construction

The source material for FinChart-Bench is a corpus of corporate presentation slides spanning 2015–2024. These slides were gathered from Bloomberg News and official corporate websites, and they include both firm-hosted settings such as corporate-sponsored conferences and third-party venues such as investment bank-sponsored events. The authors note that the majority were externally hosted and were not traditional roadshows (Shu et al., 20 Jul 2025).

Benchmark construction follows a five-stage pipeline: Data Source, Chart Extraction, Chart Filtering, QA Generation, and QA Evaluation & Correction. For chart extraction, each PDF page is converted to a high-resolution image, and Qwen2.5-VL-7B-Instruct is used to identify charts by generating bounding boxes. The extraction prompt asks the model to outline the full position of each complete chart, including title, axes, labels, legends, and data visualizations, and to return coordinates in JSON format. This stage yields more than 130,000 charts.

Because extraction quality varies substantially, the authors then apply comprehensive manual filtering. Each candidate is evaluated against four criteria: it must actually be a chart; it must be complete and legible; it must be high-resolution and visually clean; and it must be neither too simple nor too complex. The filtering policy is conservative, and borderline or uncertain charts are discarded. After this stage, over 71,000 high-quality chart images remain, averaging about 7,000 images per year. From these, the authors manually select 120 highest-quality chart images per year over 10 years, producing the final 1,200-chart image set.

Question generation is then performed with GPT-4.1. For each chart, the model generates exactly two True/False questions, two Multiple Choice questions, and two numerical reasoning QA questions, yielding 7,200 initial question-answer pairs. The final stage is a second round of manual review. Each pair is checked and either left unchanged, corrected if the answer is wrong, or removed if the question is unclear or confused. This produces the final benchmark size of 7,016 questions.

Manual review is a central methodological feature. The benchmark underwent two rounds of human evaluation: chart filtering and QA evaluation/correction. All manual annotations were conducted by the authors. The reported annotation rate is about 1,000 images per hour per annotator for chart filtering and about 80 QA pairs per hour per annotator for QA evaluation. The human evaluation effort ran from early March 2025 to end of May 2025 (Shu et al., 20 Jul 2025).

3. Task structure, chart taxonomy, and annotations

FinChart-Bench includes three task formats. True/False questions use the answer space True or False. Multiple Choice questions use four options, A, B, C, and D, with the ground truth given as the correct letter only. Question Answering is reasoning-oriented but numerically constrained: the model must output a single number, formatted as a floating-point number without punctuation or units, with the required unit specified in the question. For QA generation, the reasoning trace is stored as metadata, but only the final single-number answer is used as ground truth for evaluation (Shu et al., 20 Jul 2025).

Task	Answer form	Final count
True/False	True or False	2,383
Multiple Choice	A / B / C / D	2,349
Question Answering	single numeric token	2,284

A key design principle is that every answer can be evaluated as a single token. This allows the benchmark to use Exact Match without ambiguous normalization or judge-based answer interpretation. The benchmark is therefore designed to combine complex chart reasoning with simple answer forms.

The chart inventory covers 14 distinct chart types: Line, Bar, Pie, Bar with num, Line with num, Pie with num, Bar with line, Horizontal bar, Ring, Histogram, Area, Line with area, Bar with area, and Line with scatter. The paper states that “Bar with num” dominates at 40% of the benchmark. The task design consequently mixes conventional chart forms with variants in which values are either directly printed or must be inferred from marks and axes.

The reasoning demands are described through the kinds of operations the questions require. These include visual chart reading, comparison, trend identification, numerical lookup, arithmetic reasoning, growth and difference calculations, percentage-style computations, alignment of chart marks with axes, and spatial reasoning over bars, lines, points, and regions. The benchmark text specifically mentions questions involving growth rates, differences, percentages, and trends.

The paper does not provide train/validation/test splits in the supplied description. FinChart-Bench is presented primarily as an evaluation benchmark, and no explicit split protocol is specified in the text (Shu et al., 20 Jul 2025).

4. Evaluation protocol and benchmarked models

FinChart-Bench uses Exact Match as its primary metric. Because all answers are constrained to single tokens, Exact Match is intended to be reliable and unambiguous. The overall score is a weighted average across the three task types, with weights given by the number of questions in each task: 2,383 for True/False, 2,349 for Multiple Choice, and 2,284 for Question Answering (Shu et al., 20 Jul 2025).

To standardize inference, each question is paired with an instruction requiring the model to respond in the form Result = [[ answer ]]. The paper notes that many weaker models fail to follow this instruction reliably, especially on Multiple Choice tasks. This answer-wrapper protocol is part of the benchmark’s attempt to make evaluation reproducible without resorting to post hoc LLM judging.

The evaluation covers 25 LVLMs in three groups. The first group consists of general-purpose open-source LVLMs, including models such as LLaMa 3.2, Qwen2.5-VL, Mistral 3.1, DeepSeek-VL2, Gemma 3, and LLaMa 4. The second group consists of chart-specialized open-source models, including UniChart, Matcha, ChartGemma, ChartInstruct-LLaMA2, and ChartInstruct-FlanT5. The third group consists of closed-source proprietary systems, including GPT-4.1, GPT-4o, o3, o4-mini, Claude Sonnet 4, and Gemini 2.5 Pro. All experiments except LLaMa 4 run on one NVIDIA A100 SXM4 GPU with 80GB; LLaMa 4 uses four such GPUs, totaling 320GB (Shu et al., 20 Jul 2025).

This evaluation protocol sharply contrasts with adjacent finance benchmarks that rely on judge models. FINESSE-Bench, for example, evaluates finance knowledge and technical analysis through text questions and scores all tasks through a GPT-5.2 judge under an LLM-as-judge paradigm (Stanishevskii et al., 14 May 2026). FinChart-Bench instead adopts a visually grounded setting and avoids judge-based scoring by constraining answers to exact-matchable single tokens.

5. Empirical findings on LVLM performance

The benchmark’s main empirical picture is that True/False and Multiple Choice are relatively easy for strong models, whereas numerical QA remains difficult for all evaluated systems. Among closed-source models, Claude Sonnet 4 achieves the best weighted average at 84.32, with 96.94 on True/False, 91.66 on Multiple Choice, and 63.59 on QA. Other top closed-source results include o3 at 83.89, Gemini 2.5 Pro at 83.73, o4-mini at 83.53, and GPT-4.1 at 80.88. Among open-source models, Mistral 3.1 is the best overall at 74.37, followed by Qwen2.5-VL at 72.16. LLaMa 4 is notable for a QA score of 59.78 despite a lower overall average of 69.89 (Shu et al., 20 Jul 2025).

These results support the paper’s claim that the gap between top open-source and closed-source models is narrowing, especially on True/False and Multiple Choice. The same experiments also show that financial chart reasoning remains unsolved: on QA, even the strongest systems stay in the low 60s. This suggests that the core difficulty lies not in answer formatting alone but in extracting and manipulating financial chart evidence.

A second finding is that model-family upgrades do not uniformly improve performance. The paper highlights regressions within families: LLaMa 4 improves strongly over LLaMa 3.2 overall but scores worse on Multiple Choice, while GPT-4.1 mini outperforms GPT-4.1 on Multiple Choice despite the latter’s stronger overall profile. Large within-family gaps also appear in Qwen and LLaMa, indicating that benchmark performance is sensitive to model-generation changes rather than following a simple scaling trend.

A third major result concerns instruction following. Several chart-specialized or smaller models perform disastrously on Multiple Choice because they fail to obey the required output format. The paper reports MC scores of 0.0 for UniChart and Matcha, 0.29 for ChartGemma, 0.17 for ChartInstruct-LLaMA, 2.43 for nanoVLM, 0.77 for Blip-2, and 6.77 for DeepSeek-VL2. The authors interpret this not purely as chart-understanding failure but as a failure of structured answer production, and they explicitly hypothesize that aggressive chart-specific fine-tuning may trade off against general instruction-following ability.

Spatial reasoning is identified as a major bottleneck. Advanced models perform poorly on QA for chart types such as Line, Bar, Line with Scatter, Area, and Line with Area, but much better on Line with Number and Bar with Number. The explanation given in the paper is that in “with Number” chart types, values are explicitly printed near the mark, allowing the model to rely on text extraction rather than value inference from spatial alignment. In standard line and bar charts, by contrast, the model must localize the mark, align it to the axes, read off the value, and often then perform arithmetic. The benchmark therefore isolates a substantive distinction between OCR-like extraction and genuine chart understanding.

The paper also argues that current LVLMs are not reliable as automated evaluators. During benchmark construction, Qwen2.5-VL extracted over 10,000 charts per year, but after manual filtering almost half were discarded because of incompleteness, irrelevance, or low quality. Combined with the benchmarked models’ failures in instruction following and spatial reasoning, this motivates the benchmark’s insistence on human verification (Shu et al., 20 Jul 2025).

6. Limitations, adjacent benchmarks, and research role

FinChart-Bench has several limitations stated or implied in the paper. The benchmark size is constrained by human labor: although over 71,000 filtered charts were available, only 1,200 were retained because QA verification was labor-intensive. The source distribution is also specific, since charts come from Bloomberg News, official corporate websites, and corporate or executive presentations. This means the benchmark reflects that style of financial communication rather than every variety of financial chart. The single-token answer design improves reproducibility but restricts expressiveness, and the supplied text does not provide an explicit train/dev/test split (Shu et al., 20 Jul 2025).

Within the broader literature, FinChart-Bench occupies a specific niche. FINESSE-Bench evaluates technical analysis, trading, derivatives, and hierarchical financial competence, but it does so through text questions rather than chart images; it is therefore text-only but chart-domain-aware, not visually grounded (Stanishevskii et al., 14 May 2026). Chart-to-Text evaluates chart summarization under table-available and image-only settings and shows the importance of factuality in generated chart descriptions, but it is domain-agnostic and designed around generation rather than exact-match financial chart QA (Kantharaj et al., 2022). ChartComplete offers much broader visualization taxonomy coverage, with 30 chart types across 12 categories, but it is a classified image collection without QA pairs or direct task supervision (Mustapha et al., 15 Jan 2026).

A common misconception is that strong performance on a finance benchmark or on a general chart benchmark implies robust financial chart comprehension. FinChart-Bench’s results argue against that assumption. A model may know finance terminology textually, or perform well on simpler chart categories, yet still fail on spatially grounded reasoning over real investor-slide charts. Conversely, a model may produce fluent financial language while lacking the ability to read the underlying figure. FinChart-Bench is therefore best understood as a modality-specific and domain-specific stress test: it measures how well LVLMs can interpret real financial charts under a reproducible scoring protocol.

Its research role is consequently dual. Methodologically, it demonstrates that tightly constrained answer design and heavy human validation can make multimodal chart evaluation reproducible without LLM judging. Scientifically, it shows that current LVLMs remain brittle on financial chart reasoning, especially when exact values must be read from marks rather than copied from printed numbers. This suggests that future work will need better spatial grounding, more robust instruction following, and stronger chart-specific reasoning systems before financial chart understanding can be treated as reliable (Shu et al., 20 Jul 2025).