Financial Visual-Language Models (FinVLFMs)
- FinVLFMs are finance-specialized multimodal models that jointly process visual artifacts and text to enable comprehensive financial document analysis and decision-making.
- They utilize a three-component architecture—vision encoder, projector, and language model—to align structured financial charts and textual context.
- Current challenges include data scarcity, limited fine-grained spatial grounding, and difficulties with multi-step reasoning in high-stakes financial workflows.
Financial Visual-Language Foundation Models (FinVLFMs) are finance-specialized multimodal foundation models designed to jointly process financial visual information—such as line charts, candlestick diagrams, scanned reports, tables, and figures—and associated textual context for tasks including visual question answering, document parsing, and multimodal reasoning. In current financial foundation-model taxonomies, they form the visual-language branch alongside Financial Language Foundation Models (FinLFMs) and Financial Time-Series Foundation Models (FinTSFMs), and they also appear as the visual-language subset of broader Multimodal Financial Foundation Models (MFFMs) (Chen et al., 7 Jul 2025, Yanglet et al., 15 May 2025). Their defining premise is that financially relevant evidence is often encoded visually rather than purely in text or numerical sequences, so finance-grade reasoning requires joint visual grounding and domain-specific language competence.
1. Taxonomic position and conceptual scope
Within the recent finance foundation-model literature, FinVLFMs are defined by modality and task structure rather than by a single canonical architecture. The survey literature characterizes them as models that jointly process financial images and text to support multimodal financial understanding, in contrast to FinLFMs, which are text-centric, and FinTSFMs, which are optimized for numerical sequences such as prices, volatility, or order books (Chen et al., 7 Jul 2025). The broader MFFM framing extends this further, treating charts, tables, images, video, audio, numerical data, and time series as first-class financial modalities; in that framing, FinVLFMs are the subfamily concerned with vision-language alignment over finance-native visual artifacts such as charts, reports, tables, seals, and document images (Yanglet et al., 15 May 2025).
This framing is important because the field explicitly rejects the reduction of financial multimodality to “OCR plus chat.” VisFinEval argues that real financial practice requires extraction from visuals, alignment with text, and reasoning across the full business workflow, organized in a front-office, mid-office, and back-office decomposition (Liu et al., 13 Aug 2025). The same point appears in the survey literature: the value of FinVLFMs is not merely the recovery of surface text from images, but end-to-end financial understanding grounded in charts, tables, layouts, and domain semantics (Chen et al., 7 Jul 2025).
The visual inputs emphasized across the literature are finance-specific. The surveys and benchmarks repeatedly mention line charts, candlestick/K-line charts, financial statements, report figures, scanned reports, official seals, data tables, formulas, shareholding structure charts, webpages, application screenshots, and disclosure documents (Chen et al., 7 Jul 2025, Chen et al., 28 May 2026). This suggests that “visual-language” in finance is narrower than generic web-scale VLM multimodality in one sense—because the visual distributions are highly structured and domain-specific—but broader in another, because finance workflows often require multimodal reasoning over heterogeneous document, chart, and reporting formats rather than over natural images.
2. Architectural pattern and training methodology
The dominant architectural template in current FinVLFMs is a three-component stack: a vision encoder, a vision projector, and a base LLM (Chen et al., 7 Jul 2025). The vision encoder transforms financial visuals into embeddings; the vision projector maps those embeddings into the token space of the LLM; and the base LLM performs instruction following, reasoning, and response generation. The surveys note that current systems often rely on general-purpose visual encoders such as CLIP and relatively simple projectors such as linear layers or small MLPs, while warning that such adapters may be too weak for fine-grained financial semantics such as candlestick structure, table-cell relations, or document layout dependencies (Chen et al., 7 Jul 2025).
"FinVis-GPT" provides an early, concrete instance of this design pattern in the finance domain (Wang et al., 2023). It is built on top of LLaVA rather than introducing a new backbone from scratch, and therefore inherits the standard LLaVA-style composition: a visual encoder processes the chart image, a projection/alignment module maps visual features into language-model space, and a frozen or largely pre-trained LLM generates text conditioned on image and prompt. The model’s inputs are multimodal—chart images and, during instruction tuning, natural-language instructions or questions—and its outputs are free-form text responses covering chart descriptions, answers to finance-related questions, and future-trend predictions (Wang et al., 2023).
The training pipeline described for FinVis-GPT follows a two-stage paradigm that the survey literature treats as representative for early FinVLFMs: modal alignment pretraining followed by supervised instruction finetuning (Chen et al., 7 Jul 2025). In FinVis-GPT, the first stage aligns chart visual patterns with textual financial descriptions; the second stage teaches the model to follow financial chart-analysis instructions and generate task-specific responses (Wang et al., 2023). The formal objective is summarized as
with an implied autoregressive language-modeling loss conditioned on aligned visual embeddings: This formulation is not unique to finance, but the domain adaptation lies in the construction of financial visual-text corpora and in finance-specific instruction tuning (Wang et al., 2023).
The surveys emphasize that this alignment-plus-finetuning pattern remains dominant partly because the field is still data-constrained. FinVLFMs do not yet have a standardized training recipe beyond multimodal alignment and supervised instruction tuning, and current models remain heavily dependent on general-purpose encoders, modest multimodal adapters, and relatively small finance-specific multimodal corpora (Chen et al., 7 Jul 2025).
3. Data regimes and representative resources
A central constraint on FinVLFMs is the scarcity of large-scale, high-quality multimodal financial data. The surveys explicitly identify the lack of chart-text, table-report, and filing-image corpora as one of the main bottlenecks, noting that existing datasets are often suitable for evaluation but too small for robust large-scale pretraining or instruction tuning (Chen et al., 7 Jul 2025, Yanglet et al., 15 May 2025). This data problem is amplified by expert annotation cost, privacy restrictions, and the fact that finance-relevant visual evidence frequently appears in proprietary or regulated materials.
FinVis-GPT addresses this problem through synthetic data generation grounded in historical market data (Wang et al., 2023). For pre-training alignment, it uses historical daily stock price data of Chinese A-shares spanning 2006 to 2023. Time series are segmented into windows of 60–80 trading days, split into prompt data and predict data, and rendered as images with mplfinance, with an 80%/20% split between candlestick charts and line charts. To increase realism and diversity, the generated plots randomly include moving averages over 3, 6, and 9 days, volume bars, and different chart styles. Each example consists of an image, an instruction, and a long-form answer, with the answers generated by ChatGPT from prompts requesting professional financial analysis of K-line charts while avoiding explicit stock codes and encouraging image-like chart interpretation (Wang et al., 2023). Instruction tuning then uses a conversational dataset of about 200K sets, each containing around five questions, with Question@Answer@ formatting and future data used only during label generation (Wang et al., 2023).
Alongside model-specific corpora, the field has rapidly shifted toward benchmark construction. The benchmarks differ in language, modality coverage, workflow depth, and evaluation granularity.
| Resource | Scope | Distinctive emphasis |
|---|---|---|
| FinChart-Bench (Shu et al., 20 Jul 2025) | 1,200 real-world financial chart images; 7,016 questions | Real corporate finance charts with TF, MC, and QA |
| VisFinEval (Liu et al., 13 Aug 2025) | 15,848 Chinese QA pairs across 8 image modalities | Front–mid–back office scenario hierarchy |
| FinMTM (Zhang et al., 3 Feb 2026) | 11,133 bilingual QA pairs grounded in 3,600 images and 400 PDFs | Multi-turn dialogue and agent evaluation |
| Scribe-Finance / Multimodal Finance Eval (Mouilleron et al., 11 Feb 2026) | 1,204 French expert-validated questions | Financial documents, chart/table reasoning, multi-turn failure analysis |
| CFMME (Chen et al., 28 May 2026) | 6,052 Chinese instances across 8 image modalities and 4 tasks | QA plus detection, recognition, and information extraction |
These resources collectively show the breadth of the domain. FinChart-Bench isolates real-world financial chart comprehension (Shu et al., 20 Jul 2025). VisFinEval expands to charts, statements, seals, relationship graphs, and data tables under a scenario-driven institutional workflow (Liu et al., 13 Aug 2025). FinMTM introduces bilingual multi-turn reasoning and MCP-style agent tasks over images and PDFs (Zhang et al., 3 Feb 2026). Scribe-Finance focuses on French prospectuses, KIDs, and PRIIPs documents, with anchored excerpts and conversational settings designed to expose error propagation (Mouilleron et al., 11 Feb 2026). CFMME widens the evaluation axis further by combining knowledge assessment with application assessment over charts, seals, formulas, documents, and structured extraction tasks (Chen et al., 28 May 2026).
4. Benchmarking protocols and evaluation methodology
A defining feature of FinVLFM evaluation is the absence of a single universal metric. The field instead uses task-specific protocols, often because financial answers can be numerically precise, structurally constrained, or dependent on long-horizon interaction.
FinChart-Bench deliberately constrains all answers to be single-token and unambiguous: True or False for TF, A/B/C/D for MC, and a single numeric value for QA with units specified in the question (Shu et al., 20 Jul 2025). This permits Exact Match scoring and a weighted average across task types:
The design goal is to minimize grading ambiguity and prevent benchmark results from being dominated by judge-model variance (Shu et al., 20 Jul 2025).
FinMTM adopts a more elaborate protocol because it targets single-choice and multiple-choice questions, multi-turn open-ended dialogues, and agent tasks (Zhang et al., 3 Feb 2026). Objective questions use a strict set-overlap rule: This enforces a no-overselection constraint. Multi-turn dialogues are scored by combining turn-level capability assessments—visual precision, financial logic, data accuracy, cross-modal verification, and temporal awareness—with a session-level checklist tied to the dialogue subtype. The final score is
with (Zhang et al., 3 Feb 2026). Agent evaluation then explicitly decomposes performance into planning quality, reasoning quality, and answer correctness: This is a significant methodological shift from static chart QA toward workflow-sensitive assessment of planning, tool invocation, and evidence synthesis (Zhang et al., 3 Feb 2026).
Other benchmarks use heterogeneous protocols that reflect their task coverage. VisFinEval evaluates 21 MLLMs in a zero-shot setting and reports weighted average accuracy (WA), using Qwen-max-latest as a judge model with manual review and reported agreement above 98% (Liu et al., 13 Aug 2025). Scribe-Finance evaluates six open-weight VLMs with greedy decoding and majority-vote LLM-as-judge scoring from three independent judges; the conversational split separates Conv. Gold from Conv. to isolate error propagation (Mouilleron et al., 11 Feb 2026). CFMME uses task-specific metrics: accuracy and weighted F1 for knowledge assessment, mAP for detection, 1-NED for seal recognition, TEDS and STEDS for table recognition, CDM and ExpRate@CDM for formula recognition, field-level F1 for information extraction, and accuracy for question answering (Chen et al., 28 May 2026).
A broader implication is that financial multimodal evaluation is converging on a workflow-aware, evidence-grounded regime rather than on generic VQA accuracy alone. This suggests that FinVLFMs are increasingly being judged not only by whether they can “read” a visual artifact, but by whether they can sustain grounded reasoning across turns, tools, and task types.
5. Empirical capabilities and recurring failure modes
The empirical picture across current benchmarks is consistent: contemporary FinVLFM-like systems are often strong at shallow extraction and straightforward perception, but substantially weaker at spatial reasoning, chart abstraction, long-context consistency, and multi-step workflow execution.
At the model level, FinVis-GPT is reported to outperform LLaVA, MiniGPT-4, and mPLUG-Owl on financial chart description, question answering, and trend prediction in case-study-based evaluation (Wang et al., 2023). The paper attributes the improvement to domain-specific grounding: general-purpose multimodal LLMs reportedly misidentify chart types, confuse candlestick semantics, or hallucinate unrelated content, whereas FinVis-GPT more reliably recognizes chart structure and produces concise, financially grounded responses (Wang et al., 2023). However, the evidence is qualitative rather than benchmark-based, and the paper explicitly notes the absence of large-scale quantitative metrics and ablations (Wang et al., 2023).
Benchmark studies show a more systematic capability profile. FinChart-Bench reports that the open-source versus closed-source gap is narrowing, but that finance-specific reasoning over charts remains difficult (Shu et al., 20 Jul 2025). On overall average, Claude Sonnet 4 reaches 84.32, o3 83.89, Gemini 2.5 Pro 83.73, while strong open-source models include Mistral 3.1 at 74.37 and Qwen2.5-VL at 72.16 (Shu et al., 20 Jul 2025). The hardest task is QA: the best open-source QA score is LLaMa 4 at 59.78, and the best closed-source QA score is Claude Sonnet 4 at 63.59 (Shu et al., 20 Jul 2025). The same benchmark also reports that many chart-specialized open-source models perform very poorly in this finance setting—e.g., UniChart 17.52, MatCha 17.05, and ChartGemma 16.87 average—while general-purpose models do substantially better (Shu et al., 20 Jul 2025). This directly challenges the assumption that chart-specific finetuning automatically transfers to finance.
VisFinEval reaches a complementary conclusion at the workflow level. In zero-shot evaluation, Qwen-VL-max attains an overall accuracy of 76.3%, surpassing non-expert humans at 56.4 but trailing financial experts by more than 14 points overall, with expert overall average reported as 88.0 (Liu et al., 13 Aug 2025). Performance also degrades by business depth: for Qwen-VL-max, front-office tasks score 85.8, mid-office tasks 76.9, and back-office tasks only 59.1 (Liu et al., 13 Aug 2025). The error analysis identifies six recurring failure modes: lack of cross-modal information alignment capability, market sentiment and semantic tendency misjudgment, bias in the understanding of financial terms and indicators, perceived barriers to financial business processes, hallucination generation and irrational reasoning, and financial subject identification and causation confusion (Liu et al., 13 Aug 2025).
FinMTM shows that moving from static perception to interactive reasoning exposes further weaknesses (Zhang et al., 3 Feb 2026). The benchmark evaluates 22 VLMs and finds that proprietary models generally outperform open-source models, especially on harder multi-turn and agentic tasks. Gemini 3 Pro is strongest on open-ended tasks overall; ChatGPT-5 is strongest on multiple-choice objective questions; and Gemini 3 Flash performs best on agent tasks (Zhang et al., 3 Feb 2026). The memory setting is especially difficult: even Gemini 3 Pro reaches only 48.5 on memory, while InternVL2.5-8B scores 16.7 (Zhang et al., 3 Feb 2026). Turn-level analysis indicates that models tend to do better on visual precision and temporal awareness than on financial logic, data accuracy, and cross-modal verification (Zhang et al., 3 Feb 2026).
Scribe-Finance exposes a similar pattern in French financial document understanding (Mouilleron et al., 11 Feb 2026). Text questions are handled well, around 88–90% accuracy, and table comprehension can reach 85.8% for top models. Chart interpretation is much weaker, ranging from 34.4% to 61.7%, with Qwen3-VL-32B achieving the best average score of 75.6 across categories (Mouilleron et al., 11 Feb 2026). The most striking result concerns dialogue: in Conv. Gold, performance ranges from 63.1% to 86.2%, but in standard Conv. it drops to 46.2% to 58.5% regardless of model size (Mouilleron et al., 11 Feb 2026). The paper interprets this as evidence of cascading error propagation rather than mere undercapacity (Mouilleron et al., 11 Feb 2026).
CFMME broadens the empirical diagnosis to detection, recognition, and extraction tasks in Chinese financial settings (Chen et al., 28 May 2026). The best QA model, Qwen3-VL-235B-A22B-Thinking, reaches 66.11% overall QA accuracy, while the best overall score on recognition, detection, and information extraction is 77.18 for Qwen3-VL-235B-A22B-Instruct (Chen et al., 28 May 2026). Yet detection remains a bottleneck, with that model reporting mAP 30.27, and the benchmark shows sensitivity to image orientation: for Qwen3-VL-235B-A22B-Thinking, application QA drops from 54.94 at to 41.27–43.29 under 90°, 180°, and 270° rotations (Chen et al., 28 May 2026). The same paper also reports that HTML representations of tables systematically outperform raw table images for all tested Qwen3-VL models, indicating that dense financial tables remain harder to parse visually than as structured text (Chen et al., 28 May 2026).
6. Limitations, misconceptions, and research directions
Several misconceptions recur in discussion of FinVLFMs. One is that multimodal financial modeling is primarily an OCR problem. The benchmark and survey literature consistently rejects this view, showing that current systems may perform strongly on text extraction or standardized table reading while still failing on chart interpretation, cross-modal grounding, business-process reasoning, or interactive analysis (Liu et al., 13 Aug 2025, Mouilleron et al., 11 Feb 2026). Another misconception is that larger or newer models necessarily solve these issues. FinChart-Bench documents performance degradation within upgraded model families on some subtasks, and Scribe-Finance shows that error accumulation in multi-turn dialogue persists regardless of model size (Shu et al., 20 Jul 2025, Mouilleron et al., 11 Feb 2026).
The major limitations of present FinVLFMs are also stable across sources. The surveys emphasize data scarcity, high expert annotation cost, privacy and confidentiality constraints, weak domain-specific visual encoding, simple multimodal adapters, insufficient benchmark standardization, and persistent hallucination risk (Chen et al., 7 Jul 2025, Yanglet et al., 15 May 2025). Benchmark papers add more operational diagnoses: weak fine-grained spatial grounding on charts (Shu et al., 20 Jul 2025), poor long-context memory and cross-page evidence linking (Zhang et al., 3 Feb 2026), sensitivity to perturbation and rotation (Liu et al., 13 Aug 2025, Chen et al., 28 May 2026), and fragility in multi-turn or agentic workflows where early mistakes corrupt later reasoning (Mouilleron et al., 11 Feb 2026, Zhang et al., 3 Feb 2026).
The literature points to several concrete directions for progress. One is richer multimodal modeling: the surveys explicitly suggest moving beyond simple MLP or linear projectors toward cross-attention, gating layers, and dynamic adapters in order to preserve fine-grained relations between chart elements, table cells, and surrounding text (Chen et al., 7 Jul 2025). Another is stronger domain-specific vision encoding for financial charts, tables, and document layouts rather than continued dependence on generic encoders (Chen et al., 7 Jul 2025). Data expansion is a further priority, especially larger, realistic chart-text, table-report, and filing-image corpora with richer instruction diversity (Chen et al., 7 Jul 2025).
Grounding and uncertainty management are equally central. The surveys recommend retrieval-augmented generation and knowledge-graph support to reduce hallucinations and improve factual reliability in high-stakes settings (Chen et al., 7 Jul 2025). FinMTM further suggests that training should explicitly target long-context reasoning, self-correction, and tool-use planning rather than relying on single-turn QA competence as a proxy for financial reasoning (Zhang et al., 3 Feb 2026). A plausible implication from open-set domain-adaptation research is that entropy-based known/unknown separation and source-free adaptation may become useful for FinVLFMs operating under cross-market, cross-style, or cross-language distribution shifts, although this transfer remains prospective rather than established in finance-specific experiments (Yu et al., 2023).
Overall, the literature portrays FinVLFMs as an early but increasingly structured research area. They already support chart description, visual QA, document analysis, and limited predictive or decision-support functions, and top systems can exceed non-expert human performance on some benchmarked tasks (Wang et al., 2023, Liu et al., 13 Aug 2025). At the same time, current models remain far from expert-level financial reasoning, especially when tasks require precise chart abstraction, multi-step numerical logic, workflow consistency, robust grounding, or interactive agent behavior. In that sense, FinVLFMs are not yet a mature finance-grade substrate, but they have become a clearly defined and empirically measurable foundation-model class whose future development depends on better data, stronger multimodal alignment, and more realistic evaluation.