FinTral: Compact Multimodal Finance Model

• FinTral is a compact multimodal language model specialized for finance, integrating text, numbers, tables, and images.
• It employs finance-specific pretraining, instruction fine-tuning, and direct preference optimization to achieve competitive benchmark performance.
• Its efficient design supports real-time analysis with advanced tool integration and retrieval-augmented generation for precise financial insights.

FinTral is a family of compact (7-billion-parameter) multimodal LLMs developed for financial analysis. Built upon the open-source Mistral-7B backbone, FinTral integrates textual, numerical, tabular, and image modalities, and is specialized for finance via domain-specific pretraining, instruction fine-tuning, and alignment with Reinforcement Learning from AI Feedback (RLAIF). The FinTral-DPO-T&R variant incorporates advanced tools and retrieval methods through a direct preference optimization (DPO) framework, achieving performance competitive with or superior to proprietary models on rigorous financial benchmarks (Bhatia et al., 16 Feb 2024).

1. Architecture and Multimodal Integration

FinTral adopts the Mistral-7B model as its base, utilizing a standard transformer architecture augmented with FlashAttention-2 for optimized attention computation. It employs a byte pair encoding (BPE) tokenizer, enhanced to split numbers into single digits, thereby improving numerical precision and supporting exact arithmetic decoding. The system encodes multimodal financial data as follows:

• Textual and Numerical Inputs: Arbitrary-length sequences (up to 8,000 tokens) are BPE-tokenized, with digits maintained as atomic tokens.
• Tabular Data: Tables are serialized into “|”-delimited textual sequences and interleaved within the text context.
• Images (Charts and Visual Data): Images are processed by a pretrained CLIP vision encoder to yield 768-dimensional embeddings. A two-layer MLP (visual abstractor) projects these embeddings into the transformer’s token embedding space, with <image> prompt tokens replaced by these projected embeddings.
• Fusion Strategy: Modalities—text, number, table, image—share the same embedding and positional encoding layers, enabling unified sequence processing by the transformer.

The integration workflow is illustrated in the following high-level pseudocode from the primary source:

txt_emb = text_embedding(context_tokens) tbl_seq = serialize_table(tables) # e.g. ["Date","Value",…] tbl_emb = text_embedding(tbl_seq) img_feat = clip_encoder(images) # shape (B,768) img_emb = mlp_visual(img_feat) # shape (B, d_model) seq_emb = concat([txt_emb, tbl_emb, <image>=img_emb]) out = transformer(seq_emb) return language_model_head(out)

(Bhatia et al., 16 Feb 2024)

2. Domain-Specific Pretraining

FinTral leverages a finance-focused pretraining corpus (“FinSet”) comprising roughly 20 billion deduplicated tokens. Sources include domain-filtered C4, large-scale financial news (e.g., Yahoo, Seeking Alpha, Eastmoney, Yicai), SEC EDGAR filings from 1993–2023, and data scraped from company websites and financial social media (notably, Reddit r/WallStreetBets). This design enables broad coverage of diverse textual and visual financial scenarios.

Pretraining employs standard next-token prediction with cross-entropy loss, operationalized as:

$$\mathcal{L}_{\mathrm{pretrain}} = - \sum_{t=1}^T \log p_\theta(x_t \mid x_{<t}) = -\mathbb{E}_{x \sim \mathrm{FinSet}} \left[\log p_\theta(x)\right]$$

In practice, LoRA-based fine-tuning for one epoch (learning rate $2.5\times10^{-5}$) is conducted on four A100 GPUs over 80 hours.

3. Instruction Fine-Tuning and RLAIF Alignment

Instruction fine-tuning is performed using approximately 226,000 high-quality finance-focused prompt–response pairs derived from sources such as FLUPE, Finance-Alpaca, FinRed, and Math-Instruct. QLoRA (quantized LoRA) is applied to all linear layers, and prompts are standardized to employ a memetic “financial expert” persona, stepwise reasoning, and explicit constraint enforcement.

For RLAIF, AI feedback is generated by sampling “chosen” responses from GPT-4 and “rejected” responses from base models (FinMA-7B or LLaMa-7B). Approximately 43,000 preference triples $\{\,\mathrm{prompt}, a^+, a^-\}$ are collected. Rather than employing a learned reward model, FinTral optimizes directly using Direct Preference Optimization (DPO) with distilled LoRA (dDPO):

$$\max_\theta\,\mathbb{E}_{a^+,a^-}\left[\log\sigma\left(s_\theta(a^+)-s_\theta(a^-)\right)\right] -\lambda D_{KL}\left(\pi_\theta\|\pi_{\mathrm{ref}}\right)$$

Here, $s_\theta(a)$ is the unnormalized model score for answer $a$, $\sigma$ denotes the sigmoid activation, and $\pi_{\mathrm{ref}}$ is the reference policy.

4. Tools, Retrieval-Augmented Generation, and Direct Preference Optimization

FinTral-DPO-T&R extends the core model with two mechanisms:

• Tool Integration: The model can produce function calls (e.g., Add(a, b), Subtract(a, b)). These are subsequently executed by a Python-based post-processor, which replaces function calls with computed numeric results, thereby reducing arithmetic errors in domains such as credit scoring or financial aggregation.
• Retrieval-Augmented Generation (RAG): Using a BGE embedding index over 30,000 financial documents (Jan 2022–Sep 2023), the model performs “chain of retrieval,” extracting relevant contextual passages that are dynamically fed to the LLM.

The optimization employs the DPO loss:

$$\mathcal{L}_{\mathrm{DPO}} = -\mathbb{E}_{(a^+,a^-)}\left[\log \sigma\left(\beta\cdot(s_\theta(a^+)-s_\theta(a^-))\right)\right]$$

with a temperature parameter $\beta$.

5. Evaluation Framework and Benchmark Performance

Evaluation is carried out on the FinSet benchmark, which comprises nine task categories and 25 datasets. These tasks include chart understanding, sentiment analysis, named entity recognition, numerical reasoning over tables, text summarization, stock movement prediction, credit scoring, firm disclosure, and hallucination analysis. Representative datasets include ChartQA, FinVQAv1/v2, FiQA-SA, FOMC, FiNER, FinQA, ConvFinQA, ECTSUM, ACL18, Australian, CS, and FinanceBench.

The following table summarizes key zero-shot performance metrics (higher is better):

Model	Type	Aggregate Score
ChatGPT-3.5	RL + tools	0.53
GPT-4	RL + tools	0.69
FinTral-INST	inst.-tuned	0.49
FinTral-DPO	RLAIF	0.59
FinTral-DPO-T&R	RLAIF+tool+RAG	0.70

On hallucination-specific metrics (FinTerms-MCQ accuracy, “Hallucination Index”):

Model	Hallucination Index
ChatGPT-3.5	0.95
GPT-4	0.98
FinTral-DPO-T&R	0.97

Human evaluation on FinTerms-Gen (n=128, agreed cases) indicates that FinTral-DPO-T&R produces more fully correct (A-rated) responses than ChatGPT, although GPT-4 yields the highest overall correctness. FinTral surpasses ChatGPT-3.5 on all tasks and exceeds GPT-4 on five out of nine (Bhatia et al., 16 Feb 2024).

6. Real-Time Analysis and Deployment Considerations

A sequence length of 8,000 tokens and a compact 7-billion parameter footprint provide the basis for low-latency inference and high throughput relative to substantially larger models (>100B parameters). Fast retrieval via the BGE index returns passages within tens of milliseconds, and tool execution introduces minimal computational overhead. FinTral’s LoRA adapters (∼100 MB) enable deployment on single-GPU servers, supporting real-time analysis of streaming content such as news feeds, SEC filings, and financial dashboards.

A plausible implication is that FinTral’s efficiency and multimodal support make it well-suited to resource-constrained settings requiring rapid, reliable financial intelligence extraction.

7. Significance and Outlook

FinTral demonstrates that a 7B-parameter multimodal LLM, configured with finance-domain pretraining, robust instruction tuning, and preference-based alignment, can rival or surpass proprietary models like ChatGPT-3.5 and compete with GPT-4 on demanding financial NLP, visual reasoning, arithmetic, and retrieval-augmented tasks. Its integration of vision, executable tool-calls, and dynamic retrieval into a unified inference framework establishes a versatile, cost-effective platform for a broad range of real-time financial analysis applications. The FinSet benchmark sets a new standard for evaluating future financial AI models (Bhatia et al., 16 Feb 2024).