FinLLMs: Financial Domain LLM Advances

• FinLLMs are domain-specialized language models that integrate textual, tabular, and multimodal data to optimize financial prediction, knowledge extraction, and compliance.
• They employ continual pre-training, supervised fine-tuning, and parameter-efficient methods like LoRA to enhance domain-specific performance on tasks such as sentiment analysis and risk assessment.
• Emerging challenges include improving numerical fidelity, regulatory reasoning, and computational efficiency while leveraging adaptive learning and tool integration.

Financial LLMs (FinLLMs) are domain-specialized downstream adaptations of large Transformer-based LLMs trained or retrofitted to process, reason over, and generate finance-related content. These systems are characterized by rich integration with textual, tabular, and sometimes multimodal data modalities and are optimized for financial prediction, knowledge extraction, risk assessment, regulatory compliance, and algorithmic trading. The contemporary proliferation of FinLLMs has been driven by architectural advances, the exponential growth in financial data, and the prioritization of regulatory robustness, numerical fidelity, and explainability.

1. Core Architectures and Domain Adaptation

FinLLMs typically originate from high-capacity open-source or proprietary LLMs such as LLaMA(3), GPT-4, BLOOM, Baichuan, or Qwen, with parameter counts ranging from 7B to upward of 70B (Lee et al., 2024, Noels et al., 2024, Huang et al., 2024, Sinha et al., 4 Feb 2025). Model adaptation for finance is accomplished through several complementary techniques:

• Continual pre-training: Exposing the LLM to billions of domain-specific tokens—SEC filings, earnings calls, news, analyst reports, macroeconomic texts—to augment financial vocabulary and priors (Lee et al., 2024, Wang et al., 2024, Wu et al., 2024).
• Supervised fine-tuning (SFT): Using task-specific labeled datasets for core tasks (e.g., sentiment, QA, NER, SMP), via standard cross-entropy loss:

$$\mathcal{L}_{\mathrm{CE}}(\theta) = -\sum_i y_i \log p_\theta(y_i|x_i)$$

• Parameter-Efficient Fine-Tuning (PEFT): Adoption of techniques such as LoRA (low-rank adapters), QLoRA (4–8 bit quantized LoRA), prefix-tuning, and BitFit to enable compact adaptation and on-premise deployment by updating a small fraction of parameters (Huang et al., 2024, Wang et al., 2024, Yang et al., 2023, Liu et al., 2023).

Recent multimodal FinLLMs such as Open-FinLLMs add dedicated vision encoders (e.g., CLIP), MLP projectors, and cross-modal fusion via prepending visual embeddings as tokens, supporting direct chart, table, and time-series processing (Huang et al., 2024, Wang et al., 2024).

2. Data Curation, Pre-training Corpora, and Instruction Sets

Data scale and curation pipelines are critical to FinLLM effectiveness. Data sources encompass:

• Textual: Financial news, filings (SEC, EDGAR, A-Share, XBRL), conference calls, research reports, legal texts, and social commentary (Lee et al., 2024, Sinha et al., 4 Feb 2025, Yang et al., 2023, Wu et al., 2024).
• Tabular & time-series: OHLCV data, macroeconomic indicators, and derived features (e.g., moving averages, ratios) (Huang et al., 2024, Sinha et al., 4 Feb 2025, Wang et al., 2024).
• Multimodal: Images (charts), tables (HTML, PNG), time-series as flattened sequences or embedded visual representations (Huang et al., 2024, Wang et al., 2024).

Instruction-tuning datasets are constructed via a combination of translated domain instructions (e.g., Dutch: 140K samples (Noels et al., 2024)), synthetic augmentation (LLM-generated QAs (Yuan et al., 2024)), and retrieval-augmented templates (e.g., FinBloom 50K queries with context (Sinha et al., 4 Feb 2025)). Corpora sizes in state-of-the-art models exceed 50B financial tokens, with instruction datasets ranging from tens of thousands to over half a million examples.

3. Core Financial Tasks and Benchmarking

Systematic benchmarking of FinLLMs involves a taxonomy of tasks:

• Sentiment Analysis (SA): Market/microblog/news polarity extraction; F1/accuracy metrics (Lee et al., 2024, Wu et al., 2024).
• Text Classification and NER: News, headlines, entity extraction (F1, accuracy) (Wang et al., 2024, Wu et al., 2024).
• Question Answering (QA): Financial QA, numerical/factual reasoning (EM, RMSE, Regex Match) (Lee et al., 2024, Wu et al., 2024).
• Stock Movement Prediction (SMP): News-based up/down, using accuracy and Sharpe ratio (Lee et al., 2024, Wu et al., 2024, Tsai, 10 Sep 2025).
• Summarization: Abstractive generation from earnings calls/reports (ROUGE, BLEU) (Wu et al., 2024, Lee et al., 2024).
• Table/Chart/Multimodal Reasoning: Value retrieval, comparison, time-series linking (Huang et al., 2024).
• Financial Reasoning (MCQ, Regulatory): Professional exam QA (CFA, CPA, regulation), certificate-level benchmarks (Wang et al., 2024, Wu et al., 2024, Yang et al., 2024).

Notable bilingual and regulatory benchmarks include Golden Touchstone (Wu et al., 2024), COLING 2025 Regulations Challenge (Wang et al., 2024), and IDEA-FinBench (Yang et al., 2024).

Performance metrics span accuracy, weighted F1, MCC, EM, ROUGE, BLEU, Sharpe ratio, RMSE, and specialized factuality metrics (FActScore for regulatory QA). Leading FinLLMs surpass general LLMs by 10–30% absolute in domain tasks, with hybrid and instruction-tuned variants (e.g., FinLLaMA-Instruct, Touchstone-GPT, SNFinLLM-chat) yielding consistent top-3 rankings across tasks.

4. Learning and Optimization Strategies

FinLLMs deploy diverse learning paradigms for robust adaptation:

• Supervised Fine-Tuning: Cross-entropy minimization on curated financial QA, SA, NER, and MC datasets (Lee et al., 2024, Noels et al., 2024, Wu et al., 2024).
• Direct Preference Optimization (DPO): Preference-based alignment optimizing

$$\mathcal{L}_{\mathrm{DPO}}(\theta) = \mathbb{E}_{(x, y^+, y^-)\sim\mathcal{D}}\left[\log \sigma\left(r_\theta(x,y^+)-r_\theta(x,y^-)\right)\right]$$

conferring advantages for multi-candidate QA and hallucination reduction (Zhao et al., 2024).

• RL from Human Feedback (RLHF)/Market Feedback: PPO/policy gradients using explicit reward models or market reaction (e.g., RLSP in FinGPT) (Liu et al., 2023, Yang et al., 2023).
• Parameter-Efficient Methods: LoRA/QLoRA (rank-4 to 64, 4–8bit quantized), prefix-tuning, enabling fine-tuning on commodity GPUs with order-of-magnitude lower compute and adapter checkpoints of 5–20MB (Wang et al., 2024, Huang et al., 2024).
• Multimodal Fusion: Addition (StockTime), MLP projection (FinLLaVA), visual tokens prepended to text (FinLLaVA, Open-FinLLMs) (Huang et al., 2024, Wang et al., 2024).
• Tool Integration: Calculator/plugin triggering for accurate computational tasks (SNFinLLM-cal, DISC-FinLLM) (Zhao et al., 2024, Chen et al., 2023); retrieval modules for evidence-grounded outputs (FinBloom, RAG augmentations) (Sinha et al., 4 Feb 2025).

Scaling law studies empirically confirm task loss follows a power law in dataset size ($\mathrm{Loss}(N) = aN^{-b} + c$), with diminishing returns at extreme scale [(Rao et al., 17 Apr 2025), abstract only].

5. Advanced Features: Regulatory, Multimodal, and Numeric Sensitivity

Robust FinLLMs increasingly address complex, high-stakes financial reasoning:

• Regulatory/Compliance Reasoning: Specialized models and benchmarks (COLING 2025, FinReg Challenge (Wang et al., 2024); REG and XBRL QA tasks; abbreviation retrieval, statutory definition extraction) reveal that advanced instruction tuning, chain-of-thought (CoT), and retrieval augmentation improve professional-level QA but critical gaps persist in NER, exact link/tag retrieval, and short-domain string recall.
• Numeric Sensitivity: Models like NumLLM and SNFinLLM-cal employ dual LoRA adaptation (financial CP + numeric choice tuning), SVD-based adapter fusion, and tool calling for numerically accurate multiple-choice QA, outperforming baselines in numeric tasks by up to 2–3 pp (Su et al., 2024, Zhao et al., 2024).
• Multimodal Reasoning: Open-FinLLMs (FinLLaVA) and StockTime enable zero/few-shot table, chart, and image understanding using CLIP encoders and simple fusion, achieving top accuracy on TableBench and ChartBench (Huang et al., 2024, Wang et al., 2024).
• Dataset Generation: Synthetic QA datasets generated via graph-augmented formula enumeration (FinLLMs framework) demonstrably enhance model accuracy on numerical programmatic QA tasks beyond human-labeled baselines (Yuan et al., 2024).

6. Deployment, Efficiency, Privacy, and Limitations

Resource constraints and regulatory considerations are addressed by:

• Quantization and Adapter-driven finetuning: QLoRA allows 7B/8B models to train and run on 24–48GB GPUs, with memory reductions of 30–70% and batch sizes suitable for local/secure environments (Wang et al., 2024, Yang et al., 2023).
• Pipeline Parallelism and DDP: Layer-wise sharding for 70B models, low-rank adapter synchronization with 0/1-Adam for reduced communication cost (Wang et al., 2024).
• Confidentiality and On-Prem: Adapter-only deployment with no exposure of pretrained weights and local training for institutional data privacy (Wang et al., 2024).
• Trade-offs: Four-bit, low-rank adapters yield ~2–5% accuracy gap on some tasks versus full-precision; computation and inference latency for ultra-long inputs remains an open optimization target (Wang et al., 2024, Huang et al., 2024).

Persistent challenges include hallucination risk, sensitivity to low-quality or off-domain data, numeric reasoning error modes, and limitations in processing multimodal signals or extracting highly structured knowledge (e.g., XBRL, legal codes). Bilingual Fairness and cross-lingual generalizability are active areas of study (Wu et al., 2024, Noels et al., 2024).

7. Outlook and Emerging Directions

Frontiers for FinLLMs involve:

• Adaptive online/continual learning with domain feedback loops (Mahdavi et al., 29 Jun 2025).
• Domain-specific model architectures with explicit multimodal and temporal modules (Mahdavi et al., 29 Jun 2025, Wang et al., 2024).
• Multi-agent and agentic finance: Planner-executor structures, role simulation (e.g., analyst/manager, BDI-style market actors) (Mahdavi et al., 29 Jun 2025).
• Standardization of benchmarks and agent-based pipelines (Wang et al., 2024, Wu et al., 2024).
• Human-AI collaboration interfaces to enable explainability, real-time risk management, and compliance (Mahdavi et al., 29 Jun 2025).
• Open-source, reproducible model and data releases, with modular instruction/construction pipelines ready for under-resourced languages (FinGEITje, Dutch (Noels et al., 2024)).
• Multimodal, retrieval-augmented architectures combining tabular, image, textual, and real-time feeds (Sinha et al., 4 Feb 2025, Huang et al., 2024).

FinLLMs sit at a convergence of high-parameter language modeling, financial engineering, regulatory logic, and multimodal data fusion, with ongoing innovation focused on robustness, resource efficiency, and regulatory alignment suitable for high-stakes financial workflows.