DPFinLLM: Private FinTech LLM Fine-Tuning

• DPFinLLM is a privacy-preserving financial LLM fine-tuning framework that integrates lightweight transformers with formal differential privacy guarantees.
• It uses LoRA-based parameter-efficient adaptation combined with DP-SGD to limit training data influence and prevent sensitive data leakage.
• Experiments on financial sentiment datasets show that DPFinLLM achieves competitive accuracy while balancing privacy and utility.

DPFinLLM is a privacy-preserving financial LLM fine-tuning framework for on-device and edge FinTech applications. It combines a lightweight transformer architecture inspired by Llama2, LoRA-based parameter-efficient adaptation, and an $(\epsilon,\delta)$-differential privacy mechanism during fine-tuning, with experiments centered on financial sentiment analysis. In the paper’s formulation, the central problem is that financial fine-tuning data can be sensitive, while standard adaptation may allow memorization or leakage of individual training examples; DPFinLLM addresses this by applying differentially private optimization during financial adaptation rather than treating privacy as a purely deployment-side property (Zhu et al., 10 Sep 2025).

1. Concept and problem setting

DPFinLLM is defined as an on-device differential privacy-enhanced financial LLM. The framework is motivated by two concurrent trends: the growing use of LLMs for finance and the movement of AI systems toward local or edge deployment for real-time, privacy-sensitive applications. The paper explicitly places the method in settings where financial text used for fine-tuning may contain sensitive information, and where the resulting model should be less vulnerable to training-data inference than a conventionally fine-tuned model (Zhu et al., 10 Sep 2025).

The paper’s scope is narrower than the term “financial LLM” might suggest. Although the motivation refers to broader financial uses such as sentiment analysis, risk management, fraud detection, and credit scoring, the experimental target is primarily financial sentiment analysis. This is important for interpreting the framework: DPFinLLM is presented as a private financial adaptation method, not as a complete finance foundation model spanning retrieval, reasoning, forecasting, compliance, or multimodal document understanding.

A central distinction in the paper is between privacy as a formal training guarantee and privacy as a deployment preference. DPFinLLM adopts the former. Its stated goal is to limit how much any single training sample can influence the learned model, so that the model trained with one example included is statistically close to the model trained without it. This positions the method within formal differential privacy rather than purely operational data locality.

2. Architecture and financial adaptation pathway

The model architecture is transformer-based, with modifications inspired by Llama2 and related efficient design choices. The paper highlights grouped-query attention, RMSNorm, SwiGLU, and rotary positional embeddings as the main lightweight architectural elements. Grouped-query attention is described as sharing key and value projection matrices across multiple heads, reducing memory cost while maintaining performance. In experimental instantiations, the framework is built on Llama2-7B and ChatGLM2-6B (Zhu et al., 10 Sep 2025).

The language-modeling formulation is standard autoregressive generation: $$\pi_{\theta} (\mathbf{y} \mid \mathbf{x}) = \prod_{t=1}^{T} \pi\left(y_t \mid x_1, x_2, \ldots, x_n, y_1, y_2, \ldots, y_{t-1}\right),$$ where $\mathbf{x}$ is the tokenized input prompt and $\mathbf{y}$ is the generated output sequence. The paper also states the standard attention and multi-head attention equations: $$\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V,$$

$$\mathrm{head}_i = \mathrm{Attention}(QW_i^Q, KW_i^K, VW_i^V),$$

$$\mathrm{MultiHead}(Q,K,V) = \mathrm{Concat}(\mathrm{head}_1,\ldots,\mathrm{head}_h)W^O.$$

Financial adaptation is performed through supervised instruction tuning for sentiment classification. The instruction template used in the paper is: “What is the sentiment of this news/tweet? Please choose an answer from {negative/neutral/positive}.” The training example format is therefore an input financial text paired with a sentiment label. This makes the financial specialization narrow but explicit: the framework is evaluated as a private adaptation layer over general LLM backbones, not as a finance-pretrained model.

LoRA is the chosen adaptation mechanism. For a pretrained weight matrix $W_0 \in \mathbb{R}^{d \times k}$, the update is parameterized as

$W_0 + \Delta W = W_0 + BA,$

with $B \in \mathbb{R}^{d \times r}$, $$\pi_{\theta} (\mathbf{y} \mid \mathbf{x}) = \prod_{t=1}^{T} \pi\left(y_t \mid x_1, x_2, \ldots, x_n, y_1, y_2, \ldots, y_{t-1}\right),$$0, and $$\pi_{\theta} (\mathbf{y} \mid \mathbf{x}) = \prod_{t=1}^{T} \pi\left(y_t \mid x_1, x_2, \ldots, x_n, y_1, y_2, \ldots, y_{t-1}\right),$$1. The forward pass becomes

$$\pi_{\theta} (\mathbf{y} \mid \mathbf{x}) = \prod_{t=1}^{T} \pi\left(y_t \mid x_1, x_2, \ldots, x_n, y_1, y_2, \ldots, y_{t-1}\right),$$2

The paper notes that $$\pi_{\theta} (\mathbf{y} \mid \mathbf{x}) = \prod_{t=1}^{T} \pi\left(y_t \mid x_1, x_2, \ldots, x_n, y_1, y_2, \ldots, y_{t-1}\right),$$3 is scaled by $$\pi_{\theta} (\mathbf{y} \mid \mathbf{x}) = \prod_{t=1}^{T} \pi\left(y_t \mid x_1, x_2, \ldots, x_n, y_1, y_2, \ldots, y_{t-1}\right),$$4, although it does not give the full scaled equation in the main derivation. In effect, DPFinLLM adapts a frozen or largely frozen backbone through low-rank updates while privatizing the fine-tuning phase.

3. Differential privacy mechanism

The privacy mechanism is a DP-SGD-style procedure applied during fine-tuning (Zhu et al., 10 Sep 2025). For a training set $$\pi_{\theta} (\mathbf{y} \mid \mathbf{x}) = \prod_{t=1}^{T} \pi\left(y_t \mid x_1, x_2, \ldots, x_n, y_1, y_2, \ldots, y_{t-1}\right),$$5, the empirical loss is

$$\pi_{\theta} (\mathbf{y} \mid \mathbf{x}) = \prod_{t=1}^{T} \pi\left(y_t \mid x_1, x_2, \ldots, x_n, y_1, y_2, \ldots, y_{t-1}\right),$$6

At each step $$\pi_{\theta} (\mathbf{y} \mid \mathbf{x}) = \prod_{t=1}^{T} \pi\left(y_t \mid x_1, x_2, \ldots, x_n, y_1, y_2, \ldots, y_{t-1}\right),$$7, for a sample $$\pi_{\theta} (\mathbf{y} \mid \mathbf{x}) = \prod_{t=1}^{T} \pi\left(y_t \mid x_1, x_2, \ldots, x_n, y_1, y_2, \ldots, y_{t-1}\right),$$8, the per-sample gradient is

$$\pi_{\theta} (\mathbf{y} \mid \mathbf{x}) = \prod_{t=1}^{T} \pi\left(y_t \mid x_1, x_2, \ldots, x_n, y_1, y_2, \ldots, y_{t-1}\right),$$9

Each gradient is clipped to a norm bound $\mathbf{x}$0: $\mathbf{x}$1 The clipped gradients are then aggregated and perturbed with Gaussian noise. The intended noisy gradient expression is

$\mathbf{x}$2

where $\mathbf{x}$3 is the lot size and $\mathbf{x}$4 is the noise multiplier. Parameters are updated by

$\mathbf{x}$5

The privacy guarantee is stated using standard approximate differential privacy. A randomized mechanism $\mathbf{x}$6 satisfies $\mathbf{x}$7-differential privacy if for adjacent datasets $\mathbf{x}$8 and any measurable subset $\mathbf{x}$9,

$\mathbf{y}$0

The paper also gives a DP-SGD-style theorem: for sampling rate $\mathbf{y}$1, number of steps $\mathbf{y}$2, and constants $\mathbf{y}$3, the algorithm satisfies $\mathbf{y}$4-DP for suitable $\mathbf{y}$5 provided

$\mathbf{y}$6

This expresses the usual tradeoff: stronger privacy requires more noise, while larger sampling rates and more optimization steps worsen privacy.

The implementation is mathematically clear but operationally incomplete. The paper specifies the privacy notation $\mathbf{y}$7, $\mathbf{y}$8, $\mathbf{y}$9, $$\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V,$$0, $$\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V,$$1, $$\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V,$$2, and $$\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V,$$3, and states that some experiments use

$$\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V,$$4

but it does not report a full privacy accountant, exact noise multipliers, clipping norms, or the detailed optimizer configuration. As a result, DPFinLLM is reproducible at the algorithmic level more readily than at the engineering level.

4. Datasets, metrics, and empirical results

The evaluation uses four financial sentiment datasets: Financial PhraseBank (FPB, 4,846 news entries), FIQA (1,213 entries from financial headlines and microblogs), Twitter Financial News Sentiment (TFNS, 11,931 tweets), and NWGI (20,231 entries). The compared baselines are Llama2-7B, ChatGLM2-6B, FinGPT-llama2-mt, FinGPT v3.2, and FinGPT v3.1. Metrics are accuracy, F1 macro, F1 micro, and F1 weighted (Zhu et al., 10 Sep 2025).

Dataset	Size	Reported characteristics
FPB	4,846	Financial PhraseBank news entries
FIQA	1,213	Financial headlines and microblogs
TFNS	11,931	Twitter Financial News Sentiment tweets
NWGI	20,231	GPT-instruction-generated dataset

On FPB, the Llama2-based DPFinLLM with $$\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V,$$5 reaches accuracy $$\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V,$$6, up from $$\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V,$$7 for the Llama2-7B base model and equal to FinGPT-llama2-mt, though still below FinGPT v3.2 at $$\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V,$$8. On the same dataset, the ChatGLM2-based DPFinLLM with $$\mathrm{Attention}(Q,K,V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V,$$9 reaches $$\mathrm{head}_i = \mathrm{Attention}(QW_i^Q, KW_i^K, VW_i^V),$$0, only marginally above the ChatGLM2-6B base model at $$\mathrm{head}_i = \mathrm{Attention}(QW_i^Q, KW_i^K, VW_i^V),$$1. The paper therefore supports a strong result for the Llama2 variant on FPB, but not a uniformly strong result across backbones.

On FIQA, the Llama2-based DPFinLLM with $$\mathrm{head}_i = \mathrm{Attention}(QW_i^Q, KW_i^K, VW_i^V),$$2 reaches $$\mathrm{head}_i = \mathrm{Attention}(QW_i^Q, KW_i^K, VW_i^V),$$3 accuracy, compared with $$\mathrm{head}_i = \mathrm{Attention}(QW_i^Q, KW_i^K, VW_i^V),$$4 for the base Llama2-7B and $$\mathrm{head}_i = \mathrm{Attention}(QW_i^Q, KW_i^K, VW_i^V),$$5 for FinGPT v3.2. The ChatGLM2-based DPFinLLM with $$\mathrm{head}_i = \mathrm{Attention}(QW_i^Q, KW_i^K, VW_i^V),$$6 reaches $$\mathrm{head}_i = \mathrm{Attention}(QW_i^Q, KW_i^K, VW_i^V),$$7, matching the ChatGLM2-6B base model and remaining close to FinGPT v3.1 at $$\mathrm{head}_i = \mathrm{Attention}(QW_i^Q, KW_i^K, VW_i^V),$$8. FIQA is therefore one of the paper’s strongest demonstrations that private fine-tuning can remain competitive at relatively strict privacy levels.

On TFNS, the Llama2-based DPFinLLM with $$\mathrm{head}_i = \mathrm{Attention}(QW_i^Q, KW_i^K, VW_i^V),$$9 reaches $$\mathrm{MultiHead}(Q,K,V) = \mathrm{Concat}(\mathrm{head}_1,\ldots,\mathrm{head}_h)W^O.$$0 accuracy, well above the Llama2-7B base model at $$\mathrm{MultiHead}(Q,K,V) = \mathrm{Concat}(\mathrm{head}_1,\ldots,\mathrm{head}_h)W^O.$$1, but below FinGPT-llama2-mt at $$\mathrm{MultiHead}(Q,K,V) = \mathrm{Concat}(\mathrm{head}_1,\ldots,\mathrm{head}_h)W^O.$$2 and FinGPT v3.2 at $$\mathrm{MultiHead}(Q,K,V) = \mathrm{Concat}(\mathrm{head}_1,\ldots,\mathrm{head}_h)W^O.$$3. The ChatGLM2-based DPFinLLM with $$\mathrm{MultiHead}(Q,K,V) = \mathrm{Concat}(\mathrm{head}_1,\ldots,\mathrm{head}_h)W^O.$$4 reaches $$\mathrm{MultiHead}(Q,K,V) = \mathrm{Concat}(\mathrm{head}_1,\ldots,\mathrm{head}_h)W^O.$$5, again far above its base model at $$\mathrm{MultiHead}(Q,K,V) = \mathrm{Concat}(\mathrm{head}_1,\ldots,\mathrm{head}_h)W^O.$$6 but below FinGPT v3.1 at $$\mathrm{MultiHead}(Q,K,V) = \mathrm{Concat}(\mathrm{head}_1,\ldots,\mathrm{head}_h)W^O.$$7. On NWGI, gains are modest: the Llama2-based DPFinLLM with $$\mathrm{MultiHead}(Q,K,V) = \mathrm{Concat}(\mathrm{head}_1,\ldots,\mathrm{head}_h)W^O.$$8 reaches $$\mathrm{MultiHead}(Q,K,V) = \mathrm{Concat}(\mathrm{head}_1,\ldots,\mathrm{head}_h)W^O.$$9 versus $W_0 \in \mathbb{R}^{d \times k}$0 for the base model, while the ChatGLM2-based variant with $W_0 \in \mathbb{R}^{d \times k}$1 is effectively unchanged at $W_0 \in \mathbb{R}^{d \times k}$2 versus $W_0 \in \mathbb{R}^{d \times k}$3 for the base.

The paper also reports that, for FPB, TFNS, and FIQA, F1 scores increase at first as $W_0 \in \mathbb{R}^{d \times k}$4 increases, then peak and decline. This indicates that the privacy–utility curve is non-monotonic in the reported experiments. The paper does not print all figure values, but its interpretation is that privacy tuning is not a simple “less privacy always yields better performance” tradeoff.

Zero-shot transfer is evaluated by fine-tuning on one dataset and testing on another. Representative weighted-F1 results include Llama2-based DPFinLLM fine-tuned on TFNS and tested on FPB at $W_0 \in \mathbb{R}^{d \times k}$5 versus $W_0 \in \mathbb{R}^{d \times k}$6 for the base, and fine-tuned on FPB and tested on TFNS at $W_0 \in \mathbb{R}^{d \times k}$7 versus $W_0 \in \mathbb{R}^{d \times k}$8 for the base. A counterexample appears in the ChatGLM2 family: fine-tuning on TFNS and testing on FIQA yields $W_0 \in \mathbb{R}^{d \times k}$9, a sharp drop from the base model’s $W_0 + \Delta W = W_0 + BA,$0. The paper uses these results to argue that private fine-tuning does not necessarily destroy zero-shot ability, but the transfer behavior is clearly dataset- and backbone-dependent.

5. Position within the financial LLM literature

DPFinLLM occupies a specific niche within financial LLM research: it addresses formal privacy during fine-tuning, while much of the adjacent literature emphasizes efficiency, domain adaptation, reasoning, or trustworthy tool use rather than differential privacy. “FinLoRA: Finetuning Quantized Financial LLMs Using Low-Rank Adaptation” (Wang et al., 2024) focuses on local deployment, quantized low-rank adaptation, and accessible GPU training, but explicitly does not provide formal differential privacy guarantees. “Demystifying Domain-adaptive Post-training for Financial LLMs” (Ke et al., 9 Jan 2025) studies capability-driven finance post-training through continual pre-training, instruction tuning, and preference alignment, but not private optimization. “Fin-PRM: A Domain-Specialized Process Reward Model for Financial Reasoning in LLMs” (Zhou et al., 21 Aug 2025) targets process-level reasoning supervision, and “FinAI Data Assistant: LLM-based Financial Database Query Processing with the OpenAI Function Calling API” (Kim et al., 15 Oct 2025) addresses reliable structured database access rather than private model adaptation.

This comparison clarifies what DPFinLLM does and does not solve. It solves the problem of making financial fine-tuning formally private under a DP-SGD-style mechanism. It does not, by itself, provide the broader domain-adaptive recipe studied in FINDAP, the reasoning-verification layer introduced by Fin-PRM, or the constrained retrieval-and-execution architecture proposed by FinAI Data Assistant. This suggests that DPFinLLM is best understood as a training-time privacy layer within a larger financial LLM stack rather than as a complete architecture for trustworthy financial intelligence.

The broader literature also indicates plausible extensions. FinLoRA argues that adapter-only training on quantized frozen backbones can make local financial adaptation tractable on accessible hardware (Wang et al., 2024). FINDAP shows that mixed general-plus-finance post-training and reasoning-specific preference alignment improve broad financial competence (Ke et al., 9 Jan 2025). Fin-PRM shows that domain-specialized reward models can improve financial reasoning traces during supervised learning, reinforcement learning, and Best-of-$W_0 + \Delta W = W_0 + BA,$1 inference (Zhou et al., 21 Aug 2025). These results do not appear in DPFinLLM itself, but they define adjacent design space around efficiency, reasoning quality, and deployment reliability.

6. Limitations and open directions

DPFinLLM has several explicit and implied limitations (Zhu et al., 10 Sep 2025). First, the paper omits many reproducibility-critical hyperparameters: LoRA rank, LoRA scaling, LoRA insertion points, learning rate, optimizer choice, number of epochs, lot size, clipping norm $W_0 + \Delta W = W_0 + BA,$2, and noise multiplier $W_0 + \Delta W = W_0 + BA,$3. Second, it does not provide a controlled comparison between identical non-private and private fine-tuning pipelines, which makes it difficult to isolate the exact utility cost of privacy. Third, although the framework is motivated as an on-device method, it does not report model footprint, memory usage, latency, power consumption, or deployment on an actual edge platform.

The privacy analysis is formal but high-level. The paper provides standard DP-SGD equations and a theorem relating $W_0 + \Delta W = W_0 + BA,$4, $W_0 + \Delta W = W_0 + BA,$5, $W_0 + \Delta W = W_0 + BA,$6, $W_0 + \Delta W = W_0 + BA,$7, and $W_0 + \Delta W = W_0 + BA,$8, yet it does not specify a full accountant or step-by-step privacy accumulation report. It also does not address other failure modes central to financial deployment, including hallucination, robustness, prompt injection, model extraction at inference time, or exact temporal data retrieval.

The empirical scope is also narrow. All reported experiments are on financial sentiment datasets, and the strongest results are concentrated in FPB and FIQA, particularly for the Llama2-based variant. Performance on TFNS remains noticeably below stronger non-private financial baselines, and NWGI shows only small gains. The framework therefore demonstrates that differentially private financial fine-tuning is feasible, but not that it closes the broader gap to state-of-the-art domain-specialized systems across reasoning, retrieval, or document-heavy tasks.

A plausible implication is that future private financial LLMs will need to combine DPFinLLM’s private fine-tuning mechanism with the efficiency methods of local adapter training, the capability-aware post-training strategies developed for finance adaptation, and the retrieval or tool-use constraints required for exact financial question answering. In that sense, DPFinLLM establishes a formal privacy foundation for financial LLM fine-tuning, but not the full operational architecture of a production financial assistant.