FinFRE-RAG: Interpretable LLM Fraud Detection

• FinFRE-RAG is a two-stage framework that enhances LLM reasoning on structured financial data through feature reduction and retrieval-augmented in-context learning.
• It employs importance-guided feature reduction using Random Forests to select a compact set of features, thereby improving model interpretability and prediction accuracy.
• The methodology leverages hybrid retrieval strategies and natural language serialization to deliver transparent fraud detection with competitive F1 and MCC metrics.

FinFRE-RAG is a finetuning-free, two-stage framework for effectiveness-enhanced, interpretable LLM reasoning on high-dimensional structured financial data. Designed specifically to address the challenges of fraud detection in tabular transaction records, FinFRE-RAG combines importance-guided feature reduction with retrieval-augmented in-context learning, enabling open-domain LLMs to produce competitive, transparent predictions without training or parameter adaptation. The methodology integrates classic machine learning (for feature importance ranking) with modern retrieval-based context augmentation, operationalized entirely via natural language serialization and in-context examples, to bridge the gap between tabular data modeling and LLM-based analysis (Tan et al., 15 Dec 2025).

1. Foundational Principles and Motivation

Traditional tabular models for fraud detection, such as Random Forests and XGBoost, require significant manual feature engineering and yield models of limited interpretability, making downstream fraud analysis labor-intensive and less transparent. LLMs offer richer explanation capabilities but underperform on high-dimensional, imbalanced tabular fraud tasks, due to feature overload, lack of contextual exemplars, and the challenges of mapping structured data to a language interface. FinFRE-RAG addresses these deficits by: (i) extracting and standardizing a compact, informativeness-ranked set of input features, and (ii) retrieving instance-level, label-aware exemplars closely related to the query for in-context learning. This enables LLMs to leverage analogical reasoning while remaining robust to class imbalance and information redundancy.

2. Importance-Guided Feature Reduction

The first stage applies a Random Forest classifier on external data $\mathcal{D}_\mathrm{ext}$ solely to estimate feature importances. For each feature $f$ and each tree $t$, the total Gini impurity reduction $\Delta i_t(f)$ is aggregated. The normalized importance $I(f)$ is given by:

$$I(f) = \frac{1}{T} \sum_{t=1}^{T} \frac{\Delta i_{t}(f)}{\sum_{f'} \Delta i_{t}(f')}$$

Features are sorted by $I(f)$, and the top-$k$ features $\mathcal{F}_\mathrm{sel}$ are retained. Numeric features are standardized using their mean $\mu_f$ and standard deviation $\sigma_f$ computed over $\mathcal{D}_\mathrm{ext}$:

$z_f(x) = \frac{x[f] - \mu_f}{\sigma_f}$

This reduction phase is critical; ablation shows performance peaks at $k \approx 10$, and importance-guided selection outperforms random feature selection (e.g., $F_1$ of 0.31 vs. 0.26 and MCC of 0.36 vs. 0.30 on the ccf dataset) (Tan et al., 15 Dec 2025).

3. Serialization and Prompt Construction

Transactions are serialized into natural language using only the selected $k$ features in a consistent order. Two prompt formats are used, schema-grounded for anonymized fields, and descriptive for semantically interpretable datasets. For example, in the ccFraud dataset with $k=7$:

“The client is a Female, the state number is TX, the number of cards is 2, the credit balance is 45000, the number of transactions is 35, the number of international transactions is 3, the credit limit is 100000.”

This ensures that LLMs process well-calibrated, context-consistent textual representations.

4. Label-Aware Retrieval-Augmented In-Context Learning

At inference, FinFRE-RAG performs hybrid retrieval:

1. Categorical Filtering: Categorical features are filtered in descending order of importance. For a query $x_q$, only historical samples with matching values for the $j$ most important categorical features are retained, up to the largest $j^\star$ that yields a non-empty candidate set.
2. Numeric Similarity: Within the filtered set, each candidate $x$ is embedded as $\mathbf{z}(x)$ from standardized numeric features, and cosine similarity with $\mathbf{z}(x_q)$ is computed:

$$s(x, x_q) = \frac{\langle \mathbf{z}(x), \mathbf{z}(x_q)\rangle}{\|\mathbf{z}(x)\|_2 \|\mathbf{z}(x_q)\|_2}$$

The top-$n$ highest scoring exemplars are selected.

Prompting: A fixed prompt template is used:

System Instruction: You are a helpful financial expert… Provide Score: 1–5 and a brief explanation. Few-Shot Examples: Example 1: (features for %%%%22%%%%) It is a fraud. Example 2: (features for %%%%23%%%%) It is not a fraud. ... Query: Current case: (features for %%%%24%%%%)

The LLM outputs a score (1–5) and rationale. Transactions with $s \geq 4$ are classified as fraud.

5. Model and Inference Pipeline

FinFRE-RAG targets open-weight, instruction-tuned LLMs suited for on-premise deployment, including Qwen3-14B, Qwen3-Next-80B-A3B-Thinking, Gemma 3-12B/27B, GPT-OSS-20B/120B, with inference settings: temperature 0.6, nucleus 0.95, max length 16,384 tokens.

End-to-End Steps:

1. Input raw transaction $x_q$.
2. Extract top-$k$ features and standardize numerics.
3. Categorical filtering, numeric similarity search, select top-$n$ examples.
4. Prompt construction as described.
5. LLM inference to obtain risk score and rationale.
6. Map score to final binary classification and explanation.

Increasing $n$ (number of exemplars) to 20–30 saturates performance; $k \approx 10$ optimizes F1/MCC.

6. Experimental Evaluation and Results

FinFRE-RAG was evaluated on four public fraud datasets: ccf, ccFraud, IEEE-CIS, and PaySim, using both LLM-based and classical baselines. Metrics focus on F1 and Matthews Correlation Coefficient (MCC):

Model	ccf $(F_1,$ MCC$)$	ccFraud $(F_1,$ MCC$)$	IEEE-CIS $(F_1,$ MCC$)$	PaySim $(F_1,$ MCC$)$
Qwen3-14B (direct)	0.00, -0.01	0.14, 0.09	0.04, -0.01	0.00, -0.05
+ FinFRE-RAG	0.31, 0.36	0.48, 0.46	0.62, 0.60	0.11, 0.22
Gemma 3-12B (direct)	0.00, 0.00	0.13, 0.09	0.01, -0.03	0.00, 0.01
+ FinFRE-RAG	0.79, 0.80	0.59, 0.57	0.59, 0.57	0.71, 0.72
Random Forest	0.85, 0.85	0.52, 0.52	0.55, 0.54	0.79, 0.81
XGBoost	0.89, 0.89	0.48, 0.50	0.74, 0.73	0.68, 0.71
TabM	0.85, 0.85	0.66, 0.65	0.82, 0.82	0.92, 0.92

Directly applying LLMs yields $\mathrm{MCC} \approx 0$ (random guessing), while FinFRE-RAG improves MCC by 0.3–0.8, substantially narrowing the gap to specialized tabular models (Tan et al., 15 Dec 2025).

7. Interpretability, Analyst Support, and Conclusion

FinFRE-RAG yields transparent, instance-grounded rationales by presenting decisions alongside actual retrieved exemplars and highlighting which features drive risk assignment. The output includes a risk score (1–5) and a natural-language explanation referencing observed analogies to past fraud or legit cases. Decision rationales thus explicitly cite both the “what” (key features) and the “why” (similar past observations), providing direct analyst support and facilitating validation or override of algorithmic predictions.

In summary, FinFRE-RAG demonstrates that combining feature reduction, retrieval-augmented in-context learning, and explicit natural language prompts transforms otherwise black-box LLMs into effective and interpretable fraud detectors for structured financial data, with substantial downstream gains in F1, MCC, and decision transparency (Tan et al., 15 Dec 2025).