Fin-Vault: Diverse Financial Safety Benchmarks

Updated 14 July 2026

Fin-Vault serves as an execution-grounded security benchmark testing financial agents with 31 regulatory sandbox scenarios and 107 real-world vulnerabilities to assess compliance and risk.
It also denotes a multi-turn financial dialogue dataset curated from online forums, enabling commonsense-aware conversational AI across varied finance domains.
Additionally, Fin-Vault refers to a multi-biometric fuzzy vault that fuses face and fingerprint data to securely bind secrets, achieving near-perfect matching rates under rigorous cryptographic conditions.

Fin-Vault denotes several distinct technical artifacts in recent literature rather than a single unified framework. The dominant current usage is FinVault, an execution-grounded security benchmark for LLM-based financial agents, defined by regulatory case-driven sandbox scenarios, state-writable databases, explicit compliance constraints, and adversarial evaluation protocols (Yang et al., 9 Jan 2026). Separate literature uses Fin-Vault for a multi-turn financial dialogue dataset supporting commonsense-aware conversational AI (Das et al., 29 Sep 2025), and, in a different cryptographic lineage, for a multi-biometric fuzzy vault based on face and multiple fingerprints (Rathgeb et al., 2023). The shared label reflects a common emphasis on controlled access, constrained execution, or secure binding, but the underlying problem settings, threat models, and evaluation criteria are substantially different.

1. Terminological scope

The literature supports at least three technically distinct senses of the term.

Usage	Technical object	Source
FinVault	Execution-grounded security benchmark for financial agents	(Yang et al., 9 Jan 2026)
Fin-Vault	Multi-turn financial conversational dataset	(Das et al., 29 Sep 2025)
Fin-Vault	Multi-biometric fuzzy vault based on face and fingerprints	(Rathgeb et al., 2023)

A common source of confusion is the assumption that these usages belong to a single research program. They do not. The benchmark literature studies tool-using financial agents under adversarial, compliance-constrained execution; the conversational dataset literature studies financial dialogue generation and alignment; the biometric literature studies bio-cryptographic binding of secrets to noisy biometric sets. Related work on signature-based fuzzy vaults, DeFi lending vaults, and Bitcoin custody vaults further broadens the semantic field around “vault,” but these are separate technical domains (Eskander et al., 2014, Zbandut et al., 19 Apr 2026, Swambo et al., 2020).

2. FinVault as an execution-grounded benchmark for financial agent safety

FinVault was introduced as the first execution-grounded security benchmark specifically for financial agents. Its central claim is that safety evaluations limited to content moderation or abstract agent simulations fail to capture risks that arise when agents can plan, invoke tools, and directly manipulate state in regulated financial workflows (Yang et al., 9 Jan 2026).

The benchmark comprises 31 regulatory case-driven sandbox scenarios spanning six domains: Credit & Lending (7 scenarios), Insurance Services (4), Securities & Investment (5), Payment & Settlement (4), Compliance & Anti-Money Laundering (6), and Risk Management (5). Each scenario contains realistically modeled databases with mutable state, and agents act through defined toolchains that support querying and updating state while enforcing permission levels, quotas, and audit log mechanisms. Compliance requirements are encoded as hard environment constraints, such as prohibitions on transfers to sanctioned entities or requirements for identity verification, so benchmark failures correspond to invalid state transitions rather than merely unsafe text.

FinVault defines 107 real-world vulnerabilities, distributed across Compliance Violations (35%), Fraudulent Approvals (28%), Privilege Bypass (18%), Information Leakage (12%), and Audit Evasion (7%). The adversarial suite contains 856 attack samples plus 107 benign samples, for 963 test cases total. Attacks are organized into three classes with eight techniques: Prompt Injection via Direct JSON Injection and Instruction Overriding; Jailbreaking Attacks via Role Playing, Progressive Prompting, Encoding Obfuscation, and Hypothetical Scenarios; and Financially Adapted Attacks via Authority Impersonation and Emotional Manipulation.

Evaluation uses Attack Success Rate (ASR), Vulnerability Compromise Rate, and False Positive Rate (FPR). Reported results show substantial fragility in current models. Average ASR ranges from 50.0% for Qwen3-Max to 6.7% for Claude-Haiku-4.5, while vulnerability compromise rates range from 85.98% to 26.17% respectively. Scenario difficulty is heterogeneous: Insurance Service scenarios reach ASR values up to 65.2%, whereas Credit/Lending is lower because of more rigid business rule structures. Semantic attacks dominate technical ones: Role Playing reaches 64.5% ASR on Qwen3-Max, and models with stricter instruction-boundary enforcement resist instruction override much better, with ASR dropping from 64.5% on Qwen3-Max to 3.7% on Claude-Haiku-4.5. Defensive systems also remain inadequate: LLaMA Guard 4 attains TPR 61.1% but FPR 29.9%, while GPT-OSS-Safeguard reduces FPR to 12.15% at TPR 22.1%.

The main significance of FinVault is methodological. By combining executable environments, writable state, auditable tool use, and explicit compliance constraints, it shifts financial agent safety evaluation from linguistic refusal behavior to operational failure modes.

3. FinVault as an evaluation substrate for inline lifecycle safety

FinVault also functions as the principal evaluation environment for FinHarness, an inline safety harness designed for finance LLM agents (Jia et al., 26 May 2026). The motivating observation is that boundary filters may miss irreversible mid-trajectory tool calls, while post-hoc judges intervene only after a trace terminates.

FinHarness wraps an agent with three components. The Query Monitor fuses single-turn intent and cross-turn drift. The Tool Monitor scores each proposed tool action using permissions, parameters, and sequence context. The Cascade module accumulates risk across a window of recent steps, adaptively routes verification between a lightweight and an advanced-tier LLM judge, and supports bounded context via selective episodic memory, recalling at most 2 most salient prior steps. Fired risk factors from the monitors are re-injected into the agent prompt as ex-ante evidence, enabling refusal, re-planning, escalation, or approval before execution.

On FinVault, the routed configuration reduces ASR from 38.3% to 15.0% while largely preserving benign approval, from 41.1% to 39.3%. An always-advanced ablation lowers ASR further to 8.4% but with a benign approval rate of 37.4%. Routed FinHarness uses the advanced judge for only 20.1% of judge invocations, corresponding to 138/685 calls and a 4.7× reduction relative to the always-advanced setting with 646 calls for the same traces. Additional reported effects include agent self-rejection up by 15.7pp on synth attacks and total containment improved by +4.0pp relative to strong baselines.

This line of work is important because it operationalizes FinVault not only as a static benchmark but as a closed-loop intervention testbed for pre-execution financial agent safety.

4. Fin-Vault as a multi-turn financial dialogue dataset

A different usage appears in the Fin-Ally literature, where Fin-Vault is a multi-turn financial conversational dataset created to address the lack of domain-specific, advisory-oriented conversational corpora (Das et al., 29 Sep 2025). The dataset contains 1,417 annotated multi-turn dialogues and 4,006 utterances, with average queries per conversation of approximately 2.35. User turns average 10.8 tokens and 1.07 sentences, while bot turns average 41.18 tokens and 2.93 sentences. The reported vocabulary size is 3,398, with 17,477 unique bigrams in queries and 139,204 unique trigrams.

The data were curated from Reddit communities including r/personalfinance and r/financialplanning, as well as Bogleheads and the Financial Wisdom Forum. Domain coverage includes Stock (453 dialogues), Investment (185), Personal Finance (141), Banking (150), Loan (114), General Finance (52), Credit Card (66), Tax (102), Trading (69), and Others (85). The corpus also records geographical diversity, including globally relevant queries and region-specific content for the USA, India, UK, Canada, Australia, and Europe.

Annotation was conducted in multiple phases with finance undergraduates, a computer science doctoral student, and seasoned financial advisors as validators. Dialogues were transformed into coherent user-bot exchanges with at least three turns, and entries include conversation ID, query sequence, response sequence, region/domain labels, and politeness labels (Polite, Impolite, or Neutral). The dataset further supports attached commonsense context for training.

Technically, Fin-Vault is integrated into Fin-Ally through COMET-BART-embedded commonsense context and Direct Preference Optimization (DPO). Commonsense triples are retrieved from COMET-BART and external resources such as ConceptNet, filtered with a cosine similarity threshold of 0.7, and fused with encoder states. Evaluation includes BLEU, ROUGE, BERTScore, METEOR, politeness metrics, and human evaluation. In this usage, Fin-Vault is a dataset for response quality, contextual relevance, politeness, and human alignment, not an adversarial execution benchmark.

5. Fin-Vault in biometric fuzzy-vault cryptography

In the provided literature summaries, Fin-Vault is also used to denote a multi-biometric fuzzy vault based on face and multiple fingerprints (Rathgeb et al., 2023). The system fuses biometric integer sets at the feature level. For traits or instances $\mathbf{A}_1,\dots,\mathbf{A}_N$ , with $N=5$ for one face and four fingerprints, fusion is defined as

$\mathbf{A} = \bigcup_{i=1}^N \{ i + N x \mid x \in \mathbf{A}_i \}.$

This indexing prevents overlap between modalities and, according to the paper’s formalization, is necessary for security to be meaningful; score-level or decision-level fusion would allow separate attacks on individual components.

The fuzzy vault is built by mapping the fused set into a finite field and choosing a random secret polynomial $\kappa \in \mathbf{F}[X]$ of degree $k-1$ . The improved construction computes

$V(X) = \kappa(X) + \prod_{a \in \mathbf{P}} (X-a),$

and stores $(V(X), H(\kappa))$ . Verification forms a fused query set, evaluates the vault polynomial on the candidate points, and reconstructs the secret using an improved Guruswami-Sudan decoder when sufficient overlap exists.

Feature extraction is modality-specific. Fingerprints use minutiae quantized via a hexagonal grid and angle discretization. Face uses ArcFace, ResNet100 features, where 512 floats are quantized into 3 bits per value via Linearly Separable Subcodes (LSSC), and the indices of 1’s form an integer set. Experiments combine FRGCv2 and MCYT-100, producing 200 virtual subjects, 1,986 genuine comparisons, and 196,614 impostor comparisons.

The principal reported result is that the fused vault achieves GMR = 100% and FMR = 0% at $k=400$ , with False Accept Security greater than 30 bits, specifically 30.48 bits. At $k=416$ , the system attains GMR = 99.90%, FMR = 0%, and FAS = 33.70 bits. The authors also identify vulnerabilities that are often ignored in multimodal fuzzy vaults: error distribution imbalance, set size imbalance, and over-optimistic security estimates. Proposed countermeasures include feature balancing, harmonized feature extraction, and password hardening, with effective security approximated as biometric FAS plus password entropy.

This biometric usage sits within a broader fuzzy-vault security literature. A classic result shows that fingerprint fuzzy vaults can be vulnerable to brute-force attack with expected work approximately $(r/t)^k$ , and reported parameter settings such as $N=5$ 0 yield complexity around $N=5$ 1, well below standard cryptographic targets (0708.2974). Signature-based fuzzy vault work similarly emphasizes the security–accuracy trade-off: adaptive key-size selection increased average system entropy from 45 bits to about 51 bits while decreasing AER by about 21% (Eskander et al., 2014).

6. Broader financial “vault” semantics

The wider literature on financial vaults uses the term in yet other senses. In DeFi credit analysis, a vault is treated as a credit instrument whose risk decomposes into Level 1 mechanical loss channels, Level 2 governance quality, and Level 3 smart contract code integrity (Zbandut et al., 19 Apr 2026). The paper identifies six structural features that break standard TradFi analogies: oracle execution divergence, endogenous recovery, full information run dynamics, timelock constrained governance, oracle manipulation and latency, and congestion-driven liquidation failure. These are translated into five tractable credit risk metrics and aggregated into a vault credit score. The emphasis here is neither agent safety nor dialogue modeling, but mechanism-grounded depositor loss assessment.

In Bitcoin custody, a vault is a covenant transaction that imposes a time-lock before funds can move to a hot wallet while permitting an immediate transfer to a deep cold recovery wallet (Swambo et al., 2020). The implementation uses pre-signed transactions with secure key deletion, a watchtower for monitoring and response, and configurable time-locks that expose an explicit security–accessibility trade-off. The architecture stresses defense-in-depth, redundancy, fault-tolerant functionality, diversity across hardware and software layers, and compartmentalization of funds across numerous vault transactions.

Taken together, these usages show that “Fin-Vault” and “FinVault” are best understood as a name family spanning distinct technical traditions: execution-grounded financial agent safety, financial dialogue data curation, biometric cryptography, DeFi risk measurement, and cryptocurrency custody. The common metaphor is a controlled boundary around sensitive actions or assets; the concrete implementations, however, are domain-specific and not interchangeable.