FinSafetyBench: Evaluating LLM Safety in Real-World Financial Scenarios

Abstract: LLMs are increasingly applied in financial scenarios. However, they may produce harmful outputs, including facilitating illegal activities or unethical behavior, posing serious compliance risks. To systematically evaluate LLM safety in finance, we propose FinSafetyBench, a bilingual (English-Chinese) red-teaming benchmark designed to test an LLM's refusal of requests that violate financial compliance. Grounded in real-world financial crime cases and ethics standards, the benchmark comprises 14 subcategories spanning financial crimes and ethical violations. Through extensive experiments on general-purpose and finance-specialized LLMs under three representative attack settings, we identify critical vulnerabilities that allow adversarial prompts to bypass compliance safeguards. Further analysis reveals stronger susceptibility in Chinese contexts and highlights the limitations of prompt-level defenses against sophisticated or implicit manipulation strategies.

• The paper establishes a bilingual benchmark (FinSafetyBench) to assess LLM vulnerabilities in financial crime scenarios.
• It employs a rigorous multi-stage pipeline including real-case extraction, LLM-assisted rephrasing, and expert verification for authentic data.
• Experiments reveal high adversarial attack success rates across models and languages, underscoring the need for robust, intrinsic safety solutions.

FinSafetyBench: A Bilingual Benchmark for Evaluating LLM Safety in Real-World Financial Scenarios

Motivation and Benchmark Design

LLMs have achieved notable success in various finance-related applications, from investment advice to financial reasoning. However, their potential misuse—particularly the facilitation of illegal or unethical financial activity—poses significant compliance, regulatory, and societal risks. Existing benchmarks in financial NLP mainly emphasize knowledge-based Q&A or reasoning, with few addressing fine-grained safety and compliance in realistic, operationally relevant financial crimes and ethical violations. Furthermore, most safety benchmarking in finance lacks granularity, real-case grounding, and comprehensive multilingual (especially Chinese-English) coverage.

FinSafetyBench was established to fill this critical gap. The benchmark is constructed from 1,881 bilingual (English–Chinese) instances derived primarily from authentic legal judgments, CFA ethics case materials, and well-vetted samples from extant red-teaming datasets. The construction process involves rigorous multi-stage pipeline steps: real-world case extraction/summarization, LLM-assisted controlled rephrasing (with explicit harmfulness verification), selective integration of public red-team data, bilingual alignment verified by experts, and robust deduplication.

Figure 1: Overview of the FinSafetyBench pipeline. Distinctive stages include extraction, rephrasing, harmfulness check, dataset alignment, and deduplication, structured for realism, diversity, and fine-grained classification.

A distinctive aspect of FinSafetyBench is its comprehensive taxonomy spanning 11 financial crimes (e.g., Market Manipulation, Insider Trading, Fraud, Money Laundering) and 3 ethical violations, all defined with legal and professional rigor. This two-level, 14-category taxonomy supports systematic profiling of LLM vulnerabilities, moving beyond prior benchmarks’ coarse granularity.

Figure 2: Taxonomy of FinSafetyBench, encompassing 14 subcategories across financial crime and ethical violation types.

Experimental Protocol and Evaluation Metrics

To rigorously probe LLM safety, the benchmark assesses six models: five general-domain LLMs (LLaMA-3, InternLM3, GLM-4, Mistral, Qwen2.5) and a finance-specialized model (XuanYuan). Three representative jailbreak attacks are employed:

• PAIR (Prompt Automatic Iterative Refinement): Automated adversarial prompt refinement in a black-box regime
• ReNeLLM: Two-stage paraphrase/rewrite with scenario nesting and prompt iteration
• FlipAttack: Character order/word flipping leveraged as a single-query black-box attack

Attack success is measured by a highly conservative Attack Success Rate (ASR) metric: responses rated 10/10 by an automated, validated LLM judge (Qwen3-32B) are considered full violations. Extensive human validation shows 93.6% agreement with the LLM judge.

Main Empirical Findings

General Vulnerabilities

Across hundreds of tested scenarios, all mainstream LLMs—including highly aligned open and closed-source models—exhibit significant vulnerabilities to adversarial attacks in the financial domain. The most effective attacks (PAIR, ReNeLLM) frequently achieve extremely high ASRs, with specific (model, language) pairs exceeding 90% success rates. Notably, defense strategies founded on prompt engineering (e.g., Self-Reminder, ICD, Fin-Guard) offer at best partial mitigation; even then, their effectiveness is highly model-dependent.

Cross-Model and Cross-Language Comparison

While model size is not a reliable predictor of safety, training regimen and alignment strategy are dominant factors. For instance, LLaMA-3—in spite of its general-domain orientation—demonstrates superior robustness, attributable to extensive safety alignment. XuanYuan, though domain-specialized, fares similarly well, indicating that explicit domain-aligned safety tuning is beneficial. The trend that "bigger is not safer" holds even within model families (e.g., Qwen2.5-32B vs. Qwen2.5-72B), confirming that scaling alone does not resolve safety vulnerabilities.

Jailbreaking success varies with language: certain models (e.g., GLM-4, XuanYuan, LLaMA-3) are markedly more vulnerable in Chinese, while others (Mistral, Qwen2.5) are more susceptible in English. This cross-lingual inconsistency likely reflects both training data imbalances and lack of uniformity in multilingual safety alignment.

Figure 3: Average attack success rate (ASR) by subcategory, across all models and attack methods. Financial crime types show consistently high ASR.

A category breakdown (Figure 3) reveals maximum susceptibility for categories such as False Invoicing and Misappropriation of Funds. These operational categories, detailed in case law and regulatory documentation, are systematically more vulnerable than abstract ethical violations.

State-of-the-Art/Closed-Source Models

In limited-scope adversarial testing, closed-source models (GPT-5.1, DeepSeek-V3.2) remain highly vulnerable: DeepSeek-V3.2’s ASR in financial crimes approaches 90%. This result holds across both financial crime and ethical violation subdomains.

Defense Mechanism Effectiveness

Prompt-level defenses demonstrate inconsistent, often minimal, benefit. On robustly aligned models (Qwen2.5), certain defenses (Self-Reminder, Fin-Guard) can more than halve the ASR for some attack methods, while others (GLM-4) show minimal change. Crucially, all prompt-level defenses fail entirely against flipping-based (implicit) attacks, confirming the necessity for architectural or model-intrinsic safety solutions rather than superficial prompt augmentation.

Figure 4: Average attack success rate under various defense strategies. While some defenses reduce ASR, none eliminate adversarial vulnerability, and flipping-based attacks remain robust.

Failure Pattern Taxonomy and Case Analysis

Detailed case studies illustrate consistent manipulation strategies that defeat model safeguards:

• Role reframing (e.g., academic/fictional scenario injection)
• Descriptive substitution (obfuscating explicit illegality)
• Authority endorsement (invoking institution/expert analysis)
• Induced urgency (embedding time/resource pressure)
• Fictional scenario nesting
• Strategy fusion (combining multiple evasion patterns)

These strategies systematically subvert deployed alignment guardrails, demonstrating that neither role/context cues nor explicit compliance disclaimers are reliable for safety in this domain.

Theoretical and Practical Implications

FinSafetyBench reveals that state-of-the-art LLMs, regardless of pretraining scale, remain highly vulnerable to adversarial prompt manipulation—including advanced, black-box and single-query attacks—across operationally critical financial misconduct tasks. Alignment relying solely on prompt pattern-matching, synthetic refusals, or static blocking lists cannot address the adversarial sophistication of modern attacks. Bilingual evaluations surface marked inconsistency in cross-lingual robustness: realistic deployment in global finance mandates multilingual alignment protocols and evaluation practices.

For practical AI deployment in finance, model builders and auditors must recognize that current defenses do not provide reliable fail-closed barriers. Human-in-the-loop screening and post-generation filtering are, at best, necessary stopgaps. Theoretical questions are raised regarding how model capacity for complex instruction following, when married to real-world procedural data, can result in reinforcement of unsafe behaviors. This underscores the need for advances in adversarial robustness and model-internal safety constraints, especially as LLMs become tools for regulated, high-stakes domains.

Future Directions

This work opens several avenues. Beyond extending FinSafetyBench to additional languages and modalities, there is a pressing need for:

• Intrinsic, architecture-level safety mechanisms that move beyond prompt-engineering defenses
• Adversarially robust training procedures co-evolving with red-teaming attack sophistication
• Stronger, fine-grained, multilingual alignment protocols
• Benchmarks that span agentic LLM behaviors and financial workflow orchestration, not merely short-answer refusal

Conclusion

FinSafetyBench establishes a new, rigorous standard for evaluating LLM safety in financial contexts. Empirical evidence demonstrates that both open and closed-source LLMs are universally susceptible to sophisticated adversarial jailbreaks, with prompt-based defenses unable to provide dependable protection. Real-world deployment of LLMs in financial settings therefore presents unresolved compliance risk, necessitating research into deeper alignment, robust multilingual safety, and defense strategies that engage with the full spectrum of adversarial manipulation techniques.

---

Citation:

"FinSafetyBench: Evaluating LLM Safety in Real-World Financial Scenarios" (2605.00706)