Swiss-Bench 003: Evaluating LLM Reliability and Adversarial Security for Swiss Regulatory Contexts

Abstract: The deployment of LLMs in Swiss financial and regulatory contexts demands empirical evidence of both production reliability and adversarial security, dimensions not jointly operationalized in existing Swiss-focused evaluation frameworks. This paper introduces Swiss-Bench 003 (SBP-003), extending the HAAS (Helvetic AI Assessment Score) from six to eight dimensions by adding D7 (Self-Graded Reliability Proxy) and D8 (Adversarial Security). I evaluate ten frontier models across 808 Swiss-specific items in four languages (German, French, Italian, English), comprising seven Swiss-adapted benchmarks (Swiss TruthfulQA, Swiss IFEval, Swiss SimpleQA, Swiss NIAH, Swiss PII-Scope, System Prompt Leakage, and Swiss German Comprehension) targeting FINMA Guidance 08/2024, the revised Federal Act on Data Protection (nDSG), and OWASP Top 10 for LLMs. Self-graded D7 scores (73-94%) exceed externally judged D8 security scores (20-61%) by a wide margin, though these dimensions use non-comparable scoring regimes. System prompt leakage resistance ranges from 24.8% to 88.2%, while PII extraction defense remains weak (14-42%) across all models. Qwen 3.5 Plus achieves the highest self-graded D7 score (94.4%), while GPT-oss 120B achieves the highest D8 score (60.7%) despite being the lowest-cost model evaluated. All evaluations are zero-shot under provider default settings; D7 is self-graded and does not constitute independently validated accuracy. I provide conceptual mapping tables relating benchmark dimensions to FINMA model validation requirements, nDSG data protection obligations, and OWASP LLM risk categories.

• The paper introduces a Swiss-adapted benchmark that quantitatively evaluates LLM reliability and adversarial security against FINMA and nDSG standards.
• It utilizes the expanded HAAS v2 framework with eight dimensions and custom Swiss tasks to reveal a pronounced reliability-security gap.
• Empirical results show that no model excels in both factual reliability (D7) and adversarial security (D8), indicating critical deployment risks.

Swiss-Bench 003: LLM Reliability and Security in the Swiss Regulatory Landscape

Motivation and Benchmark Scope

The proliferation of LLMs in high-stakes, regulated environments—especially in Swiss finance, compliance, and governance—necessitates multidimensional evaluation protocols that extend beyond Anglophone-centric paradigms. "Swiss-Bench 003: Evaluating LLM Reliability and Adversarial Security for Swiss Regulatory Contexts" (2604.05872) systematically addresses this gap by operationalizing two critical, previously uncaptured properties: (1) model production reliability, tightly mapped onto Swiss FINMA and nDSG requirements, and (2) adversarial security under domain-specific threat scenarios.

SBP-003 expands the Helvetic AI Assessment Score (HAAS) framework to eight dimensions, notably incorporating D7 (Self-Graded Reliability Proxy) and D8 (Adversarial Security). The design is decisively jurisdiction-adapted: all tasks and benchmarks are constructed from Swiss law, financial regulation, linguistic diversity (DE, FR, IT, EN), and security threats. The benchmark suite includes Swiss-specific adaptations of TruthfulQA, IFEval, SimpleQA, NIAH (for D7), and custom PII-Scope, system prompt leakage, and dialectal comprehension challenges (for D8).

Methodological Design

SBP-003's dataset encompasses 808 Swiss-adapted items across seven tasks, with stringent content generation—a cycle of expert specification, AI-assisted drafting, validation against Swiss sources, human review, and multilingual back-translation. The evaluation spans ten production LLMs, selected for prominence and diversity in model families, and leverages OpenRouter API under zero-shot, provider-default decoding.

HAAS v2 aggregates per-dimension scores into configurable profiles for compliance, security, and reliability officers, with weighting regimes reflecting the priorities of disparate stakeholders. D7 aggregates scores across four reliability tasks, scored via the model's own facticity judgments (self-grading). D8 composes security across Swiss PII extraction resistance and system prompt leakage, both externally judged (Qwen3-235B) with tiered scoring rubrics designed to reflect practical operational risks.

Empirical Findings: Reliability (D7) and Adversarial Security (D8)

SBP-003 reveals a pronounced disjunction between model capabilities in reliability and adversarial security, with only weak correlation between D7 and D8. The best-performing models by dimension are:

• D7 (Self-Graded Reliability Proxy): Qwen 3.5 Plus, achieving 94.4%, leads by a significant margin, outperforming both closed and open LLM competitors on Swiss factual, instruction-following, and retrieval benchmarks.
• D8 (Adversarial Security): GPT-oss 120B outperforms all evaluated models (60.7%) despite its lower inference cost and open-weights status, a result contrary to standard expectations for security-hardening.

Self-graded reliability scores (D7) are systemically higher than adversarial security scores (D8) by a mean gap of 41.7 points, reflecting either difference in task difficulty or the effects of targeted RLHF optimization in mainstream LLMs. Crucially, no model achieves strong, simultaneous reliability and adversarial security—an empirical contradiction to the notion of broad cross-dimensional excellence.

Figure 1: D7 self-graded reliability proxy scores across four Swiss-adapted benchmarks.

Figure 2: D8 security heatmap showing PII-Scope and System Prompt Leakage resistance. All models score below 43% on PII extraction defense (left column), while system prompt leakage resistance (right column) varies by over 63 percentage points.

The PII extraction defense is universally weak—no model exceeds 42.4% under the PII-Scope rubric—emphasizing a critical vulnerability relative to the Swiss nDSG's data minimization requirements. In contrast, system prompt leakage exhibits significant variance (24.8% to 88.2%), highlighting idiosyncratic deployment security profiles:

• System prompt leakage resistance: GPT-oss 120B achieves 88.2%, while Mistral Large 3 scores just 24.8%.

Swiss German comprehension is high (>70%) for most models, illustrating robust cross-dialectal linguistic performance, but this does not translate to hardened security outcomes.

Multi-Dimensional Trustworthiness Profiles

Aggregated HAAS v2 radar plots and D7 vs. D8 scatter distributions visualize the non-overlapping strengths among models. Notably, none of the top performers in D7 rank equivalently in D8, and vice versa.

Figure 3: HAAS v2 radar profiles for the top 5 models across all seven Swiss-adapted benchmarks. The ``collapse'' from D7 (outer ring) to D8 benchmarks (inner ring, particularly PII-Scope and Leakage) is visible for all models.

Figure 4: D7 (self-graded reliability proxy) vs. D8 (adversarial security) scatter plot. Dashed lines indicate median values. GPT-oss 120B occupies a unique position: lowest D7 but highest D8. No model scored near the top of both dimensions in this evaluation.

This dissociation has material implications. For instance, financial institutions required by FINMA to demonstrate "reliability, reproducibility, and security" cannot draw strong assurance from high factuality alone; weaknesses on adversarial axes or PII leakage constitute acute governance risks. The analysis also demonstrates that open-weight models can outperform proprietary models in some security dimensions, contesting assumptions that security maturity maps reliably to closedness or cost.

Regulatory Alignment and Theoretical Implications

SBP-003 systematically maps HAAS v2's quantitative axes onto Swiss regulatory expectations (FINMA 08/2024, nDSG, SDI Trust Label) and the OWASP LLM Application Top 10. This mapping is consequential: SBP-003 provides—for the first time—clear operational alignment between empirical LLM evaluation and Swiss legal risk, with direct hooks for compliance, risk, and product officer reporting. However, the study identifies areas not captured by model-centric benchmarking, such as explainability and human-in-the-loop decision rights, pointing toward future expansion of evaluation dimensions.

The observed reliability-security gap supports the hypothesis that current frontier models are over-optimized for provider leaderboard tasks but under-optimized for domain-specific adversarial threats; exposure to a restricted set of safety finetuning prompts is insufficient to ensure deployable security in multilingual, regulated environments. The rare profile of GPT-oss 120B—a model with strong adversarial security but weak self-reported reliability—suggests that RLHF reward modeling for helpfulness and for risk aversion can, in practice, drive capabilities in divergent directions.

Methodological Notes

The study employs rigorous statistical methodology (Wilson score intervals, judge validation with external models), but also acknowledges limitations: score inflation or deflation due to self-grading, single time-point evaluation, rubric subjectivity, and possible benchmark leakage. Notably, D8 security evaluation has stronger judge disagreement (81% inter-judge agreement, varying tier calibration) and requires further human annotation for definitive ranking.

SBP-003's evaluation items are withheld to minimize contamination and competitive overfitting. However, all protocols, scoring tiers, and aggregation methods are described to enable conceptual replication.

Practical and Future Directions

The results have direct implications for LLM deployment risk management in Swiss-regulated settings. The differentiated profiles support evidence-based model selection, targeted risk mitigation, and ongoing monitoring protocols. Regulatory alignment facilitates clear communication with Swiss supervisory authorities.

Future work should include: extension to agentic multi-step scenarios, inclusion of additional standard (non-Swiss-adapted) adversarial benchmarks, pass$^k$ consistency assessment, local infrastructure evaluation, and formal judge calibration against human experts. The modular structure of SBP-003 supports adoption by other jurisdictions via content substitution.

Conclusion

Swiss-Bench 003 (2604.05872) establishes a new, jurisdiction-specific standard for LLM evaluation in regulatory contexts—empirically revealing that high factual reliability does not predict adversarial robustness, and that benchmarking must reflect local threat models and compliance structures. The findings are directly actionable for practitioners, model evaluators, and regulators focused on Swiss deployments, and the framework offers a template for broader European adaptation as multilingual LLM deployment accelerates.