Cross-Lingual Jailbreak Detection via Semantic Codebooks

Abstract: Safety mechanisms for LLMs remain predominantly English-centric, creating systematic vulnerabilities in multilingual deployment. Prior work shows that translating malicious prompts into other languages can substantially increase jailbreak success rates, exposing a structural cross-lingual security gap. We investigate whether such attacks can be mitigated through language-agnostic semantic similarity without retraining or language-specific adaptation. Our approach compares multilingual query embeddings against a fixed English codebook of jailbreak prompts, operating as a training-free external guardrail for black-box LLMs. We conduct a systematic evaluation across four languages, two translation pipelines, four safety benchmarks, three embedding models, and three target LLMs (Qwen, Llama, GPT-3.5). Our results reveal two distinct regimes of cross-lingual transfer. On curated benchmarks containing canonical jailbreak templates, semantic similarity generalizes reliably across languages, achieving near-perfect separability (AUC up to 0.99) and substantial reductions in absolute attack success rates under strict low-false-positive constraints. However, under distribution shift - on behaviorally diverse and heterogeneous unsafe benchmarks - separability degrades markedly (AUC $\approx$ 0.60-0.70), and recall in the security-critical low-FPR regime drops across all embedding models.

• The paper introduces a training-free semantic similarity filtering method that blocks unsafe prompts by comparing embeddings against a static English codebook.
• It systematically evaluates cross-lingual transfer using metrics like ROC-AUC and TPR at FPR ≤1%, highlighting strong performance in template-driven attacks.
• Results reveal a trade-off under distribution shifts, underscoring the need for adaptive, hybrid safety mechanisms for diverse, non-English prompt scenarios.

Cross-Lingual Jailbreak Detection via Semantic Codebooks: An Expert Analysis

Motivation and Problem Statement

Current alignment and safety interventions for LLMs overwhelmingly focus on English. This linguistic imbalance produces exploitable vulnerabilities: empirical studies demonstrate that translating adversarial prompts enables systematic jailbreaking across multilingual deployments, and refusal behaviors are frequently non-transferable. Internal mitigation techniques (e.g., RLHF, DPO, LoRA) require privileged access and retraining, rendering them ineffective in black-box scenarios. External English-centric classifiers and rule-based filters can be trivially bypassed by cross-lingual translation and paraphrasing, as documented in jailbreak benchmarks. Semantic similarity filtering, which leverages the structure of multilingual embedding spaces, offers a robust alternative—but its generalization across languages and under distribution shift had not been systematically characterized prior to this study.

Detection Framework and Methodology

The paper introduces a training-free guardrail for cross-lingual jailbreak detection. User input, in any language, is encoded with a multilingual embedding model (BGE-M3 as default), and its embedding is compared via cosine similarity against a static English codebook of known unsafe prompts. If the similarity exceeds a threshold $\tau$ (selected on validation to ensure FPR $\leq 1\%$), the prompt is blocked; otherwise, it is forwarded to the LLM. The codebook is curated from prompt injection safety datasets and cross-checked by ensemble guard models to reduce annotation noise.

Figure 1: Overview of the semantic filtering pipeline as an external guardrail requiring neither translation nor model fine-tuning.

Cross-lingual transfer is evaluated systematically: unsafe queries are translated into Russian, Chinese, and Arabic via both commercial (Google Translate) and neural (M2M100) pipelines. Four heterogeneous safety benchmarks, three embedders, and three target LLMs (Qwen, Llama, GPT-3.5) provide comprehensive coverage. Evaluation focuses on classification metrics (ROC-AUC, TPR, FPR), especially in the low-FPR regime (TPR at FPR $\leq 1\%$), and mitigation measured by reduction in attack success rate (ASR).

Results: Two Distinct Regimes of Cross-Lingual Transfer

Separability and ROC Analysis

Semantic similarity generalizes robustly on curated benchmarks (prompt-injection safety and template-driven attacks), achieving high cross-lingual AUC (up to 0.993 in English; 0.847–0.884 under translation). Even with typologically distant languages, canonical attack structures are reliably captured. However, on behaviorally diverse benchmarks (JailbreakBench, Aegis-AI Content Safety), AUC drops markedly (0.593–0.627), and discriminative power decays as the distributional overlap between unsafe and safe prompts increases.

Figure 2: ROC curves contrasting strong separability in templated attacks (Benchmark 1) with degraded discrimination in heterogeneous settings (Benchmark 4).

Low-FPR Regime

Strict false-positive constraints are required in deployment. On template-driven benchmarks, recall at FPR $\leq 1\%$ is high (TPR 78–92% depending on language), making similarity filtering deployable as a first-line defense. In contrast, on heterogeneous benchmarks, recall collapses to single digits (<7%), confirming the limitation of semantic-only detection in complex unsafe distributions.

End-to-End Jailbreak Mitigation

When deployed as a pre-filter, the semantic codebook removes 96.2% of successful attacks on prompt-injection benchmarks and 50% on broader unsafe content benchmarks. Under distribution shift, mitigation weakens significantly: mean attack reduction drops to 43.7% (Benchmark 3) and 18.6% (Benchmark 4), with increased instability across languages and models. The absolute reduction in successful jailbreaks, rather than relative ASR, is the critical security metric.

Sensitivity Analysis: Codebook and Embedder Choice

Larger codebooks increase TPR but inflate FPR, highlighting a coverage–false-alarm trade-off. Embedder choice is pivotal; BGE-M3 outperforms alternatives (e.g., multilingual-e5-large, jina-v3) especially in the low-FPR regime. All embedders fail in highly heterogeneous unsafe distributions, indicating that advances in alignment or representation isotropy may be insufficient without architectural innovations.

Discussion and Theoretical Implications

The results establish semantic codebooks as a deployable, stable guardrail for canonical attacks in cross-lingual settings. The explicit demonstration of two distinct behavioral regimes is significant: semantic similarity generalizes in templated spaces but cannot exhaustively capture adversarial, rephrased, or novel unsafe behaviors outside the English-centric template family. This empirical bifurcation suggests that current embedding architectures, though aligned, are not semantically isomorphic across languages under all prompt distributions, particularly for typologically distant or syntactically obfuscated queries.

The research challenges the assumption that well-aligned embedding spaces suffice for zero-shot cross-lingual attack detection and underscores the necessity for hybrid, cascaded, or adaptive mechanisms. It also reifies the fundamental blindness of static English codebooks to emergent harm patterns in non-English scenarios.

Practical Implications and Future Directions

Practically, similarity-based filtering is lightweight and training-free, allowing integration as an external guardrail in black-box settings, and effective for canonical attack patterns. However, it should not be relied upon for comprehensive safety; integration with secondary signals (perplexity anomaly, syntactic patterning, adversarial LLM adjudication) is imperative for realistic threat models.

Future developments should target:

• Continuous codebook adaptation via active learning on production false negatives.
• Language-native codebooks and isotropy-aware deduplication to improve recall in typologically distant languages.
• Cascaded multi-signal pipelines maintaining operational FPR constraints under distribution shift.
• Systematic analysis of input sensitivity to guide normalization strategies and prompt-aware architecture design.
• Longitudinal live deployment to validate resilience and operational cost.

Conclusion

This study provides a rigorous characterization of semantic similarity filtering for cross-lingual jailbreak detection, establishing its efficacy and clear limitations across diverse unsafe prompt distributions. The proposed guardrail offers a deployable solution for canonical attacks but is insufficient for broader unsafe behaviors, highlighting the need for adaptive, hybrid safety architectures in multilingual LLM deployments.