SEALSBench: Multilingual Safety Benchmark

• SEALSBench is a multilingual safety alignment benchmark that provides a comprehensive testbed of 296,040 safety-critical prompts across ten languages.
• It standardizes evaluation through metrics like Defense Success Rate, Precision, and F1-Score, enabling controlled cross-lingual comparison of LLM guardrail systems.
• Benchmark results highlight that LoRA-based adaptation significantly boosts multilingual safety detection, addressing performance gaps in low-resource Southeast Asian languages.

SEALSBench is a large-scale, multilingual safety alignment benchmark specifically constructed to evaluate and support the development of guardrail systems for LLMs in Southeast Asian and related languages. Developed as part of the SEALGuard project, SEALSBench targets the notable gap in multilingual safety assessment, particularly for low-resource languages, by providing a comprehensive testbed of safety-critical prompts spanning safe, unsafe, and jailbreak scenarios (Shan et al., 11 Jul 2025).

1. Dataset Construction and Multilingual Scope

SEALSBench is created through the expansion and translation of a curated set of English seed prompts, resulting in a dataset of approximately 296,040 prompts across ten languages. The ten supported languages are: English, Chinese, Indonesian, Vietnamese, Thai, Khmer, Lao, Malay, Burmese, and Tagalog.

The dataset’s composition is as follows:

Prompt Type	Unique English Prompts	Percentage of Total
Safe	18,846	63.6%
Unsafe	8,959	30.3%
Jailbreak-style	1,799	6.1%

All English prompts are translated into the nine other target languages (via Google Translate API), yielding ten language variants per unique prompt for cross-lingual comparison. This approach ensures identical translation structure across train, dev, and test splits by assigning unique English-IDs to prompt clusters.

2. Source Corpora, Categories, and Representation

SEALSBench’s prompts originate from established and vetted public corpora:

• Safe prompts are derived from instruction-following datasets such as Alpaca.
• Unsafe prompts are constructed from the BeaverTails moderation corpus, encompassing ten safety categories:
  • C1: Violent Criminal Activity
  • C2: Non-Violent Criminal Conduct
  • C3: Child Sexual Abuse
  • C4: False & Defamatory Claims
  • C5: Hazardous Professional Guidance
  • C6: Personal Information Exposure
  • C7: Discriminatory & Hateful Expression
  • C8: Self-Destructive Behavior Promotion
  • C9: Explicit Sexual Material
  • C10: Misinformation & Extremist Content
• Jailbreak-style prompts are adapted from Do-Not-Answer, CatQA, AdvBench, and Forbidden Questions, employing nine attack templates for defense bypassing.

No additional human re-annotation is performed on the translated prompts; therefore, translation-induced anomalies are uniformly distributed and non-differential across splits and languages due to prompt ID tracking.

3. Data Splitting, Annotation, and Evaluation Protocol

SEALSBench provides three splits:

Split	Percentage	Absolute Size (All Languages)	Notes
Train	5%	~14,800	Split by English-ID; all languages present
Dev	5%	~14,800	Same criteria as train
Test	90%	266,444	Safe: 169,433; Unsafe: 80,601; Jailbreak:16,410

Annotation criteria are inherited from the source corpora. No inter-annotator agreement is reported due to reliance on human-vetted benchmarks for English seeds. Data alignment between translations is strictly maintained by prompt ID, enabling controlled cross-lingual evaluation by filtering on language or prompt class.

4. Metrics and Assessment Methodology

SEALSBench standardizes evaluation with metrics at both the aggregate (all-languages) and per-language level. Let TP be true positives (unsafe/jailbreak blocked), FP false positives (safe blocked), and FN false negatives (unsafe/jailbreak passed). Metrics are defined as:

• Defense Success Rate (DSR):

$$\mathrm{DSR} = \frac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FN}}$$

Interpretable as recall for unsafe or jailbreak classes.

• Precision:

$$\mathrm{Precision} = \frac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FP}}$$

• Recall: (identical to DSR)

$\mathrm{Recall} = \mathrm{DSR}$

• F1-Score:

$$\mathrm{F1} = 2 \times \frac{\mathrm{Precision} \times \mathrm{Recall}}{\mathrm{Precision} + \mathrm{Recall}}$$

Metrics may be micro- or macro-averaged for overall system comparison, and per-language reporting is directly supported.

5. Baseline Performance and Comparative Results

SEALSBench demonstrates that state-of-the-art guardrails for English inputs, such as LlamaGuard and OpenAI Moderation, exhibit substantially degraded performance on Southeast Asian languages:

System	DSR (%)	Precision (%)	F1 (%)
SEALGuard (SeaLLM-7B+LoRA)	97.2	98.9	98.1
LlamaGuard-3-8B	49.6	96.4	64.6
LlamaGuard-3-1B	44.7	91.2	40.0
OpenAI Moderation	31.0	36.4	46.0

Per-language F1 for SEALGuard consistently exceeds 98%, while for Khmer and Burmese, LlamaGuard-3-8B and OpenAI Moderation often fall below 50%. By safety category, SEALGuard maintains DSR ≥ 95% in all ten unsafe classes, while LlamaGuard-3-8B drops below 30% DSR in categories C5, C7, and C10. Against all nine jailbreak attack templates, SEALGuard achieves DSR ≥ 95%, with LlamaGuard-3-8B falling below 30% in several strategies (e.g., Deep-Inception, Code Chameleon, Zulu, Dual-Use).

6. Adaptation Strategies, Ablations, and Significance

Ablation studies demonstrate the pronounced impact of LoRA-based adaptation on cross-lingual safety detection:

Model Variant	DSR (%)	F1 (%)
SeaLLM-7B + LoRA (SEALGuard)	97.23	98.05
SeaLLM-1.5B + LoRA	96.50	97.34
SeaLLM-7B + NeMo guardrails	49.88	77.17
SeaLLM-1.5B + NeMo guardrails	89.45	50.20
SeaLLM-7B (no adaptation)	15.07	25.80
SeaLLM-1.5B (no adaptation)	18.48	28.27

The LoRA adaptation provides up to a 72% absolute gain in F1 over unadapted models, while model size (7B vs. 1.5B) yields only a marginal improvement (~1% F1) when LoRA is present. This suggests that adaptation strategy dominates model scaling in enhancing multilingual safety alignment on SEALSBench.

7. Release, Accessibility, and Usage

SEALSBench, along with the pre-trained SEALGuard models and all necessary training and evaluation scripts, are released at https://github.com/awsm-research/SEALGuard. The provided resources enable researchers to replicate findings, benchmark novel guardrail systems, and extend safety alignment research for multilingual LLMs in Southeast Asian and other languages. The licensing terms are specified in the repository documentation (Shan et al., 11 Jul 2025).