LinguaSafe: Multilingual LLM Safety Benchmark

• LinguaSafe is a multilingual safety benchmark that evaluates large language models across 12 languages, emphasizing linguistic authenticity and cultural localization.
• It integrates native, translated, and transcreated data and employs a hierarchical safety taxonomy with 5 domains, 23 subtypes, and 4 severity levels.
• The evaluation framework measures both under-alignment and over-alignment through direct and indirect metrics, offering detailed diagnostics of multilingual safety.

Searching arXiv for the specified LinguaSafe paper and closely related multilingual safety work to ground the article with current references. Tool unavailable in this environment, so I will rely on the supplied arXiv records and cite them directly. LinguaSafe is a comprehensive multilingual safety benchmark for LLMs, designed to evaluate safety behavior across diverse linguistic and cultural contexts rather than only in English. The benchmark comprises 45,000 entries in 12 languages, from Hungarian to Malay, and combines translated, transcreated, and natively sourced data with a multidimensional evaluation framework that includes direct safety assessment, indirect safety assessment, and explicit oversensitivity evaluation (Ning et al., 18 Aug 2025).

1. Definition and research context

LinguaSafe addresses a central problem in multilingual LLM safety: safety alignment and evaluation have been predominantly English-centric, while real deployment is multilingual. Existing multilingual safety resources had already shown important cross-lingual vulnerabilities, including higher unsafe response rates for non-English queries and substantial language-specific inconsistencies, but they often relied heavily on translated English data, focused narrowly on toxicity or jailbreaks, or lacked explicit oversensitivity analysis (Wang et al., 2023); (Friedrich et al., 2024).

Within that landscape, LinguaSafe is defined as a multilingual safety benchmark that emphasizes linguistic authenticity and balanced evaluation. It is explicitly presented as filling a gap left by earlier multilingual safety benchmarks such as RTP-LX, PTP, MultiJail, Aya Red-teaming, and XSAFETY. Its distinguishing claim is not merely broader language coverage, but the combination of native data, transcreated data, translated data, hierarchical safety labels, and evaluation procedures that separate under-protection from over-restriction (Ning et al., 18 Aug 2025).

A common misconception is that multilingual safety benchmarking can be reduced to translating English prompts. LinguaSafe is built on the opposite premise. Its design treats translation, cultural localization, and native-language sourcing as distinct operations because literal translation often fails to preserve idioms, implicit harms, and legal or social context. This suggests that the benchmark is intended not only to measure refusal robustness, but also to reveal where safety systems misread or overread culturally grounded content.

2. Corpus design and linguistic authenticity

The dataset contains 12 languages spanning high-, mid-, and low-resource settings based on CommonCrawl. The high-resource set is English, Russian, Chinese, Vietnamese, and Czech; the mid-resource set is Arabic, Korean, Thai, Hungarian, and Serbian; the low-resource set is Malay and Bengali. The benchmark is described as the first comprehensive safety benchmark explicitly covering Hungarian and Malay (Ning et al., 18 Aug 2025).

LinguaSafe uses three data types. Native Data consists of organically generated content from online forums, social media, and web-scraped corpora such as The Pile and Culturax. Harmful native content is pre-filtered using Llama Guard 3 and MD-Judge-v0.2-internlm2_7b, and K-Means clustering with mGTE sentence embeddings is used to reduce redundancy while preserving diversity. Translated Data propagates English safety benchmark content into other languages. Transcreated Data adapts content so that it reads as naturally native in the target language while preserving intent, tone, harmfulness level, and task function (Ning et al., 18 Aug 2025).

The transcreation pipeline is formalized as TATER, short for Task-Aware Translate, Estimate and Refine. Its three stages are Translate, Estimate, and Refine. In the Estimate stage, an LLM produces MQM-style error annotations and a task-based transcreation score, including fields such as critical, major, minor, task_acc, and task_error. In the Refine stage, the model rewrites the translation using those annotations and task requirements, after which human annotators review the output for cultural appropriateness, preservation of harm type, and preservation of harm level (Ning et al., 18 Aug 2025).

The empirical motivation for TATER is explicit. For Bengali and Malay, manual evaluation of 500 transcreated instances per language reported baseline error rates of 71% and 36%, respectively, for vanilla LLM translation, and 12% and 3% after the full TATER pipeline plus human refinement (Ning et al., 18 Aug 2025). This is one of LinguaSafe’s strongest methodological claims: multilingual safety evaluation is not reliable if the linguistic form of the benchmark itself is unstable.

3. Taxonomy, severity structure, and annotation

LinguaSafe organizes content through a hierarchical safety taxonomy with 5 top-level domains, 23 subtypes, and 4 severity levels. The five domains are Crimes & Illegal Activities; Fairness & Discrimination (Justice); Explicit Content; Privacy & Property; and Harm & Misuse. The subtype inventory covers both universal and culture-specific harms, and local legal references are incorporated using resources such as Pile-of-Law and HFforLegal case-law to keep prompts realistic and jurisdictionally grounded (Ning et al., 18 Aug 2025).

Severity is encoded as four levels. L3 corresponds to severe crimes or high risk of serious harm, such as explicit instructions for terrorism or severe violence. L2 denotes medium-risk harmful or malicious prompts, including universally toxic content such as hate speech and harassment. L1 covers potential harm and context-dependent controversy. L0 denotes benign or borderline prompts that may be misclassified as risky. L0 is especially important because it is the basis for the benchmark’s oversensitivity analysis (Ning et al., 18 Aug 2025).

Annotation combines automatic and human processes. Initial severity and domain labeling use four LLM evaluators, followed by human review and adjustment. Reported inter-annotator agreement is Krippendorff’s $\alpha = 0.77$ for human annotators alone, and $\alpha = 0.89$ between pure human labels and AI+human collective labels on a sample of 500 instances. Borderline and edge cases are systematically re-evaluated. This suggests that LinguaSafe is intended as a calibrated benchmark rather than a raw collection of translated prompts (Ning et al., 18 Aug 2025).

4. Evaluation framework and metrics

LinguaSafe’s evaluation framework is explicitly multidimensional. Direct evaluation measures whether the model can recognize unsafe content. It uses a multiple-choice template in which the model must select all unsafe prompts from a set of options labeled across severity levels. Performance is summarized with a Severity-Weighted Confusion Matrix rather than an unweighted safe/unsafe score (Ning et al., 18 Aug 2025).

The weighting matrix is defined as

$$\mathbf{W}_s = \begin{bmatrix} 1 & 1 - \frac{1}{3}\alpha & 1 - \frac{2}{3}\alpha & 0 \ 0 & \frac{1}{3}\alpha & \frac{2}{3}\alpha & 1 \end{bmatrix},$$

and the Severity-Weighted Confusion Matrix is

$\mathbf{C}_{sw} = \mathbf{W}_s \times \mathbf{C}.$

When $\alpha = 0$, severity is ignored; when $\alpha = 1$, the four severity levels are fully accounted for. LinguaSafe sets $\alpha = 0.6$ in its experiments. From $\mathbf{C}_{sw}$, the benchmark derives severity-weighted F1 for Safety Recognition and a severity-weighted False Positive Rate, which is used as a direct measure of oversensitivity (Ning et al., 18 Aug 2025).

Indirect evaluation moves from recognition to interaction. It includes intentional harmful prompts, unintentional unsafe prompts, and extended L0 borderline prompts. For L3 prompts, only whether the model rejects the prompt is evaluated. For L2-L0 prompts, both rejection and helpfulness are assessed. Rejections are detected automatically using keyword-based filters and a small LLM classifier. If the model does not reject a harmful prompt, the response is examined with omni-moderation-latest and open-source guard models such as Llama Guard 3 and MD-Judge (Ning et al., 18 Aug 2025).

From this pipeline, LinguaSafe computes an Unsafe Rate and a severity-weighted Unsafe Rate. Oversensitivity is measured through both direct and indirect channels: the benchmark defines an Oversensitivity Rate as the average of the severity-weighted FPR from direct evaluation and the Overrefusal Rate from indirect evaluation. This design is important because it distinguishes two failure modes that are often conflated in multilingual safety work: under-alignment, where harmful content is missed, and over-alignment, where benign content is unnecessarily refused (Ning et al., 18 Aug 2025).

5. Empirical findings and safety patterns

LinguaSafe evaluates both closed-source and open-source models. The closed-source set includes GPT-4o, Claude-3.5-Sonnet, and Gemini-2.0-Flash. The open-source set includes Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, Llama-3.1-8B-Instruct, Phi-4, Gemma-2-27B-IT, and DeepSeek-V3-0324 (Ning et al., 18 Aug 2025).

The main reported summary is a language- and domain-wise vulnerability score, where lower is better. Claude-3.5-Sonnet achieves the lowest vulnerability scores in most languages, including 6.97 in Arabic and 5.93 in Thai. GPT-4o is generally second best, with 15.60 in English, 24.47 in Bengali, and 19.92 in Malay. Among open-source systems, Llama-3.1-8B-Instruct exhibits comparatively high vulnerability in several languages, including 39.51 in Russian and 47.02 in Bengali. Phi-4 and Gemma-2-27B-IT show stronger average open-source performance, and Qwen2.5-7B and Mistral-7B perform notably well for their size (Ning et al., 18 Aug 2025).

The language pattern is not reducible to a simple high-resource versus low-resource ranking. For most models, English is safer than the other languages, but the benchmark also reports cases in which a model appears safer in certain medium-resource languages than in English. Claude-3.5-Sonnet, for instance, shows very low vulnerability in Arabic and Thai, yet extra results report very high oversensitivity in some languages, especially medium-resource ones such as Arabic and Thai. This suggests that very low vulnerability scores can sometimes reflect over-alignment rather than a balanced safe-helpful tradeoff (Ning et al., 18 Aug 2025).

Domain-wise, the benchmark reports substantial variation. Phi-4 has the best overall domain averages among the cited results, with Crimes 36.72, Harm 35.34, Fairness 39.80, Privacy 36.53, Explicit Content 36.64, and an average of 37.01. Claude-3.5-Sonnet is extremely cautious in Privacy & Property, with a vulnerability score of 1.78, but the same pattern is described as potentially over-refusing benign privacy-related content. More broadly, direct Unsafe Rates are often relatively low, while oversensitivity and severity-weighted true-negative behavior reveal that models frequently default to refusal under uncertainty (Ning et al., 18 Aug 2025).

A plausible implication is that LinguaSafe’s main empirical contribution is diagnostic rather than purely competitive. It does not only rank models; it exposes where safety systems are permissive in some languages and over-restrictive in others, and does so at the level of domains, severities, and evaluation modes.

6. Subsequent use, related benchmarks, and limitations

LinguaSafe was subsequently used as a major benchmark in later alignment work. In SHARD, it is described as “a multilingual safety benchmark containing 45,705 prompts across 12 languages,” and the paper uses the English subset of 3,267 prompts. That subset is distributed across five harm categories—Crimes and Illegal Activities, Harm and Misuse, Explicit Content, Fairness and Justice, and Privacy and Property—and includes harm levels from 0 to 3. In that setting, LinguaSafe functions simultaneously as an evaluation benchmark, a source of harm categories and severity labels, and training data for a RoBERTa-large harm-category classifier (Manoranjan et al., 14 Jun 2026).

LinguaSafe also sits within a broader family of multilingual safety benchmarks rather than exhausting the space by itself. Related work includes LSR, which focuses on cross-lingual refusal degradation in Yoruba, Hausa, Igbo, and Igala and introduces Refusal Centroid Drift; M-ALERT, which studies category-wise safety inconsistency across English, French, German, Italian, and Spanish; and Lingua-SafetyBench, which extends multilingual safety evaluation to vision-LLMs through 100,440 harmful image-text pairs across 10 languages (Faruna, 27 Feb 2026); (Friedrich et al., 2024); (Shi et al., 30 Jan 2026).

Its limitations are explicitly acknowledged. Twelve languages remain a limited subset of global linguistic diversity, and native data collection and annotation are resource-intensive. The taxonomy covers five domains and 23 subtypes, but new harms such as misinformation, political harms, and evolving adversarial patterns may require extension. The benchmark is also presented as a foundation for future multilingual safety preference datasets rather than a completed alignment dataset in itself. Proposed future directions include more languages, new safety dimensions, dynamic or adversarial data generation, and human preference studies on the helpfulness–harmlessness tradeoff (Ning et al., 18 Aug 2025).

LinguaSafe is publicly available for research through a Hugging Face dataset and a GitHub repository, and it is released under CC BY-NC-SA 4.0 for non-commercial use (Ning et al., 18 Aug 2025). In the current multilingual safety literature, its main significance lies in making multilingual safety measurable at several levels at once: content recognition, response behavior, severity calibration, and over-refusal. That combination has made it both a benchmark for model comparison and an analysis instrument for subsequent alignment research.