- The paper introduces ML-Bench, a 56K-instance benchmark built directly from 17 regional AI regulations across 14 languages without machine translation, covering seed, refined, and adversarial safety queries.
- The paper shows ML-Guard-7B reaches 0.97, 0.90, and 0.61 F1 on seed, refined, and response tasks, respectively, plus 0.92 attack accuracy, substantially outperforming most existing guardrails.
- The paper demonstrates that policy conditioning, native-language data, and diffusion-based generation support rule identification, useful rationales, staged policy updates without catastrophic forgetting, and about 1.5× better latency and throughput than a comparable autoregressive model.
Motivation and problem statement
Existing multilingual safety benchmarks for LLMs are built on general, language-agnostic risk taxonomies that are assumed to transfer across languages, and non-English data is typically produced via machine translation. This design has two consequences: guardrail models are confined to predefined hazard categories (e.g., those of Llama Guard or Qwen3Guard), and translation pipelines fail to preserve jurisdiction-specific legal expressions, culturally grounded meanings, and regulatory nuance. The paper argues that safety requirements are inherently defined by local regulations — a position consistent with region-level frameworks such as the EU AI Act — and therefore cannot be fully captured by taxonomy-plus-translation approaches. The work addresses this gap with two artifacts: ML-Bench, a policy-grounded multilingual safety benchmark covering 14 languages, and ML-Guard, a Diffusion LLM (dLLM)-based guardrail family supporting both binary safety classification and policy-conditioned compliance checking.
ML-Bench construction
ML-Bench is derived from 17 regional AI regulations spanning 14 countries and 14 languages (Arabic, Chinese, Dutch, English, French, French-Canada, German, Hindi, Italian, Japanese, Korean, Portuguese, Spanish, Turkish). A notable methodological commitment is that regulatory texts, extracted rules, and generated data are handled entirely in their native languages; machine translation is not used at any stage. For languages without an enacted AI-specific law, the EU AI Act serves as the reference framework.
The pipeline proceeds in two stages. First, article-level rule extraction: each regulation article is independently processed by GPT-5 to extract concrete safety-relevant rules, maximizing coverage of fine-grained normative requirements. Second, language-specific risk category formation: extracted rules are de-duplicated, decomposed into atomic rules, merged where redundant, and clustered into a two-layer hierarchy of high-level risk categories and fine-grained safety rules, all expressed in the native language.
Data generation then produces three progressively harder query tiers under the same regulatory standards:
- Seed queries: unsafe queries explicitly violating a given rule, paired with safe variants obtained by minimal edits that reverse intent while preserving the sensitive concept.
- Refined queries: unsafe queries rephrased into legitimate professional or bureaucratic framings (compliance review, technical documentation) in culturally realistic native-language style; safe refined queries deliberately mention prohibited concepts while remaining compliant, inducing classification confusion.
- Attack-enhanced queries: refined unsafe queries augmented with adversarial suffixes optimized via PAIR and AutoDAN to bypass guardrails.
Responses are constructed as borderline cases: unsafe responses come from Qwen3-8B, whose partial safety alignment yields policy-violating but non-overtly malicious outputs, while safe responses engage with the request before subtly redirecting toward compliant alternatives without explicit refusals.
Ground-truth labels are assigned by majority agreement among five strong annotator models (GPT-5, Claude Sonnet 4.6, Qwen3.5 Plus, Grok-4, DeepSeek-V3.2), each performing rule-conditioned verification; instances lacking agreement are discarded. Human validation is conducted at two levels. Policy-level evaluation by pre-screened legal professionals on Prolific yields an average score of 4.3/5 across languages, with secondary verification by AI-regulation researchers at 4.6/5. Instance-level validation shows a striking contrast: human–ground-truth agreement is only 68.9% when annotators see the instance and rule alone, but rises to 94.3% when the model-generated explanation is provided. This result has two implications: policy-aware safety judgment is genuinely difficult even for humans, and the explanations used to supervise ML-Guard-7B accurately capture rule semantics. The final dataset contains 56K instances (34K training, 22K evaluation).
A limitation worth noting: instance-level human validation samples only 50 instances per language, and fr-CA uses French annotators as proxies due to annotator scarcity; the reported agreement figures should be read with this sample size in mind.
ML-Guard
ML-Guard is fine-tuned from Fast_dLLM_v2 base models (1.5B and 7B), making it, per the authors, an early application of dLLM architectures to guardrails. Parallel diffusion-based generation suits structured, moderately long outputs such as violation judgments with rationales.
ML-Guard-1.5B performs fast binary safe/unsafe classification for latency-sensitive deployment. ML-Guard-7B additionally supports policy-conditioned compliance checking: given an optional SAFETY_POLICY and input, it emits a four-line structured output — global judgment, violation flag, violated rule(s), and an English rationale. Supervisory rationales are GPT-5-generated. Training includes multi-rule inputs (3–6 candidate rules) to test rule selection, and a staged update scheme in which 10% of rules are withheld initially and introduced later alongside ~5 instances per previously seen rule; the appendix reports near-identical performance on initial-stage versus second-stage rules (e.g., F1 of 0.97/0.90/0.61 vs. 0.97/0.90/0.61 on Seed/Refined/Response), indicating no catastrophic forgetting. For unseen policies, the model falls back to general hazard categories rather than enforcing ungrounded policy decisions. Training combines ML-Bench with 25K-sample subsets each of PolyGuardMix and Nemotron-Safety-Guard-Dataset-v3 (unsafe:safe ratio 2:3).
Experimental results
Evaluation covers ML-Bench plus six existing benchmarks (PGP, XSafety, RTP-LX, MultiJail, CSRT, Nemotron) against 11 baselines including Llama Guard 3/4, DuoGuard, PolyGuard-Qwen, Nemotron-8B, Qwen3Guard-Gen (0.6B/4B/8B), gpt-oss-safeguard-20B, and OpenAI's omni-moderation.
On ML-Bench, baseline performance collapses sharply, which is itself a substantive finding about existing guardrails:
| Model |
F1 (Seed) |
F1 (Refined) |
F1 (Response) |
Attack Acc |
| Llama-Guard-3-8B |
0.49 |
0.06 |
0.10 |
0.03 |
| PolyGuard-Qwen |
0.72 |
0.34 |
0.33 |
0.07 |
| gpt-oss-safeguard-20B |
0.85 |
0.84 |
0.61 |
0.44 |
| ML-Guard-1.5B |
0.85 |
0.70 |
0.35 |
0.50 |
| ML-Guard-7B |
0.97 |
0.90 |
0.61 |
0.92 |
Several observations follow. Most category-based guardrails achieve F1 below 0.5 on refined queries and below 0.2 on responses, indicating that translation-derived, taxonomy-based training does not transfer to policy-grounded borderline content. gpt-oss-safeguard-20B is competitive but pays a precision cost (FPR 0.27 on seed queries) and extreme latency (~13.3 s/sample on average, roughly 9× ML-Guard-7B's 1.46 s). ML-Guard-7B's attack-enhanced accuracy of 0.92 versus 0.44 for the strongest baseline indicates substantial robustness gains attributable to training on adversarially enhanced policy-grounded data.
On existing benchmarks, ML-Guard-7B attains the best average F1 (0.81) despite never being trained on those taxonomies directly, with top scores of 0.92 on both Nemotron query and response splits; ML-Guard-1.5B achieves the best XSafety score (0.48) and second-best average (0.79). Notably, ablations show ML-Guard trained on ML-Bench alone still reaches 0.77 average F1 on external benchmarks, suggesting policy-grounded risk definitions subsume many general hazards rather than overfitting to one distribution. Policy conditioning provides consistent gains over the no-policy variant of ML-Guard-7B (e.g., Response F1 0.61 vs. 0.53; attack accuracy 0.92 vs. 0.90), while the no-policy variant remains strong, indicating internalized general safety knowledge.
Rule-violation prediction achieves F1 of 0.94 (seed) and 0.87 (refined) on ML-Bench and 0.85/0.92 on PGP/Nemotron subsets, closely tracking binary classification behavior. Rationale quality, scored by GPT-5 on a 0–5 scale, averages 4.11 (seed), 4.02 (refined), 3.37 (response), 4.13 (PGP), and 4.20 (Nemotron) — all above the correctness threshold of 3. Under multi-rule inputs (3–6 candidates), the model maintains high F1 on seed/refined settings and zero FPR on responses, though response-level F1 drops to 0.60.
Efficiency comparisons against a fine-tuned autoregressive Qwen2.5-7B under identical training conditions show the diffusion architecture delivers approximately 1.5× lower per-token latency and 1.5× higher throughput, supporting the claim that parallel generation benefits structured guardrail outputs.
Limitations and open questions
The paper concedes several constraints. Ground-truth labels are ultimately LLM-majority annotations rather than expert legal judgments; the 68.9% human agreement without explanations suggests residual label noise or genuine ambiguity in borderline cases, even if the 94.3% agreement with explanations mitigates this concern. Human validation is small-scale (50 instances/language, 10 annotators each) and relies on proxy annotators for fr-CA. Regulatory coverage is uneven: several languages inherit the EU AI Act as a stand-in framework rather than jurisdiction-native law, so "policy grounding" for Hindi, Dutch, German, Italian, and Spanish reflects EU regulation rather than distinct national statutes. Response-level performance remains the weakest setting for all models, including ML-Guard-7B (F1 0.61, rule-prediction recall 0.40), leaving open how well guardrails can distinguish intent within long, nuanced generations. Finally, rationale quality is scored by GPT-5, the same model family used to generate supervisory rationales, introducing potential evaluator bias that independent human scoring would resolve.
Conclusion
This paper contributes a regulation-native alternative to taxonomy-and-translation multilingual safety evaluation: ML-Bench grounds 56K instances in rules extracted directly from 17 regional AI regulations across 14 languages, and ML-Guard demonstrates that dLLM-based guardrails trained on such data achieve state-of-the-art results both on the new benchmark (F1 up to 0.97, attack accuracy 0.92) and on six prior benchmarks (average F1 0.81), while offering policy-conditioned rule identification, rationales, staged policy updates without forgetting, and favorable latency relative to larger policy-aware baselines. The central empirical claim — that policy-grounded native-language training data transfers broadly beyond its source distribution — is supported by the ML-Bench-only ablation, and the sharp failure of existing guardrails on refined and response-level inputs establishes ML-Bench as a demanding complement to current evaluation suites.
