Establish semantic fidelity of translated multilingual benchmarks

Establish, through independent bilingual human evaluation, whether the translated StrongREJECT, JBB-Behaviors, XSTest, MATH-500, and AIME 2024 benchmark items preserve the task-relevant meaning, safety intent, refusal-relevant distinctions, mathematical constraints, and other information required for valid cross-lingual comparison.

Background

The paper distinguishes operational and structural validation from semantic fidelity: a translation can be parseable, preserve the required fields, and pass automated checks without preserving the source item’s meaning. The authors therefore design a bilingual human-evaluation protocol covering safety intent, benign-versus-unsafe distinctions, mathematical constraints, and other task-critical information across five benchmark resources and 11 target languages.

The proposed human evaluation is ongoing, and the paper reports that verified human assessments are not yet available. Consequently, the reported automatic fidelity signals and LLM-based judgments do not resolve whether the translated resources are semantically equivalent to their English counterparts.

References

Bilingual human evaluation remains necessary to establish translation fidelity, and verified human assessments are not yet available (Appendix~\ref{app:human_evaluation}).

— Scaling Multilingual Safety and Reasoning Evaluation through Reasoning-Guided Translation  (2610.20221 - Sahili et al., 14 Oct 2026) in Limitations; Appendix: Human Evaluation Protocol, Section “Reporting and availability”

Whether models hold different associations in each language or translation changes what an item expresses, our data cannot separate, since all models read the same translation.

— Lost in Translation: Why Lower Bias Scores Don't Mean Fairer Models  (2610.20479 - Kumar, 14 Oct 2026) in Section 4.2, “What the Lower Scores Do and Do Not Show” (paragraph: “The scores are not model-specific quirks”)