Diagnosing Translated Benchmarks: An Automated Quality Assurance Study of the EU20 Benchmark Suite

Abstract: Machine-translated benchmark datasets reduce costs and offer scale, but noise, loss of structure, and uneven quality weaken confidence. What matters is not merely whether we can translate, but also whether we can measure and verify translation reliability at scale. We study translation quality in the EU20 benchmark suite, which comprises five established benchmarks translated into 20 languages, via a three-step automated quality assurance approach: (i) a structural corpus audit with targeted fixes; (ii) quality profiling using a neural metric (COMET, reference-free and reference-based) with translation service comparisons (DeepL / ChatGPT / Google); and (iii) an LLM-based span-level translation error landscape. Trends are consistent: datasets with lower COMET scores exhibit a higher share of accuracy/mistranslation errors at span level (notably HellaSwag; ARC is comparatively clean). Reference-based COMET on MMLU against human-edited samples points in the same direction. We release cleaned/corrected versions of the EU20 datasets, and code for reproducibility. In sum, automated quality assurance offers practical, scalable indicators that help prioritize review -- complementing, not replacing, human gold standards.

• The paper presents an audit pipeline that identifies structural inconsistencies and integrity issues in the EU20 multilingual benchmark suite.
• It employs xCOMET-XXL metrics to quantify translation quality differences across tasks and languages, revealing significant performance variations.
• The study integrates automated neural and LLM-based span diagnostics to provide actionable insights for scalable, reproducible benchmark curation.

Automated Quality Assurance for Translated Benchmarks: An In-Depth Analysis of EU20

Introduction and Context

The expansion of multilingual LLM benchmarks is crucial for robust, cross-lingual evaluation of NLU systems, especially across underrepresented European languages. Native data resources have high linguistic validity but are logistically and economically infeasible for extensive task and language coverage. Consequently, translated benchmarks—constructed via high-resource machine translation services—are increasingly adopted, but their diagnostic reliability remains a major concern, due to translation-induced noise, loss of structure, and uneven quality. "Diagnosing Translated Benchmarks: An Automated Quality Assurance Study of the EU20 Benchmark Suite" (2604.01957) systematically investigates these issues using EU20, which covers five English benchmarks (ARC, GSM8K, HellaSwag, MMLU, TruthfulQA) translated into 20 European languages.

Structural Diagnostics and Dataset Integrity

The first contribution is an audit pipeline addressing corpus structure and consistency across languages and tasks. Four critical integrity criteria are established: (A) answer-index alignment, (B) field completeness, (C) split/subset consistency, and (D) cross-language coverage. Systematic analysis uncovers typical failure modes concentrated in complex, context-dependent tasks when translated, such as the HellaSwag validation set, where DeepL often produces empty answer options due to context fragmentation. Additional structural flaws include inconsistent train/test split allocation and missing items in select language splits, with issues such as context-leakage having quantifiable but bounded effects on $k$-shot accuracy comparability.

A crucial practical outcome is the publication of repaired and completed EU20 splits, leveraging targeted re-translation and ensuring field-level provenance guarantees. This artifact enables reproducibility and regression analysis, providing a template for scalable cleaning and maintenance of multilingual benchmark suites.

Automated Translation Quality Estimation

A core aspect of the paper is the multi-axis evaluation of translation quality via advanced neural quality estimation (QE) metrics, primarily xCOMET-XXL, operating in both reference-free and reference-based modes.

Reference-Free Quality and Structural Trends

EU20's quality landscape, as measured by xCOMET-XXL, reveals pronounced dataset and language effects. For instance, standardized QA tasks (ARC, MMLU) yield consistently higher median xCOMET-XXL scores (0.97–0.98 for ARC; 0.83–0.94 for MMLU), while open-ended tasks such as HellaSwag display markedly lower medians (as low as 0.42) and higher score dispersion. The analysis demonstrates a robust negative correlation between target-side sentence length and predicted quality, with the strongest effect in HellaSwag (Spearman $\rho$ often $<-0.60$).

Figure 1: Reference-free xCOMET-XXL median quality scores, median target-side length, and length–quality correlation across 20 languages and 5 datasets.

Cross-System Comparison: EU20 vs. Okapi

The study performs rigorous, paired comparisons between EU20 (DeepL-based) and Okapi (ChatGPT-based) translations using reference-free xCOMET-XXL scores on ARC, HellaSwag, and MMLU across ten languages. EU20 exhibits a consistent, statistically significant quality advantage in ARC and MMLU, with median differences up to 0.054 (MMLU) and win-rates between 0.55–0.71. However, on HellaSwag, Okapi closes the gap or slightly outperforms EU20 in some Romance languages, which is attributed to the better handling of colloquial style in those settings.

Figure 2: Reference-free xCOMET-XXL median difference and win-rate for EU20 vs. Okapi across datasets and languages; positive values favor EU20.

Reference-Based Evaluation with Human-Edited Data

The study further validates the reference-free findings in a reference-based setting using human-edited Global-MMLU as gold data. EU20's translations are significantly closer (p<0.05, as established by paired bootstrapped CI) to the human reference than Okapi translations in 4 out of 5 languages—even when effect sizes are small (0.015–0.029 absolute difference). This supports the claim that the observed ref-free quality gaps are systematic and generalize to reference-based settings.

Figure 4: Median reference-based xCOMET-XXL differences ($\Delta_{\text{ref}}$) for EU20 vs. Okapi, relative to human-edited references on MMLU.

System Ranking on MMLU

A formal ranking experiment based on critical difference analysis over five languages reveals that EU20 statistically outperforms both Okapi and Global-MMLU translations in reference-free quality (average rank gap = 1.6 > CD), but the difference to Okapi is not statistically significant. This effect is robust across large item overlaps.

Figure 3: Critical-difference ranking diagram for MMLU, indicating statistically significant rank gaps among EU20, Okapi, and Global-MMLU translations.

Fine-Grained Span-Level Error Diagnosis

The paper augments sentence-level QE metrics with span-level diagnostics using LLM-as-a-judge span annotation via GEMBA-ESA, aggregated under MQM categories: Accuracy/Mistranslation (A+M), Fluency (F), and Other (O).

The translation error landscape uncovers that HellaSwag manifests the highest A+M error rates (up to 744 per 1000 samples in LV), with most errors judged as major. Cleanliness varies substantially across datasets, with ARC being the cleanest and HellaSwag the most problematic. Language effects are weaker than dataset effects; Germanic/Romance languages tend to be cleaner than Baltic/Balkan languages, but the primary signal is dictated by task design and translation complexity.

Figure 5: Error annotation rates per MQM super-category and clean segments, collapsed across severity, for each language and dataset.

Span-level results are congruent with the sentence-level quality landscape: high A+M spans systematically co-occur with low xCOMET-XXL medians, reinforcing that adequacy/mistranslation dominates quality degradation, followed by fluency and negligible “other” errors.

Implications, Limitations, and Prospects

The findings have practical implications for the conduct of multilingual LLM evaluation. Automated QA pipelines (neural QE + LLM-judge error annotation) provide scalable, diagnostic guidance for triaging and prioritizing human review, especially in high-noise, high-variance scenarios (e.g., long-form or context-sensitive tasks). However, the study exposes inherent metric/annotator biases and the persistent risk of missing deep semantic or culturally contingent errors in translated data, underscoring the necessity for domain-adapted, human-in-the-loop processes. Methodologically, the approach highlights the importance of joint structural and semantic checks and robust statistical procedures (paired bootstrap, critical difference tests) in benchmark curation.

Broader theoretical questions loom regarding the transferability of findings to non-European languages, transfer learning limits of existing QE models, and the ultimate role of automatic vs. expert adjudication in high-stakes NLU evaluation. Emerging LLM-based evaluators (neural and human preference-tuned) will likely advance the landscape, but reliance on proprietary or non-transparent models remains an unresolved risk factor for reproducibility and interpretability.

Conclusion

"Diagnosing Translated Benchmarks" establishes a rigorous and reproducible framework for automated QA of large-scale, machine-translated benchmarks. Through an overview of corpus audit, neural QE profiling, and span-level annotation, it identifies distinct quality regimes across tasks and languages, highlights systematic strengths of specific MT pipelines (notably EU20/DeepL over Okapi/ChatGPT in structured QA), and provides actionable insight for benchmark curation. The study confirms that statistical QA signals (xCOMET-XXL, GEMBA-ESA) are effective for triage but remain imperfect proxies for comprehensive, gold-standard evaluation; hybrid QA regimes will be essential as multilingual LLM research expands.