WMT24++ Multilingual MT Benchmark

• WMT24++ is a multilingual English-to-target translation benchmark that extends WMT24 with 55 languages and preserves document context across literature, news, social, and speech domains.
• The benchmark incorporates human-written references and detailed post-edits, enabling precise evaluation of translation quality and dialect-specific variations.
• Evaluations reveal that frontier LLMs often outperform traditional MT providers under automatic metrics, though the results warrant further human evaluation for confirmation.

WMT24++ is a multilingual English-to-X machine translation benchmark created by extending the original WMT24 test set from a small set of WMT language pairs to 55 target languages and dialects through new human-written references and post-edits over the same 998 English source paragraphs. It is designed as an expanded multilingual evaluation layer on top of WMT24 rather than a new source corpus, and it preserves the original benchmark’s multi-domain structure across literary, news, social, and speech material. In its initial study, WMT24++ was used to compare conventional MT providers and frontier LLMs, with frontier LLMs ranked highest across all 55 languages under automatic metrics; the authors explicitly state that such findings should be confirmed with human evaluation (Deutsch et al., 18 Feb 2025).

1. Definition, lineage, and benchmark structure

WMT24++ reuses the same 998 English source paragraphs from WMT24 and adds new target-side annotations so that the benchmark can evaluate translation into 55 languages and dialects rather than the original WMT24 scope of 9 English-to-target pairs. The benchmark is paragraph-level rather than sentence-level, preserves some document context, and functions as an evaluation-only test set rather than a training corpus; the paper does not define train, development, or test splits (Deutsch et al., 18 Feb 2025).

The source inventory is organized into four domains inherited from WMT24, plus one canary segment used as a unique identifier for contamination detection.

Domain	Paragraphs	Documents
Literary	206	8
News	149	17
Social	531	34
Speech	111	111

Across all non-canary material, the benchmark contains 171 documents. This structure matters because WMT24++ was designed not simply to broaden language count, but to preserve a consistent English source inventory across domains and across all target languages, making cross-language comparison possible within a shared evaluation frame (Deutsch et al., 18 Feb 2025).

The core WMT24++ task is English→target-language translation only. This is an important boundary condition: the original benchmark does not define a bidirectional setup, even though later extensions derived from WMT24++ do (Deutsch et al., 18 Feb 2025).

2. Language coverage and dialect-sensitive design

WMT24++ contains 55 English-to-target language pairs. Of the 9 original WMT24 target languages, 8 received new post-edits—Czech, German, Spanish (Mexico), Hindi, Japanese, Russian, Ukrainian, and Mandarin (China)—while Icelandic was retained without a new post-edit because the vendor was unavailable. The benchmark also adds 46 new target languages and dialects, each with human reference plus post-edit: Arabic (Egypt), Arabic (Saudi Arabia), Bulgarian, Bengali (India), Catalan, Danish, Greek, Estonian, Farsi, Finnish, Filipino, French (Canada), French (France), Gujarati, Hebrew, Croatian, Hungarian, Indonesian, Italian, Kannada, Korean, Lithuanian, Latvian, Malayalam, Marathi, Dutch, Norwegian, Punjabi, Polish, Portuguese (Brazil), Portuguese (Portugal), Romanian, Slovak, Slovenian, Serbian, Swedish, Swahili (Kenya), Swahili (Tanzania), Tamil, Telugu, Thai, Turkish, Urdu, Vietnamese, Mandarin (Taiwan), and Zulu (Deutsch et al., 18 Feb 2025).

A distinctive design feature is the explicit inclusion of regional varieties rather than only abstract language labels. The paper highlights Arabic (ar_EG vs ar_SA), French (fr_CA vs fr_FR), Portuguese (pt_BR vs pt_PT), Swahili (sw_KE vs sw_TZ), and Mandarin (zh_CN vs zh_TW) as cases where MT systems may differ by regional standard, lexical choice, orthography, and script conventions. The paper does not provide a formal script typology, but it states that the language inventory spans Latin, Cyrillic, Arabic script, Han characters, Devanagari and other Indic scripts, Hangul, Thai, and Hebrew (Deutsch et al., 18 Feb 2025).

Dialect validation was handled manually rather than by automatic dialect detection. Approximately 20 randomly sampled segments per dialect were checked by native speakers of the target dialect for included regional varieties. The paper also reports a rejected case: human translations for bn_BD were collected but excluded because native-speaker checks found that they did not reliably represent the target regional variant. This aspect of the benchmark is central to its sociolinguistic design, because it treats language variation as an evaluation variable rather than a nuisance to be normalized away (Deutsch et al., 18 Feb 2025).

3. Reference creation, post-editing, and released assets

For the 46 newly added languages and dialects, WMT24++ collected new human-written reference translations created by professional translators who were fairly compensated according to their region. It then collected post-edits for all 46 newly added languages and dialects and for 8 original WMT24 target languages. The benchmark’s primary reference-based scoring uses the post-edits rather than the initial references (Deutsch et al., 18 Feb 2025).

The post-editing protocol deliberately prioritized translation quality over strict post-edit discipline. Translators were not required to minimally edit the existing translation, and the paper notes that some post-edits are light edits while others are effectively complete rewrites. This makes the benchmark better understood as containing paired human references and high-quality human revisions than as enforcing a narrow notion of minimal post-edit distance (Deutsch et al., 18 Feb 2025).

Although translation and post-editing were done at the segment or paragraph level, translators were given document context and, when available, the URL of the original source. This provision of context is part of the benchmark’s attempt to avoid the decontextualized sentence-level conditions that often characterize MT test sets. The release also includes source URLs where available and full-page screenshots of original source pages for multimodal research. The paper reports that 94% of original sources were available as screenshots, covering 85% of total segments, and that the screenshots are full-page, variable length, and have uniform width of 750 px (Deutsch et al., 18 Feb 2025).

Quality control combined dialect verification, inspection of low-quality translations according to QE metrics, human spot-checking, and analysis comparing references, post-edits, and MT outputs. The authors also identified 38 problematic source segments—mostly URLs, user tags, HTML fragments, emojis, miscellaneous non-translatable fragments, and the canary item—and removed them from the main analyses. As a result, the benchmark begins from 998 English source segments, but many reported system comparisons use 960 evaluated segments after filtering (Deutsch et al., 18 Feb 2025).

The dataset is released on Hugging Face at https://huggingface.co/datasets/[google](https://www.emergentmind.com/topics/service-weaver-google)/wmt24pp, and the screenshot collection is released separately at https://huggingface.co/datasets/google/wmt24pp-images (Deutsch et al., 18 Feb 2025).

4. Evaluation protocol and benchmark findings

The main WMT24++ paper evaluates only English→target directions. Its main automatic metrics are MetricX-24 and MetricX-24-QE, where MetricX-24 is reference-based and MetricX-24-QE is reference-free. The appendix additionally reports BLEU, chrF, XCOMET, XCOMET-QE, COMETKiwi-23, Gemini-DA, and Gemini-DA-QE. For MetricX figures, the paper reports negative MetricX and negative MetricX-QE scores, with higher values interpreted as better (Deutsch et al., 18 Feb 2025).

System rankings are presented as significance clusters rather than as a single uninterrupted total ordering. The significance testing procedure uses a one-sided permutation test with $\alpha = 0.05$. The benchmarked systems include DeepL Translate, Google Translate, Microsoft Translate, Yandex Translate, Aya 23, Claude 3.5 Sonnet, Command R+, Gemini-1.5 Pro, Gemini-1.5 Flash, GPT-4o, OpenAI o1, OpenAI o1-mini, and Tower-70B. LLM translation was run in 0-shot prompting with a prompt that asks for a translation suitable for a specific region; the appendix states that two different prompts were unintentionally used across model groups, which the authors identify as a benchmark caveat (Deutsch et al., 18 Feb 2025).

The headline benchmark result is that frontier LLMs are ranked highest across all 55 languages under automatic metrics. The three top systems are OpenAI o1, Claude 3.5, and Gemini-1.5 Pro, with reported average ranks across 55 languages of 1.5, 1.9, and 2.1, respectively. The authors also state that these three have very similar absolute metric scores. At the same time, the paper emphasizes that no full human evaluation has yet been conducted and that automatic metrics may be biased against human translations and are largely untested in many of the 55 languages (Deutsch et al., 18 Feb 2025).

Several benchmark-specific caveats are explicit. First, the authors warn that claims such as “LLMs outperform humans” or “LLMs are best in all languages” should be confirmed by human evaluation. Second, absolute metric scores are not considered reliable for cross-language comparison, because metric behavior may differ substantially across languages; the reported MetricX scores vary roughly from -2 to -6, but the authors do not interpret this as necessarily reflecting true quality differences. Third, post-editing generally improves the reference for all languages except ar_EG and ar_SA; the paper suggests that the drop for these Arabic variants likely reflects a shift from accidentally produced Modern Standard Arabic toward the intended regional variants, which metric behavior may penalize even if dialectal quality improved (Deutsch et al., 18 Feb 2025).

5. Extensions and derived benchmarks

WMT24++ has already served as the base for benchmark extensions that preserve its multilingual parallelism while adding new linguistic granularity. A prominent example is the Romansh extension in “Expanding the WMT24++ Benchmark with Rumantsch Grischun, Sursilvan, Sutsilvan, Surmiran, Puter, and Vallader” (Vamvas et al., 3 Sep 2025). That work adds six Romansh varieties—Rumantsch Grischun as a supra-regional variety, plus Sursilvan, Sutsilvan, Surmiran, Puter, and Vallader as regional varieties—by translating the existing WMT24++ German segments into each Romansh variety while preserving document and segment IDs. The resulting resource remains parallel not only across the six Romansh varieties but also with the other 55+ languages already covered by WMT24++. Its reference creation used a three-step workflow of translation, review, and revision, prohibited AI tools, and preserved document context. The paper reports that translation from Romansh into German is handled relatively well for all varieties, whereas translation into Romansh remains challenging (Vamvas et al., 3 Sep 2025).

A different form of derivation appears in “MTQE.en-he: Machine Translation Quality Estimation for English-Hebrew” (Rosenbaum et al., 6 Feb 2026). That dataset is not a reuse of original WMT24++ labels. Instead, it takes 959 English segments from English-Hebrew WMT24++, after removing 38 samples flagged with is_bad_source and one duplicate, generates Hebrew MT outputs with Google Translate, and collects new Direct Assessment scores from three human experts. The resulting benchmark is sentence-level QE data derived from WMT24++ source material rather than an alteration of WMT24++ itself. This establishes WMT24++ not only as a direct MT benchmark, but also as a source pool for newly annotated evaluation resources (Rosenbaum et al., 6 Feb 2026).

These developments show two distinct expansion paths. One path adds new languages or varieties while preserving WMT24++ segment alignment and domain structure. The other path repurposes the benchmark’s source inventory and language coverage to create new evaluation tasks, such as quality estimation. This suggests that WMT24++ is functioning as infrastructure rather than only as a one-off leaderboard.

6. Use in subsequent multilingual MT research and benchmark interpretation

Subsequent research has adopted WMT24++ as a broad automatic multilingual test bed. The “TranslateGemma Technical Report” evaluates TranslateGemma on WMT24++ across 55 English→target language pairs using MetricX24 and Comet22, and reports consistent gains over the corresponding Gemma 3 baselines at 27B, 12B, and 4B. On the aggregate WMT24++ results table, the 27B model improves from MetricX 4.037 and Comet22 83.1 for Gemma 3 to MetricX 3.087 and Comet22 84.4 for TranslateGemma; the paper further states that improvements are consistent across all 55 language pairs evaluated (Finkelstein et al., 13 Jan 2026).

WMT24++ has also been used to probe inference methodology rather than only model scale. “Unlocking Reasoning Capability on Machine Translation in LLMs” evaluates several reasoning-oriented LLMs on a 9-language subset of WMT24++—en-ar, en-cs, en-fa, en-fr, en-hi, en-ja, en-ko, en-ru, and en-zh—and reports that explicit generic reasoning degrades average XCOMET-XL in every tested model. The same paper then proposes a structured reasoning framework tailored to translation and reports that post-training with structured reasoning traces yields higher average XCOMET-XL than direct translation fine-tuning on that subset. This suggests that WMT24++ is sufficiently broad to act as a stress test for translation-specific inference strategies rather than only for static model comparisons (Rajaee et al., 16 Feb 2026).

The benchmark has also been used in work on benchmark localization and translation-pipeline quality. “Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets” evaluates translation pipelines on WMT24++ and FLORES for eight Eastern and Southern European languages, using methods such as USI and T-RANK. The paper reports higher reference-based COMET scores than baseline one-shot translation in multiple WMT24++ directions, including EN→UK, EN→SK, EN→RO, EN→BG, EN→TR, and EN→EL, and argues that translation quality of multilingual benchmarks materially affects downstream model assessment (Yukhymenko et al., 25 Feb 2026).

Taken together, these later uses position WMT24++ as a benchmark with two complementary roles. First, it is a multilingual MT evaluation set with unusually broad language and dialect coverage under a shared source inventory. Second, it is a reusable substrate for new evaluation problems, including dialect-sensitive extensions, quality-estimation benchmarks, and studies of evaluation reliability. Its central limitations remain those already identified in the original paper: only English→target directions in the core release, no full human evaluation yet, prompt inconsistency across some LLM evaluations, and the possibility that automatic metrics are biased or insufficiently validated in many of the included languages (Deutsch et al., 18 Feb 2025).