---
title: 'ACES: Translation Accuracy Challenge Set'
url: https://www.emergentmind.com/topics/aces
type: topic
---

# ACES: Translation Accuracy Challenge Set

ACES, the **Translation Accuracy ChallengE Set**, is a contrastive challenge-set benchmark for evaluating **segment-level machine translation metrics**. Its defining objective is not to estimate generic translation quality, but to test whether a metric reliably ranks a **good translation** above a minimally contrasted **incorrect translation** when the error belongs to a specific translation phenomenon. In the WMT 2023 metrics evaluation, ACES comprised **36,476 examples**, covered **146 language pairs**, and spanned **68 phenomena**, ranging from shallow word- and character-level perturbations to discourse-level and real-world-knowledge errors. Its importance lies in making visible the phenomenon-specific blind spots that ordinary aggregate evaluations can obscure [2311.01153].

## 1. Origin and rationale

ACES was introduced to evaluate MT metrics specifically on **translation accuracy** at the **segment level**, rather than only through overall correlation with human judgments on broad test sets. The benchmark is motivated by the observation that fluent MT output can still contain meaning-critical errors, including wrong numbers, modality shifts, factual inconsistencies, untranslated material, or source-dependent ambiguities that are especially consequential in settings such as **medical** or **legal** translation [2210.15615].

The original benchmark is grounded in the **MQM** branch for **Accuracy** errors, but extends it with additional categories such as **real-world knowledge** violations and **wrong language** outputs. Its construction is uniformly contrastive: each item includes a **source sentence**, a **reference translation**, and two hypotheses, one **good** and one **incorrect**. The “good” hypothesis is not required to be perfect in an absolute sense; it is required to be **better than** the incorrect one. This design makes ACES a benchmark for pairwise discrimination of accuracy failures rather than for absolute sentence scoring [2210.15615].

The original release assembled phenomena from multiple sources, including **FLORES-101**, **PAWS-X**, **XNLI**, **XTREME**, **WinoMT**, **MuCoW**, **Wino-X**, **Europarl ConcoDisco**, and **PIE**, using automatic generation, manual filtering, and manual construction. That provenance matters because it explains ACES’s breadth: the benchmark was deliberately built to cover both simple corruption-style failures and harder cases requiring source disambiguation, discourse interpretation, or commonsense reasoning [2210.15615].

## 2. Dataset design and phenomenon coverage

In the WMT 2023 study, ACES is explicitly described as a benchmark of breadth. The dataset contains **36,476 examples**, spans **68 phenomena**, and covers **146 language pairs**. Each example contains a **source sentence**, a **reference translation**, a **phenomenon label**, a **good** candidate translation, and an **incorrect** candidate translation containing the targeted error. Because the core task is to rank the good candidate above the bad one, ACES is suitable for both **reference-based** and **reference-free** metrics [2311.01153].

The **68 phenomena** are organized into **ten top-level categories**:

| Top-level category | Role in ACES |
|---|---|
| addition | accuracy |
| omission | accuracy |
| mistranslation | accuracy |
| untranslated | accuracy |
| do not translate errors | accuracy |
| overtranslation | accuracy |
| undertranslation | accuracy |
| real-world knowledge | accuracy |
| wrong language | accuracy |
| punctuation | small fluency category |

The benchmark’s shallow end includes **word- and character-level additions and omissions**, punctuation issues, untranslated tokens, and token substitutions. Its harder end includes **ambiguity**, **discourse connectives**, **pronouns**, **lexical overlap traps**, **overly literal idiom translations**, **ordering mismatches**, **entailment failures**, **synonym/antonym confusions**, and **hallucinations involving numbers, named entities, units, dates, and nonsense words**. It also includes **discourse-level** and **commonsense/world-knowledge** errors, **wrong language** cases in which the output is a good translation but in a related language rather than the target language, and **do not translate** cases in which content that should remain in the source language is incorrectly translated [2311.01153].

Within **mistranslation**, the WMT 2023 analysis further groups phenomena into **discourse**, **hallucination**, and **other**. The discourse group includes pronouns and discourse connectives; hallucination includes numbers, named entities, units, dates/times, and nonsense words; and the “other” group includes overly literal translations, ambiguities, and related meaning errors. This subdivision is central to ACES’s diagnostic role because it distinguishes source-sensitive semantic failures from surface perturbations and from lexically confounded hallucinations [2311.01153].

## 3. Evaluation formalism

ACES evaluates metrics through a **Kendall’s tau-like pairwise accuracy correlation**. For each contrastive example, a metric is scored according to whether it assigns a higher score to the **good** translation than to the **incorrect** translation. The score is

$$
T = \frac{\mathrm{concordant} - \mathrm{discordant}}{\mathrm{concordant} + \mathrm{discordant}},
$$

where “concordant” means that the metric prefers the good translation, and “discordant” means that it scores the good translation equal to or below the incorrect one. In the original benchmark formulation, **ties are treated as discordant** [2311.01153; 2210.15615].

These \(T\) scores are computed over sets of examples corresponding to a phenomenon, subcategory, or top-level category. To provide a single aggregate number, the WMT papers define **ACES-Score** as a weighted sum of the top-level category scores:

$$
\mathrm{ACES}
=
5T_{\mathrm{addition}}
+ 5T_{\mathrm{omission}}
+ 5T_{\mathrm{mistranslation}}
+ 1T_{\mathrm{untranslated}}
+ 1T_{\mathrm{do\ not\ translate}}
+ 5T_{\mathrm{overtranslation}}
+ 5T_{\mathrm{undertranslation}}
+ 1T_{\mathrm{real\text{-}world\ knowledge}}
+ 1T_{\mathrm{wrong\ language}}
+ 0.1T_{\mathrm{punctuation}}.
$$

The weighting reflects category severity or importance, and the resulting score ranges from **-29.1** to **29.1**. The authors repeatedly caution, however, that this aggregate is secondary to the benchmark’s principal purpose, which is the production of **diagnostic profiles** across phenomena and categories rather than a single leaderboard number [2311.01153].

The WMT 2023 paper also introduces targeted auxiliary analyses. For source-disambiguation tests it reports **correlation gain**, defined as the change in correlation when disambiguating source information is made available. For year-to-year comparison it reports simple **deltas**, defined as the **2023 score minus the 2022 score**. These analyses are designed to isolate specific metric properties—source sensitivity, lexical-overlap dependence, and the effect of multilingual representations—rather than only overall ranking performance [2311.01153].

## 4. Use in the WMT 2023 Metrics Shared Task

For WMT 2023, the authors benchmarked only those submitted metrics that produced **segment-level scores** and covered **all ACES language pairs and directions**. Metrics that produced only **system-level outputs** or failed to score all pairs were excluded. This yielded **33 metrics** for the main analysis: **10 baselines**, **11 reference-based submissions**, and **12 reference-free submissions** [2311.01153].

The analyzed baselines include **BLEU**, **chrF**, **BERTScore**, **BLEURT-20**, **COMET-22**, **COMETKiwi**, **MS-COMET-QE-22**, **YiSi-1**, **f200spBLEU**, and the random baseline **Random-sysname**. The WMT 2023 submissions analyzed include **MetricX-23 / MetricX-23-QE**, **XCOMET / XCOMET-QE / XCOMET-Ensemble**, **COMETKiwi-XL / XXL**, **KG-BERTScore**, **Cometoid22-wmt21/22/23**, **XLsim / XLsimQE**, **tokengram_F**, **partokengram_F**, **embed_llama**, and **GEMBA-MQM** [2311.01153].

The paper reports results at multiple granularities. First come the **ten top-level categories**; second, the **mistranslation** breakdown into **discourse**, **hallucination**, and **other**; third, targeted challenge-set analyses for **source context**, **lexical overlap**, and **multilingual embedding alignment**. For year-to-year comparisons, the authors use a subset of **33,817 examples** rather than the full dataset, because they identified preprocessing differences between the WMT 2022 and WMT 2023 score files for a small subset of examples, possibly related to handling of double quotes. All other analyses use the full **36,476-example** dataset. The paper also stresses that ACES currently targets **sentence/segment-level** evaluation and therefore does not measure **document-level** metric performance [2311.01153].

## 5. Diagnostic findings on metric behavior

The central empirical conclusion of the WMT 2023 study is that **no metric clearly dominated across ACES**. On aggregate ACES-Score, the strongest systems were **COMETKiwi** and **KG-BERTScore**, with **XCOMET-Ensemble** the strongest among reference-based metrics. The authors are explicit that this does not identify a universal winner, because category-level performance varies sharply and strong aggregate systems can still fail badly on particular phenomena [2311.01153].

A consistent broad pattern is that **neural metrics** outperform classic surface-overlap metrics, but remain uneven. The most difficult top-level categories include **addition**, **undertranslation**, **real-world knowledge**, and **wrong language**. Within mistranslation, **hallucination** phenomena are especially difficult, while **discourse** phenomena are generally easier than hallucinations. **BLEU** remains among the worst performers overall, ranking only above the random baseline in the paper’s main table. The paper also reports that some LLM-related submissions, notably **embed_llama** and **GEMBA-MQM**, perform only moderately or poorly, reinforcing the point that rich LLM representations alone do not guarantee robust MT metric behavior [2311.01153].

One of the strongest recurring findings is that **reference-free metrics are often as good as or better than reference-based metrics**, especially on **mistranslation**, **overtranslation**, **undertranslation**, and **real-world knowledge**. The authors connect this to insufficient use of the **source sentence** by many reference-based metrics. On ambiguity challenge sets where only the source disambiguates the correct output—such as ambiguous discourse connectives or gendered occupation names—reference-free metrics generally perform better. The paper identifies **XCOMET-Ensemble** as a partial exception among reference-based systems and suggests that jointly training reference-based and reference-free behavior may encourage stronger source use [2311.01153].

A second recurring result is that **surface overlap still exerts too much influence**, particularly for reference-based metrics. In challenge sets where the incorrect translation is lexically very similar to the reference and the correct translation is more paraphrastic, reference-based metrics deteriorate more steeply than reference-free metrics. On the **Mistranslation - Lexical Overlap** challenge set, reference-based metrics average only **0.05 ± 0.16** correlation, versus **0.27 ± 0.16** for reference-free metrics. The same pattern appears when the correct translation uses a **synonym** rather than copying the exact reference token: reference-based metrics are much stronger in the copied-reference condition than in the synonym condition [2311.01153].

A third major finding concerns **multilingual embeddings**. The paper asks whether language-agnostic multilingual representations can hurt MT evaluation by blurring distinctions that are essential for adequacy judgments. The stress tests here are **wrong-language** outputs and **untranslated full-sentence** outputs. Some 2023 metrics improve over 2022 on full-sentence untranslated cases, especially among reference-free metrics, but performance on **wrong language** remains poor for many systems. The broader conclusion is that multilingual embedding alignment can be harmful if not carefully designed, because a copied source sentence or a closely related but wrong-language output may still appear semantically plausible [2311.01153].

## 6. Incremental change, recommendations, and limitations

The WMT 2023 paper also studies **incremental performance** from WMT 2022 to WMT 2023, defined as **2023 score minus 2022 score** within recurring metric families. The pattern is mixed rather than monotonic. **KG-BERTScore** improves over its 2022 version. The **COMETKiwi** family is roughly flat or slightly worse depending on variant: **COMETKiwi-XL** drops slightly, while **COMETKiwi-XXL** is about the same as the previous year. The **XCOMET** family is heterogeneous: **XCOMET-Ensemble** improves substantially over **COMET-22**, **XCOMET-XL** improves slightly, but **XCOMET-XXL** degrades. The paper stresses that even when the overall ACES-Score improves, the gains are not uniform across categories; some categories still worsen [2311.01153].

A related scaling analysis uses **Cometoid22-wmt21/22/23**, trained on successively more pseudo-labeled data. The variant with the most training data, **Cometoid22-wmt23**, performs best on almost all top-level categories, with especially noticeable gains on **untranslated**, **do not translate**, **overtranslation**, and **wrong language**. The authors do not overgeneralize from this result, because the training data are pseudo-labeled rather than human-labeled, but they state that the trend suggests that **more training data can help metric robustness** [2311.01153].

The recommendations to metric developers are direct and largely unchanged from the previous year. First, there is **“No metric to rule them all.”** Since no single design wins across all ACES phenomena, developers should build **ensembles combining different metric families and design principles** rather than relying on a single architecture; naive majority voting is explicitly described as insufficient. Second, **the source matters**: metrics should pay more attention to the source sentence, because source-grounded evaluation is often more reliable than reference matching. Third, **surface overlap still prevails**, so developers should reduce lexical-overlap bias, for example through paraphrases during training and losses that discourage over-reliance on overlap. Fourth, metrics should handle **language-specific information** more carefully and use **multilingual embeddings/LLMs** with caution, because excessive language agnosticism can damage robustness on wrong-language and untranslated attacks [2311.01153].

For metric users, the paper’s implication is equally clear: there is no universally best MT metric. Metric choice depends on which translation failures matter for the application. If the priority is source-grounded adequacy, **source-sensitive** and often **reference-free** metrics may be preferable. If broad robustness is needed, **ensemble or hybrid approaches** may be safer than single-metric deployment. ACES’s diagnostic structure is specifically designed to make those tradeoffs visible. Its principal limitation is equally explicit: it is a **sentence/segment-level** benchmark and therefore does not capture **document-level** metric behavior, even though document context can be essential for translation evaluation in practice [2311.01153].

Source: https://www.emergentmind.com/topics/aces