ACES: Translation Accuracy Challenge Set
- ACES is a benchmark that defines translation accuracy by comparing a good translation against a minimally contrasted, error-prone candidate across 68 phenomena.
- The dataset employs a uniform contrastive design with source sentences, reference translations, and paired hypotheses to diagnose metric behavior.
- In WMT 2023, ACES exposed that no single MT metric excels universally, highlighting the strength of reference-free methods in addressing source-sensitive errors.
ACES, the Translation Accuracy ChallengE Set, is a contrastive challenge-set benchmark for evaluating segment-level machine translation metrics. Its defining objective is not to estimate generic translation quality, but to test whether a metric reliably ranks a good translation above a minimally contrasted incorrect translation when the error belongs to a specific translation phenomenon. In the WMT 2023 metrics evaluation, ACES comprised 36,476 examples, covered 146 language pairs, and spanned 68 phenomena, ranging from shallow word- and character-level perturbations to discourse-level and real-world-knowledge errors. Its importance lies in making visible the phenomenon-specific blind spots that ordinary aggregate evaluations can obscure (Amrhein et al., 2023).
1. Origin and rationale
ACES was introduced to evaluate MT metrics specifically on translation accuracy at the segment level, rather than only through overall correlation with human judgments on broad test sets. The benchmark is motivated by the observation that fluent MT output can still contain meaning-critical errors, including wrong numbers, modality shifts, factual inconsistencies, untranslated material, or source-dependent ambiguities that are especially consequential in settings such as medical or legal translation (Amrhein et al., 2022).
The original benchmark is grounded in the MQM branch for Accuracy errors, but extends it with additional categories such as real-world knowledge violations and wrong language outputs. Its construction is uniformly contrastive: each item includes a source sentence, a reference translation, and two hypotheses, one good and one incorrect. The “good” hypothesis is not required to be perfect in an absolute sense; it is required to be better than the incorrect one. This design makes ACES a benchmark for pairwise discrimination of accuracy failures rather than for absolute sentence scoring (Amrhein et al., 2022).
The original release assembled phenomena from multiple sources, including FLORES-101, PAWS-X, XNLI, XTREME, WinoMT, MuCoW, Wino-X, Europarl ConcoDisco, and PIE, using automatic generation, manual filtering, and manual construction. That provenance matters because it explains ACES’s breadth: the benchmark was deliberately built to cover both simple corruption-style failures and harder cases requiring source disambiguation, discourse interpretation, or commonsense reasoning (Amrhein et al., 2022).
2. Dataset design and phenomenon coverage
In the WMT 2023 study, ACES is explicitly described as a benchmark of breadth. The dataset contains 36,476 examples, spans 68 phenomena, and covers 146 language pairs. Each example contains a source sentence, a reference translation, a phenomenon label, a good candidate translation, and an incorrect candidate translation containing the targeted error. Because the core task is to rank the good candidate above the bad one, ACES is suitable for both reference-based and reference-free metrics (Amrhein et al., 2023).
The 68 phenomena are organized into ten top-level categories:
| Top-level category | Role in ACES |
|---|---|
| addition | accuracy |
| omission | accuracy |
| mistranslation | accuracy |
| untranslated | accuracy |
| do not translate errors | accuracy |
| overtranslation | accuracy |
| undertranslation | accuracy |
| real-world knowledge | accuracy |
| wrong language | accuracy |
| punctuation | small fluency category |
The benchmark’s shallow end includes word- and character-level additions and omissions, punctuation issues, untranslated tokens, and token substitutions. Its harder end includes ambiguity, discourse connectives, pronouns, lexical overlap traps, overly literal idiom translations, ordering mismatches, entailment failures, synonym/antonym confusions, and hallucinations involving numbers, named entities, units, dates, and nonsense words. It also includes discourse-level and commonsense/world-knowledge errors, wrong language cases in which the output is a good translation but in a related language rather than the target language, and do not translate cases in which content that should remain in the source language is incorrectly translated (Amrhein et al., 2023).
Within mistranslation, the WMT 2023 analysis further groups phenomena into discourse, hallucination, and other. The discourse group includes pronouns and discourse connectives; hallucination includes numbers, named entities, units, dates/times, and nonsense words; and the “other” group includes overly literal translations, ambiguities, and related meaning errors. This subdivision is central to ACES’s diagnostic role because it distinguishes source-sensitive semantic failures from surface perturbations and from lexically confounded hallucinations (Amrhein et al., 2023).
3. Evaluation formalism
ACES evaluates metrics through a Kendall’s tau-like pairwise accuracy correlation. For each contrastive example, a metric is scored according to whether it assigns a higher score to the good translation than to the incorrect translation. The score is
where “concordant” means that the metric prefers the good translation, and “discordant” means that it scores the good translation equal to or below the incorrect one. In the original benchmark formulation, ties are treated as discordant (Amrhein et al., 2023, Amrhein et al., 2022).
These scores are computed over sets of examples corresponding to a phenomenon, subcategory, or top-level category. To provide a single aggregate number, the WMT papers define ACES-Score as a weighted sum of the top-level category scores:
The weighting reflects category severity or importance, and the resulting score ranges from -29.1 to 29.1. The authors repeatedly caution, however, that this aggregate is secondary to the benchmark’s principal purpose, which is the production of diagnostic profiles across phenomena and categories rather than a single leaderboard number (Amrhein et al., 2023).
The WMT 2023 paper also introduces targeted auxiliary analyses. For source-disambiguation tests it reports correlation gain, defined as the change in correlation when disambiguating source information is made available. For year-to-year comparison it reports simple deltas, defined as the 2023 score minus the 2022 score. These analyses are designed to isolate specific metric properties—source sensitivity, lexical-overlap dependence, and the effect of multilingual representations—rather than only overall ranking performance (Amrhein et al., 2023).
4. Use in the WMT 2023 Metrics Shared Task
For WMT 2023, the authors benchmarked only those submitted metrics that produced segment-level scores and covered all ACES language pairs and directions. Metrics that produced only system-level outputs or failed to score all pairs were excluded. This yielded 33 metrics for the main analysis: 10 baselines, 11 reference-based submissions, and 12 reference-free submissions (Amrhein et al., 2023).
The analyzed baselines include BLEU, chrF, BERTScore, BLEURT-20, COMET-22, COMETKiwi, MS-COMET-QE-22, YiSi-1, f200spBLEU, and the random baseline Random-sysname. The WMT 2023 submissions analyzed include MetricX-23 / MetricX-23-QE, XCOMET / XCOMET-QE / XCOMET-Ensemble, COMETKiwi-XL / XXL, KG-BERTScore, Cometoid22-wmt21/22/23, XLsim / XLsimQE, tokengram_F, partokengram_F, embed_llama, and GEMBA-MQM (Amrhein et al., 2023).
The paper reports results at multiple granularities. First come the ten top-level categories; second, the mistranslation breakdown into discourse, hallucination, and other; third, targeted challenge-set analyses for source context, lexical overlap, and multilingual embedding alignment. For year-to-year comparisons, the authors use a subset of 33,817 examples rather than the full dataset, because they identified preprocessing differences between the WMT 2022 and WMT 2023 score files for a small subset of examples, possibly related to handling of double quotes. All other analyses use the full 36,476-example dataset. The paper also stresses that ACES currently targets sentence/segment-level evaluation and therefore does not measure document-level metric performance (Amrhein et al., 2023).
5. Diagnostic findings on metric behavior
The central empirical conclusion of the WMT 2023 study is that no metric clearly dominated across ACES. On aggregate ACES-Score, the strongest systems were COMETKiwi and KG-BERTScore, with XCOMET-Ensemble the strongest among reference-based metrics. The authors are explicit that this does not identify a universal winner, because category-level performance varies sharply and strong aggregate systems can still fail badly on particular phenomena (Amrhein et al., 2023).
A consistent broad pattern is that neural metrics outperform classic surface-overlap metrics, but remain uneven. The most difficult top-level categories include addition, undertranslation, real-world knowledge, and wrong language. Within mistranslation, hallucination phenomena are especially difficult, while discourse phenomena are generally easier than hallucinations. BLEU remains among the worst performers overall, ranking only above the random baseline in the paper’s main table. The paper also reports that some LLM-related submissions, notably embed_llama and GEMBA-MQM, perform only moderately or poorly, reinforcing the point that rich LLM representations alone do not guarantee robust MT metric behavior (Amrhein et al., 2023).
One of the strongest recurring findings is that reference-free metrics are often as good as or better than reference-based metrics, especially on mistranslation, overtranslation, undertranslation, and real-world knowledge. The authors connect this to insufficient use of the source sentence by many reference-based metrics. On ambiguity challenge sets where only the source disambiguates the correct output—such as ambiguous discourse connectives or gendered occupation names—reference-free metrics generally perform better. The paper identifies XCOMET-Ensemble as a partial exception among reference-based systems and suggests that jointly training reference-based and reference-free behavior may encourage stronger source use (Amrhein et al., 2023).
A second recurring result is that surface overlap still exerts too much influence, particularly for reference-based metrics. In challenge sets where the incorrect translation is lexically very similar to the reference and the correct translation is more paraphrastic, reference-based metrics deteriorate more steeply than reference-free metrics. On the Mistranslation - Lexical Overlap challenge set, reference-based metrics average only 0.05 ± 0.16 correlation, versus 0.27 ± 0.16 for reference-free metrics. The same pattern appears when the correct translation uses a synonym rather than copying the exact reference token: reference-based metrics are much stronger in the copied-reference condition than in the synonym condition (Amrhein et al., 2023).
A third major finding concerns multilingual embeddings. The paper asks whether language-agnostic multilingual representations can hurt MT evaluation by blurring distinctions that are essential for adequacy judgments. The stress tests here are wrong-language outputs and untranslated full-sentence outputs. Some 2023 metrics improve over 2022 on full-sentence untranslated cases, especially among reference-free metrics, but performance on wrong language remains poor for many systems. The broader conclusion is that multilingual embedding alignment can be harmful if not carefully designed, because a copied source sentence or a closely related but wrong-language output may still appear semantically plausible (Amrhein et al., 2023).
6. Incremental change, recommendations, and limitations
The WMT 2023 paper also studies incremental performance from WMT 2022 to WMT 2023, defined as 2023 score minus 2022 score within recurring metric families. The pattern is mixed rather than monotonic. KG-BERTScore improves over its 2022 version. The COMETKiwi family is roughly flat or slightly worse depending on variant: COMETKiwi-XL drops slightly, while COMETKiwi-XXL is about the same as the previous year. The XCOMET family is heterogeneous: XCOMET-Ensemble improves substantially over COMET-22, XCOMET-XL improves slightly, but XCOMET-XXL degrades. The paper stresses that even when the overall ACES-Score improves, the gains are not uniform across categories; some categories still worsen (Amrhein et al., 2023).
A related scaling analysis uses Cometoid22-wmt21/22/23, trained on successively more pseudo-labeled data. The variant with the most training data, Cometoid22-wmt23, performs best on almost all top-level categories, with especially noticeable gains on untranslated, do not translate, overtranslation, and wrong language. The authors do not overgeneralize from this result, because the training data are pseudo-labeled rather than human-labeled, but they state that the trend suggests that more training data can help metric robustness (Amrhein et al., 2023).
The recommendations to metric developers are direct and largely unchanged from the previous year. First, there is “No metric to rule them all.” Since no single design wins across all ACES phenomena, developers should build ensembles combining different metric families and design principles rather than relying on a single architecture; naive majority voting is explicitly described as insufficient. Second, the source matters: metrics should pay more attention to the source sentence, because source-grounded evaluation is often more reliable than reference matching. Third, surface overlap still prevails, so developers should reduce lexical-overlap bias, for example through paraphrases during training and losses that discourage over-reliance on overlap. Fourth, metrics should handle language-specific information more carefully and use multilingual embeddings/LLMs with caution, because excessive language agnosticism can damage robustness on wrong-language and untranslated attacks (Amrhein et al., 2023).
For metric users, the paper’s implication is equally clear: there is no universally best MT metric. Metric choice depends on which translation failures matter for the application. If the priority is source-grounded adequacy, source-sensitive and often reference-free metrics may be preferable. If broad robustness is needed, ensemble or hybrid approaches may be safer than single-metric deployment. ACES’s diagnostic structure is specifically designed to make those tradeoffs visible. Its principal limitation is equally explicit: it is a sentence/segment-level benchmark and therefore does not capture document-level metric behavior, even though document context can be essential for translation evaluation in practice (Amrhein et al., 2023).