COMET-polycand: Comparative MT Evaluation

• The paper introduces COMET-polycand as a multiple-candidate MT evaluation metric that scores a translation by comparing it with alternative outputs rather than in isolation.
• It extends the traditional COMET architecture by separately encoding extra candidate translations and using learned pairwise comparison features to gauge quality.
• Empirical results demonstrate that the inclusion of extra candidates significantly boosts performance in detecting undertranslation and other MT errors, aligning closer with human judgments.

Searching arXiv for COMET-polycand and related COMET metric papers. COMET-polycand is a machine-translation evaluation metric that extends COMET by scoring a translation in the context of other translations of the same source sentence rather than in isolation. It is introduced as the “multiple-candidate” extension of COMET in “COMET-poly: Machine Translation Metric Grounded in Other Candidates” (Züfle et al., 25 Aug 2025). The central premise is that standard automatic metrics typically estimate quality from the source sentence and a single hypothesis, whereas human annotators, especially in benchmark settings such as WMT, often judge one system output against alternative outputs for the same source. COMET-polycand operationalizes that comparative setting by encoding the main candidate together with one or more additional candidate translations and using learned pairwise comparison features to predict quality (Züfle et al., 25 Aug 2025).

1. Problem formulation and motivation

The motivation for COMET-polycand is a mismatch between the input structure of standard MT metrics and the context in which human judgments are commonly formed. The paper states that standard MT metrics such as COMET usually score a single hypothesis $t$ given the source $s$ and sometimes a reference $r$, formalized as $f(s,t)$, while human annotators often see multiple system outputs for the same source sentence and implicitly calibrate their judgments by comparing them side by side (Züfle et al., 25 Aug 2025).

This framing makes COMET-polycand a contextualized quality-estimation model rather than a single-hypothesis scorer. The extra candidate translations are intended to provide contrastive evidence: they can help expose undertranslation, wrong numbers, redundant explanations, refusals, and related failure modes when the compared candidate does not share the same error (Züfle et al., 25 Aug 2025). The paper further argues that such contextual grounding can improve consistency, help detect failure modes, and reduce sensitivity to out-of-domain examples (Züfle et al., 25 Aug 2025).

A practical consequence is that COMET-polycand is inherently tied to evaluation settings in which multiple translations of the same source sentence exist at scoring time. The paper therefore presents it as most useful for benchmark scoring, system comparison, or reranking rather than for evaluating a single isolated translation with no pool of alternatives (Züfle et al., 25 Aug 2025).

2. Architecture and feature construction

COMET-polycand retains COMET’s basic encoder-plus-regression-head structure and augments it with features derived from additional translations of the same source (Züfle et al., 25 Aug 2025). Baseline COMET is described as an encoder $e_{\theta_1}$ plus an MLP head $\mathrm{MLP}_{\theta_2}$, with scoring defined as

$$\mathrm{COMET}_\theta(s,t)=\mathrm{MLP}_{\theta_2}\big(g_{\theta_1}(s,t)\big),$$

where

$$g_{\theta_1}(s,t)=\langle s^e, t^e, |s^e-t^e|, s^e*t^e\rangle,$$

and $s^e=e_{\theta_1}(s)$, $t^e=e_{\theta_1}(t)$ (Züfle et al., 25 Aug 2025).

In COMET-polycand, the candidate translation $s$0 remains the primary object of evaluation, but the model additionally receives other translations $s$1 of the same source sentence (Züfle et al., 25 Aug 2025). These extra candidates are not concatenated into a single long input sequence. Instead, each additional translation is encoded separately and transformed into pairwise comparison features with respect to the main candidate:

$s$2

The complete feature vector is then

$s$3

which is passed to the regression head (Züfle et al., 25 Aug 2025).

This design preserves COMET’s source–candidate backbone while making the score explicitly dependent on inter-candidate contrasts. A plausible implication is that the model can learn relative quality cues that are unavailable to single-hypothesis metrics, because the feature space now includes direct evidence of similarity and discrepancy between competing outputs. The paper itself emphasizes that the alternatives are encoded separately and used through learned comparison features rather than through prompt-style concatenation (Züfle et al., 25 Aug 2025).

3. Training variants and supervision

The paper describes two training variants for COMET-polycand (Züfle et al., 25 Aug 2025). In the simplest form, the model predicts only the quality score of the main translation $s$4. In a “joint predictions” variant, it can simultaneously predict the quality of $s$5 and of the additional translation or translations, which reduces inference cost when both need to be scored (Züfle et al., 25 Aug 2025). The training objective is described as predicting multiple targets of the form $s$6, with the regression head trained using an average mean-squared error across all targets (Züfle et al., 25 Aug 2025).

The implementation details given in the paper state that COMET-poly models are trained on RoBERTa/xlm-roberta-large backbones with AdamW for 5 epochs, and the encoder is frozen for the first 30% of the first epoch (Züfle et al., 25 Aug 2025). The loss is the average mean-squared error across all supervised score targets (Züfle et al., 25 Aug 2025).

An optional extension incorporates the human score of the additional translation or translations directly into the input representation. In that case, the feature vector becomes

$s$7

The paper characterizes this as a practical “benchmark” setting in which alternative translations already have gold human scores from prior annotation rounds (Züfle et al., 25 Aug 2025). It also notes that this assumption is not always realistic, but when available it yields the strongest performance (Züfle et al., 25 Aug 2025).

4. Candidate selection and inference setting

The selection of extra candidates is a central part of COMET-polycand’s evaluation setup. During training, the paper states that additional translations are ensured to be different from the main translation $s$8, and that the number of additional candidates is fixed across examples (Züfle et al., 25 Aug 2025).

At inference time, the main selection strategy is to choose the “closest” additional translation using embedding similarity between candidate translations (Züfle et al., 25 Aug 2025). The paper also studies random additional candidates and settings with multiple additional candidates, up to five (Züfle et al., 25 Aug 2025). The nearest candidate is presented as an intuitive comparator because a translation similar to the one being scored should be the most informative contrast, but the ablation results show that even random extra candidates improve over the baseline COMET (Züfle et al., 25 Aug 2025).

The dependence on the candidate pool is one of the model’s defining properties. The paper explicitly notes that scores depend on which candidates are present, so changing the candidate pool changes the metric input and therefore effectively defines a different scoring setting (Züfle et al., 25 Aug 2025). This distinguishes COMET-polycand from conventional COMET usage, where the score for a translation is ordinarily a function of the source, the translation, and optionally a reference, but not of competing outputs.

5. Experimental protocol and empirical performance

The reported training data consist of WMT direct-assessment scores up to 2023, totaling about 600k segments (Züfle et al., 25 Aug 2025). The principal test set is WMT 2024, comprising 105 segments across eleven language pairs, evaluated at segment level (Züfle et al., 25 Aug 2025). The paper reports Pearson correlation, Kendall’s $s$9, and MAE, macro-averaged across languages (Züfle et al., 25 Aug 2025). When MQM-style error span annotations are used, they are converted to 0–100-like scores with

$r$0

to make them roughly compatible with direct-assessment scores (Züfle et al., 25 Aug 2025).

The central empirical result is that adding even one extra translation improves segment-level performance. In the reference-less setting, baseline COMET achieves 0.105 Pearson and 0.079 Kendall’s $r$1, while poly-cand with one additional candidate reaches 0.160 Pearson and 0.127 $r$2 (Züfle et al., 25 Aug 2025). The variant that also includes the gold score of the extra candidate reaches 0.267 Pearson and 0.207 $r$3 (Züfle et al., 25 Aug 2025).

In the reference-based setting, baseline COMET achieves 0.245 Pearson and 0.166 $r$4, poly-cand reaches 0.281 Pearson and 0.180 $r$5, and the variant with the additional candidate’s gold score reaches 0.374 Pearson and 0.243 $r$6 (Züfle et al., 25 Aug 2025).

The paper also studies the effect of the number of additional candidates. In the reference-less single-prediction setup, Kendall’s $r$7 rises from 0.127 with one extra candidate to 0.145 with two, and then stays around 0.127–0.130 for three to five (Züfle et al., 25 Aug 2025). With gold scores for the extra candidates, $r$8 reaches about 0.229–0.235 and is strongest around four to five extras (Züfle et al., 25 Aug 2025). The paper’s summary formulation is that including a single additional translation improves segment-level metric performance from 0.079 to 0.118–0.127 Kendall’s $r$9, with further gains when more translations are added (Züfle et al., 25 Aug 2025).

Setting	Baseline COMET	COMET-polycand
Reference-less Pearson	0.105	0.160
Reference-less Kendall’s $f(s,t)$0	0.079	0.127
Reference-based Pearson	0.245	0.281
Reference-based Kendall’s $f(s,t)$1	0.166	0.180

These results position COMET-polycand as an explicitly comparative metric whose strongest gains occur when informative alternatives are available and when additional human scores can also be supplied (Züfle et al., 25 Aug 2025).

6. Ablations, error analysis, and interpretation

The ablation studies in the paper are important for understanding what COMET-polycand is learning (Züfle et al., 25 Aug 2025). First, the model is shown to be robust to candidate choice in the sense that even random additional candidates improve over baseline COMET, though less than carefully chosen close candidates (Züfle et al., 25 Aug 2025). Second, the paper reports a monotonic relationship in the similarity analysis: the closer the additional candidate is to the main translation, the better the performance (Züfle et al., 25 Aug 2025).

Manual inspection of large-error cases provides the paper’s qualitative explanation for these gains. Extra candidates are reported to be especially useful for failure modes such as undertranslation, wrong numbers, redundant explanations, and refusals (Züfle et al., 25 Aug 2025). In such cases, another candidate can function as a quasi-reference by not sharing the same defect, thereby helping the metric detect that the main translation is poor (Züfle et al., 25 Aug 2025).

The paper also reports that the benefits are largest when the evaluated translation itself is low quality, whereas the improvement is smaller for high-quality translations (Züfle et al., 25 Aug 2025). This suggests that alternative translations contribute most when they expose deficiencies that would be difficult to infer from the source sentence alone. A plausible implication is that COMET-polycand behaves as a contrastive error detector more than as a uniformly better semantic similarity scorer.

7. Relation to other evaluators, use cases, and limitations

COMET-polycand is compared against two additional baselines beyond standard COMET: a $f(s,t)$2-nearest-neighbor approach and GEMBA, an LLM-based evaluator (Züfle et al., 25 Aug 2025). The $f(s,t)$3-NN baseline retrieves similar examples and averages their scores; in poly-cand mode, it averages COMET scores of retrieved candidate translations (Züfle et al., 25 Aug 2025). The paper reports that this baseline is much weaker in correlation, particularly for poly-ic, and argues that naïve aggregation of neighbors is inferior to learning how to combine contextual evidence (Züfle et al., 25 Aug 2025).

GEMBA is reported to be much larger and better than baseline COMET, yet it does not reliably benefit from additional candidates (Züfle et al., 25 Aug 2025). The paper states that poly-cand often outperforms GEMBA variants when extra context is injected, especially when the additional candidate is close to the main translation and its gold score is available (Züfle et al., 25 Aug 2025). This is presented as evidence that the utility of extra candidates depends on the evaluation architecture: COMET’s learned representation of source–translation relations appears to exploit candidate context better than prompt-only LLM judging in the reported experiments (Züfle et al., 25 Aug 2025).

The paper also describes several limitations. COMET-polycand is most applicable where multiple translations of the same source exist at evaluation time (Züfle et al., 25 Aug 2025). If additional human scores are used as features, those labels must be available, which may not hold in deployment (Züfle et al., 25 Aug 2025). There is also a dependence on the quality and bias of the underlying human-annotated data, since noisy or skewed prior annotations may propagate into the metric (Züfle et al., 25 Aug 2025). Finally, inference is more expensive than standard COMET because more text must be encoded, although much of the encoding can be cached, and the joint-prediction setup can reduce cost when scoring multiple translations together (Züfle et al., 25 Aug 2025).

Taken together, these properties place COMET-polycand in a specific methodological niche. It is not merely a stronger drop-in replacement for standard QE in all circumstances. Rather, it is a metric for comparative MT evaluation settings in which “knowing what else could have been said” is itself treated as signal (Züfle et al., 25 Aug 2025). This suggests a broader shift in MT evaluation from isolated scoring toward context-conditioned judgment models that more closely resemble human benchmarking practice.