COMET-polyic: Retrieval-Augmented MT Scoring

• COMET-polyic is a machine translation evaluation metric that enhances traditional COMET by conditioning candidate scoring on retrieved similar examples and human-annotated quality scores.
• It employs retrieval-based in-context conditioning using cosine similarity of multilingual embeddings (e.g., MiniLM) to integrate contextual evidence in translation assessment.
• Empirical results show improved segment-level Kendall’s tau and reduced MAE, indicating better alignment with human judgment and robust domain adaptation.

Searching arXiv for the target paper and closely related COMET metric work. COMET-polyic is a machine translation evaluation metric that extends COMET by grounding the score of a candidate translation in retrieved examples of similar source texts, their translations, and human-annotated quality scores. It is proposed as a response to a mismatch between automated evaluation and human assessment: conventional metrics typically score each translation candidate in isolation, whereas humans often calibrate judgments by comparison with alternatives or with prior experience on similar texts. In COMET-polyic, this calibration is made explicit through retrieval-based in-context conditioning, yielding higher segment-level agreement with human judgment in the reference-less setting, including an improvement in Kendall’s $\tau_b$ from $0.079$ for standard COMET to $0.116$ with one retrieved example (Züfle et al., 25 Aug 2025).

1. Motivation and evaluative context

The motivation for COMET-polyic is rooted in the observation that traditional automated MT metrics such as COMET assess translation quality by considering only the source sentence and a single translation candidate. The paper argues that humans, by contrast, evaluate translations in context: they may compare multiple outputs or rely on prior experience with analogous texts, a behavior linked in the paper to the sequence effect (Züfle et al., 25 Aug 2025).

Two consequences are emphasized. First, isolated scoring can produce less human-like, less calibrated assessments. Second, it can reduce adaptability to out-of-domain content, because the metric is not informed by annotated examples from the relevant domain. COMET-polyic addresses both issues by retrieving similar source sentences and exposing the metric to the associated translations and human labels during scoring (Züfle et al., 25 Aug 2025).

This framing distinguishes COMET-polyic from purely source–candidate metrics. It also suggests a broader interpretation of metric evaluation as a conditional inference problem in which the score for $(s,t)$ is not estimated solely from that pair, but from $(s,t)$ together with a compact, retrieval-selected evidence set. A plausible implication is that the metric behaves less like a static scorer and more like a dynamically calibrated estimator conditioned on a task-relevant local neighborhood.

2. Retrieval-based in-context conditioning

For a given evaluation pair consisting of a source sentence $s$ and translation $t$, COMET-polyic retrieves $k$ examples $(s_i, t_i, y_{s_i,t_i})$ from a knowledge base such as WMT annotation datasets. The retrieved examples are chosen because their source sentences are similar in meaning to the source sentence under evaluation (Züfle et al., 25 Aug 2025).

The retrieval mechanism described in the paper typically computes cosine similarity between an embedding of $s$ and embeddings of candidate $0.079$0 using an external multilingual sentence embedding model, with [all-MiniLM-L12-v2] given as an example. Each retrieved item contains three components: a source sentence $0.079$1, a translation $0.079$2, and a gold human label $0.079$3. These examples function as in-context calibration, allowing the model to observe how similar translations were previously judged by humans (Züfle et al., 25 Aug 2025).

The paper reports that retrieval using source embeddings yields the best results. Using COMET’s own embeddings for retrieval is slightly worse than using external models such as MiniLM, and cross-lingual sentence embeddings are described as the most robust for retrieval. This supports the design choice that retrieval quality is not merely an auxiliary implementation detail but part of the metric’s effective inductive bias (Züfle et al., 25 Aug 2025).

An important conceptual point is that the retrieved examples need not be translations of the exact same source sentence. COMET-polyic is built for the case where only one system output is available for the current source sentence, but a larger bank of annotated MT outputs exists elsewhere. That makes the method particularly relevant to single-system or single-output evaluation settings and to domain adaptation.

3. Model architecture and feature construction

COMET-polyic builds directly on the COMET architecture. In the baseline COMET formulation, the source $0.079$4 and translation $0.079$5 are encoded with a transformer encoder such as XLM-RoBERTa. The feature vector is the concatenation of the source embedding $0.079$6, the translation embedding $0.079$7, the element-wise absolute difference $0.079$8, and the Hadamard product $0.079$9. An MLP then predicts the quality score (Züfle et al., 25 Aug 2025).

The baseline formulation is given as

$0.116$0

with

$0.116$1

COMET-polyic augments this representation with one block per retrieved example. For each retrieved triplet $0.116$2, the appended feature block is

$0.116$3

The final input vector is then written as

$0.116$4

and this concatenated vector is fed to the MLP regression head to produce the predicted score

$0.116$5

The architecture therefore remains recognizably COMET-like: the encoder and regression head are preserved, while the principal modification is the injection of retrieval-conditioned evidence into the feature space. This is significant because the method is inspired by retrieval-based in-context learning, but it is not an LLM prompting scheme; it is a learned regression model whose conditioning variables include retrieved human-scored examples (Züfle et al., 25 Aug 2025).

4. Training protocol and inference regime

The model is trained in a supervised manner by minimizing mean squared error with respect to human scores. During training, retrieval is performed from the training data. At test time, retrieval comes from a held-out knowledge base (Züfle et al., 25 Aug 2025).

This train–test separation is central to the method’s formulation. The metric is not simply memorizing labels attached to the evaluation set; instead, it learns how to incorporate retrieved supervision signals as contextual evidence. The retrieved human score $0.116$6 is always part of the COMET-polyic input, unlike COMET-polycand, where gold scores for additional candidates are optional (Züfle et al., 25 Aug 2025).

The paper also reports a characteristic scaling behavior with the number of retrieved examples. Performance improves as in-context examples are added up to three, after which gains diminish or reverse, presumably as retrieved examples become less similar or relevant. This suggests that the marginal utility of retrieval is governed by example relevance rather than by raw context size. A plausible implication is that retrieval precision is more important than aggressive expansion of the evidence set (Züfle et al., 25 Aug 2025).

The domain-transfer behavior is also emphasized. COMET-polyic is reported to be robust to the domain, and using in-context examples from a new domain helps with domain transfer, especially for metrics such as MAE. In this sense, the model can be read as combining learned MT evaluation with a lightweight retrieval-time adaptation mechanism.

5. Empirical results and comparative performance

In the reference-less setting summarized from Tables 1 and 2 of the paper, standard COMET obtains segment-level Kendall’s $0.116$7 of $0.116$8, Pearson $0.116$9 of $(s,t)$0, and MAE of $(s,t)$1. With one retrieved example, COMET-polyic reaches Kendall’s $(s,t)$2 of $(s,t)$3, Pearson $(s,t)$4 of $(s,t)$5, and MAE of $(s,t)$6 (Züfle et al., 25 Aug 2025).

The paper presents these gains as substantial and consistent at the segment level. In particular, the increase in Kendall’s $(s,t)$7 from $(s,t)$8 to $(s,t)$9 indicates improved ranking agreement with human judgments. The MAE reduction from $(s,t)$0 to $(s,t)$1 further indicates that retrieval-conditioned scoring is not only better at pairwise ordering but also at absolute score estimation (Züfle et al., 25 Aug 2025).

COMET-polyic is also reported to outperform non-parametric $(s,t)$2-NN and to approach or match larger LLM-based evaluators such as GEMBA, while remaining much smaller and faster. When gold scores of additional retrieved translations are available, performance improves even more, although the paper notes that this is a rare scenario (Züfle et al., 25 Aug 2025).

These empirical results place COMET-polyic in an intermediate position between conventional parametric learned metrics and retrieval-only evaluators. It remains a compact learned model, but it leverages external annotated evidence at inference time. This suggests that some of the benefits commonly associated with larger in-context systems can be realized within a smaller COMET-style architecture when retrieval and feature construction are properly integrated.

6. Relationship to standard COMET and COMET-polycand

The paper contrasts three evaluation regimes. Standard COMET uses only $(s,t)$3 and $(s,t)$4, and optionally a reference translation $(s,t)$5, with no contextual information from other examples. COMET-polycand augments the current candidate with additional translations $(s,t)$6 of the same source sentence $(s,t)$7, and can optionally include their gold human scores if available. COMET-polyic instead uses translations of similar but not identical source sentences together with their human quality scores (Züfle et al., 25 Aug 2025).

This distinction has direct consequences for applicability. COMET-polycand is motivated by scenarios in which multiple system outputs exist for the same source sentence, such as system comparisons or reranking. COMET-polyic targets settings in which multiple candidates for the exact sentence are unavailable, but a large corpus of annotated MT outputs can be searched for analogous examples. The paper therefore associates COMET-polyic with quality estimation, low-resource settings, and domain adaptation (Züfle et al., 25 Aug 2025).

A frequent misconception is that the “poly” variants differ only in how many additional translations are provided. In fact, the conditioning structure is different. COMET-polycand relies on alternative translations of the same source sentence, whereas COMET-polyic relies on a retrieval layer over a broader corpus and always incorporates the human labels of the retrieved triplets. Another potential misconception is to equate COMET-polyic with a non-parametric nearest-neighbor method; the paper explicitly reports that the model outperforms non-parametric $(s,t)$8-NN, indicating that the retrieved examples are used as structured features within a learned regression pipeline rather than as direct vote-based predictors (Züfle et al., 25 Aug 2025).

The comparison also clarifies why COMET-polyic is more relevant to realistic general evaluation tasks. It allows dynamic, retrieval-enhanced context even in single-system and single-output settings, whereas COMET-polycand presupposes access to competing translations of the same source sentence.

7. Availability, deployment scenarios, and interpretive significance

The COMET-polyic models are released publicly under an open license. The paper lists pretrained models in a HuggingFace collection for poly-ic, including COMET-poly-ic1-wmt25 for one in-context example and COMET-poly-ic3-wmt25 for three in-context examples, together with code at https://github.com/zouharvi/COMET-multi-cand/ and documentation covering training, evaluation, and inference usage (Züfle et al., 25 Aug 2025).

The release matters because the method depends not only on learned weights but also on a retrieval pipeline and a knowledge base of annotated examples. Public model and code availability makes it possible to examine how retrieval source, similarity model, and domain-specific annotation pools affect calibration behavior in practice. This suggests that COMET-polyic is best understood not as a single monolithic metric, but as a retrieval-augmented evaluation framework whose behavior depends on both the base model and the evidence store it queries.

Within the scope of the paper, COMET-polyic is best suited to scenarios where rapid calibration to new data is needed, or where only one system output is available per source sentence. More broadly, it exemplifies a shift in MT evaluation away from strictly pointwise scoring and toward context-conditioned assessment grounded in prior human judgments.