Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domains

Abstract: Automatic evaluation metrics are central to the development of machine translation systems, yet their robustness under domain shift remains unclear. Most metrics are developed on the Workshop on Machine Translation (WMT) benchmarks, raising concerns about their robustness to unseen domains. Prior studies that analyze unseen domains vary translation systems, annotators, or evaluation conditions, confounding domain effects with human annotation noise. To address these biases, we introduce a systematic multi-annotator Cross-Domain Error-Span-Annotation dataset (CD-ESA), comprising 18.8k human error span annotations across three language pairs, where we fix annotators within each language pair and evaluate translations of the same six translation systems across one seen news domain and two unseen technical domains. Using this dataset, we first find that automatic metrics appear surprisingly robust to domain-shifts at the segment level (up to 0.69 agreement), but this robustness largely disappears once we account for human label variation. Averaging annotations increases inter-annotator agreement by up to +0.11. Metrics struggle on the unseen chemical domain compared to humans (inter-annotator agreement of 0.78-0.83 vs. 0.96). We recommend comparing metric-human agreement against inter-annotator agreement, rather than comparing raw metric-human agreement alone, when evaluating across different domains.

• The paper introduces the CD-ESA dataset, demonstrating that human annotation variability can mask metric failures under domain shift.
• It reveals that segment-level aggregation improves metric-human agreement, exposing biases in state-of-the-art quality estimation metrics.
• The study underscores the need for robust, cost-effective aggregation methods and LLM-based evaluators for reliable MT evaluation in technical domains.

Evaluating the Reliability of Machine Translation Metrics Across Domains

Introduction

The robustness of automatic machine translation (MT) evaluation metrics is a fundamental concern as research and industry increasingly deploy MT systems in specialized domains where annotated data and reference translations are scarce. The paper "Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domains" (2604.17393) introduces the Cross-Domain Error-Span-Annotation (CD-ESA) dataset, systematically evaluates state-of-the-art quality estimation (QE) metrics under domain shift, and exposes critical limitations in current evaluation paradigms. This essay provides a comprehensive and technical analysis of the paper's methodology, findings, and implications for future MT evaluation.

Motivation and Dataset Design

The central problem addressed is the conflation of metric robustness with variability in human annotation when transitioning from benchmark (WMT23-like) domains to low-resource or technical domains. Most prior work fails to disentangle domain shift effects from human label inconsistency due to changing annotator pools, translation systems, and annotation protocols across datasets.

To isolate domain shift effects, CD-ESA provides 18.8K high-density, cross-domain error-span annotations for three language pairs (en-de, en-zh, en-ko) and three domains (WMT23 news, Emea medical, proprietary PharmaChem technical logs). Importantly, all domains are annotated by fixed, domain-experienced annotator pools, and each translation is produced by the same set of MT systems, controlling for confounding variables.

Figure 1: Metrics may show unexpected behavior on unseen domains, reflecting domain familiarity over translation quality, while variability in human judgments (e.g., overlooking errors) can mask these effects.

Metric Robustness on Unseen Domains

Segment-Level Evaluation

Initial segment-level analysis of metric-human agreement reveals that supervised fine-tuned (SFT) metrics and modern LLM-as-a-judge approaches (e.g., GEMBA) maintain pairwise agreement with human judgments across both seen and unseen domains. Segment-level pairwise accuracy (e.g., acc$_\text{eq}$ up to 0.69) remains similar or, in some instances, even marginally higher on technical domains relative to news. Remedy's agreement with human annotations is often within 0.03–0.05 of human–human inter-annotator agreement, giving the impression of out-of-domain robustness.

System-Level Evaluation

At the system ranking level, differences emerge. While segment-level robustness persists, system-level soft pairwise accuracy (SPA) scores degrade in technical domains, especially PharmaChem, where the inter-annotator agreement reaches 0.96–1.00 but metrics lag at 0.78–0.87, revealing a substantive and systematic gap between machine and human evaluations once annotation noise is reduced.

The Masking Effect of Annotation Variability

A critical insight from the CD-ESA analysis is that variability inherent in human judgment—particularly at the segment level—systematically obscures metric failures on domain-shifted data. The paper demonstrates that simple aggregation strategies, such as annotation averaging and grouping $n$ segments for joint evaluation, significantly increase inter-annotator and metric–human agreement in both seen and unseen domains.

Figure 2: Pairwise accuracy gains from score aggregation on WMT23 and Emea. Bars report changes in $acc_{eq}$ when replacing single with averaged scores.

In German, averaging four annotators per segment results in a +0.11 improvement in inter-annotator accuracy on WMT23 and +0.10 on Emea. Averaging removes domain-independent human noise, sharpening the signal and revealing true metric weaknesses, particularly on out-of-domain data.

Figure 3: Pairwise accuracy for grouping segments ($n$) and its effect on both metric–human and inter-annotator accuracy—grouping exposes metric failures masked at $n = 1$.

Systematic Biases of MT Metrics Under Domain Shift

The paper identifies and characterizes two robust modes of bias that affect leading QE metrics under domain shift:

Over-preference for Open-Source Systems

On technical domains such as PharmaChem and Emea, SFT metrics systematically overrank open-source and in-domain fine-tuned models (e.g., X-ALMA) relative to human judgments, while underranking proprietary systems (e.g., DeepL, Azure). In Emea, SFT metrics and LLM-based rankers invert the human orderings, a contradictory phenomenon not driven by label noise but by misalignment between metric representations and domain-specific adequacy.

Figure 4: Visualization of the ranking of three MT systems (X-ALMA, Azure, DeepL), with human and metric rankings diverging under domain shift.

Systematic Underestimation of Translation Quality

Across all language pairs (en-de, en-zh, en-ko), SFT metrics consistently underestimate average translation quality in the most domain-shifted setting (PharmaChem) by up to 0.45 normalized z-score points compared to human annotators. Remedy exhibits inconsistent directionality, while LLM-as-a-judge approaches are markedly more domain-agnostic.

Figure 5: Average differences of normalized human and metric scores across domains. Negative values reflect metric pessimism (underestimation) in technical domains.

Error span analysis demonstrates that SFT metrics over-predict errors on PharmaChem, frequently marking large portions of sentences as erroneous even where human annotators do not, a direct consequence of over-indexing on domain cues rather than content fidelity.

Practical Recommendations and Theoretical Consequences

The findings strongly recommend that evaluation of metric robustness under domain shift must be contextualized relative to inter-annotator agreement, not by absolute metric–human agreement alone. Without mitigating human annotation noise, domain-dependent metric weaknesses will continue to go unnoticed, compromising model selection and deployment especially for industrial or life-critical applications.

The study underscores the necessity for adoption of cost-effective aggregation schemes, such as segment grouping, which balance annotation cost with robust error surface estimation. It further demonstrates that LLM-based evaluation is less susceptible to systematic biases in the absence of contamination, suggesting that LLM-as-a-judge may currently provide the most stable and reliable out-of-domain MT assessment protocol.

Implications for Future AI Research

The demonstrated domain sensitivity of representation-based MT metrics has profound implications for evaluation transparency and trustworthy deployment. Embedding analysis shows that learned representations from fine-tuned metrics are predominantly clustered by domain and not by human-perceived quality, revealing deep limitations in their capacity for generalization. As research transitions toward models and metrics with less reliance on static domain-specific features, further investigation into domain-adaptive training and robust prompting protocols will be critical.

The practical necessity of robust, cost-effective annotation strategies is highlighted, including short-segment prioritization, re-use, and group-based aggregation.

Conclusion

This paper provides a rigorous and methodologically controlled analysis of metric reliability under domain shift in MT evaluation. It establishes that human annotation variability can systematically mask metric deficiencies; true metric–human misalignments in out-of-domain deployment are both measurable and significant. The research advocates for inter-annotator agreement baselines in evaluation reporting, greater emphasis on aggregation strategies for reducing noise, and prioritization of LLM-based evaluators for cross-domain scenarios. Addressing these recommendations will be instrumental for the creation and trustworthy assessment of future generalist MT systems and their evaluation metrics.