MetricAlign: MT Meta-Evaluation Protocol

• MetricAlign is a meta-evaluation dataset that systematically assesses machine translation metrics by comparing them with expert human ratings in literary translation.
• It operationalizes six translation dimensions—idiom translation, lexical ambiguity, tense consistency, zero-pronoun resolution, terminology localization, and cultural safety—to capture nuanced translation quality.
• The protocol employs statistical measures such as Spearman’s rho and R², revealing that LLM-driven metrics like AgentEval exhibit significantly higher alignment with expert judgments compared to traditional metrics.

MetricAlign is a meta-evaluation dataset and alignment protocol introduced to rigorously assess the correspondence between automatic machine translation (MT) metrics and expert human judgments in the context of Chinese-to-English web novel translation. Developed within the DITING framework, MetricAlign addresses the limitations of conventional MT metrics by providing a controlled dataset, precise error taxonomies, and statistical methodologies for benchmarking both traditional and LLM-based evaluation systems against human annotations spanning six genre-specific translational phenomena (Zhang et al., 10 Oct 2025).

1. Design Objectives and Scope

MetricAlign targets the persistent misalignment between surface-level MT metrics and the nuanced demands of literary translation, particularly in web novels. It functions as a meta-evaluation resource, enabling quantitative comparison of how well various automatic scoring methods reproduce the scalar and categorical judgments rendered by domain experts. The design spans six dimensions critical for narrative and cultural fidelity: idiom translation, lexical ambiguity, terminology localization, tense consistency, zero-pronoun resolution, and cultural safety. Each dimension is operationalized both as a set of human-assigned error labels and as a task-specific quality function $C_i$.

2. Dataset Construction

MetricAlign comprises 300 Chinese–English sentence pairs constructed through a two-stage process:

• Source Selection: Twelve Chinese sentences are uniformly sampled from the DiTing-Corpus, with two representative samples per DITING dimension, ensuring coverage across the six core translation phenomena.
• Model Outputs: Each source is translated by 25 systems representing open-source LLMs, closed-source LLMs, and machine translation-specialized models, yielding a matrix of 12 sentences × 25 systems = 300 translations.

This construction ensures fine-grained, phenomenon-specific benchmarking and immersion in genre-specific translation challenges. The seeded test set enables direct cross-metric comparison under controlled conditions.

3. Annotation Schema and Procedure

Expert annotation forms the reference gold standard for metric alignment. The annotation protocol involves three key elements:

3.1. Sub-metric Taxonomy

Each translation is scored along three axis-aligned sub-metrics tied to the relevant DITING dimension:

• One dimension-specific metric (e.g., Idiomatic Fidelity for idiom translation, Contextual Resolution for ambiguity),
• Two general or supporting metrics (e.g., Cultural Adaptation, Tone & Style, Information Integrity, Fluency, Naturalness, etc.).

Scoring is performed on a 0–2 ordinal scale (2 = high, 1 = medium, 0 = low), using anchors specified in the annotation guidelines and substantiated by calibration rounds.

3.2. Workflow and Calibration

Three domain experts (two professional literary translators, one advanced student) use Label Studio for labeling, ensuring both systematic reviewer assignments and change tracking. The protocol requires reading the original, expert reference, and MT output; referencing task-specific sub-metrics; assigning discrete scores; logging comments on ambiguous cases; and verifying totals prior to advancing. Weekly adjudication and pilot annotation rounds harmonize rater policy and interpretation.

3.3. Scalar Aggregation

The three sub-metric scores per translation are summed to yield a scalar human score:

$Q = r_{sp} + r_{g1} + r_{g2}$

where $Q$ ranges from 0 to 6, enabling direct correlation with real-valued automatic metric outputs.

4. Formal Evaluation Functions and Statistical Measures

The evaluative backbone of MetricAlign is grounded in mathematical formalization of both the translation phenomena and the metric–human comparison protocols:

• Dimension-specific Task Functions: For instance, lexical ambiguity is formalized as:

$$C_2(a_i, f_\theta(a_i)) = \mathbb{I}\left[ s(f_\theta(a_i)) = s^* \right], \quad s^* = \arg\max_{s_j \in \mathcal S(a_i)} P(s_j \mid S_{zh})$$

where $f_\theta$ is the translation model, $a_i$ the input, and $s^*$ the correct sense.

• Correlation Metrics:
  • Spearman’s rank correlation ($\rho$) between metric and human scores:

  $\rho = 1 - \frac{6 \sum_i d_i^2}{n(n^2-1)}$ - Variance explained ($R^2$):

  $$R^2 = 1 - \frac{\sum_i (h_i - m_i)^2}{\sum_i (h_i - \bar{h})^2}$$ - Cohen’s $\kappa$ for inter-annotator agreement:

  $\kappa = \frac{p_o - p_e}{1 - p_e}$

  where $p_o$ is observed agreement and $p_e$ is chance agreement.

These formal measures underwrite the meta-evaluation, enabling robust cross-metric alignment analysis.

5. Metric Alignment Protocol

For each automatic metric (e.g., BLEU, chrF, ROUGE, BLEURT, COMET, COMETkiwi-da, M-MAD, AgentEval), MetricAlign computes the correlation between its score vector $\{m_i\}$ and the human gold vector $\{h_i\}$ over 300 test samples:

• Primary criterion: Spearman’s $\rho$ quantifies monotonic association, aligning with rank order preference.
• Secondary criterion: $R^2$ measures the proportion of variance in human assessments explained by the metric.

This protocol is consistently applied across string-overlap-based, reference-aware neural, and deliberation-driven LLM metrics, ensuring comparability. A higher alignment score reflects greater faithfulness to human expert judgment.

6. Comparative Findings and Implications

Evaluation using MetricAlign reveals the following:

• Traditional string-overlap and reference-aware neural metrics (BLEU, BLEURT, COMET) achieve modest rank correlation with human assessments ($\rho \approx 0.47$) and explain approximately 22% of the variance.
• chrF and ROUGE show even lower alignment ($\rho \approx 0.31$, $R^2 < 11\%$).
• The multi-dimensional multi-agent debate metric M-MAD is negatively correlated ($\rho \approx -0.32$), indicating task/domain mismatch.
• The LLM-driven, reasoning-based AgentEval metric achieves substantially higher alignment, with the single-agent variant ($\text{AgentEval}_{DS\mbox{-}R1}$) reaching $\rho = 0.655$, $R^2 = 42.9\%$, and the debate-augmented variant ($\text{AgentEval}_{Debate\mbox{-}R1}$) slightly improving to $\rho = 0.669$, $R^2 = 44.8\%$.

Metric Type	Spearman’s $\rho$	$R^2$ (%)
BLEU, BLEURT, COMET	$\sim 0.47$	22
chrF, ROUGE	$\sim 0.31$	<11
M-MAD	$\sim -0.32$	—
AgentEval (1-agent)	0.655	42.9
AgentEval (debate)	0.669	44.8

The analysis confirms that no standard metric reliably tracks expert assessments for web novel translation; overlap-based and reference-based metrics underperform, while simulated expert deliberation via AgentEval provides the most human-aligned evaluations among those tested (Zhang et al., 10 Oct 2025).

A plausible implication is that genre-specialized, dimension-aware, multi-agent protocols offer a more faithful proxy for human literary assessment than conventional or even general neural MT metrics.

7. Significance in Translation Evaluation Methodology

MetricAlign introduces a replicable, richly-annotated benchmark for meta-evaluating translation metrics, with unique sensitivity to genre-specific narrative and cultural fidelity. The protocol’s multidimensionality, error taxonomies, and scalar quality treatments support granular diagnosis of metric–human gaps and substantiate the need for advanced evaluators such as AgentEval. By codifying both data and methodology, MetricAlign advances empirical scrutiny in automatic MT metric development and highlights the persistent gap between surface-form metrics and human literary standards (Zhang et al., 10 Oct 2025).