Sentinel-src: MT Difficulty Prediction

• Sentinel-src models are specialized regressors that predict MT difficulty by estimating expected translation quality solely from the source text.
• They leverage XLM-RoBERTa large with dense output layers, trained with direct assessment and MQM fine-tuning to achieve state-of-the-art performance.
• The models enable efficient selection of challenging text subsets for MT evaluation, outperforming heuristic and LLM-based baselines.

Sentinel-src Models

Sentinel-src models are specialized learned regressors for automatic estimation of machine translation (MT) difficulty, where "difficulty" is operationalized as the expected translation quality for a given source text in the context of specific MT systems and language pairs. The Sentinel-src family represents the state of the art in source-only difficulty prediction, significantly outperforming heuristic, LLM-as-a-judge, and general-purpose text regressors and providing a scalable foundation for constructing discriminative MT evaluation datasets and benchmarks (Proietti et al., 13 Aug 2025).

1. Task Definition and Formalization

The core task addressed by Sentinel-src models is translation difficulty estimation. Given a source sentence $x$, a translator $m$ (machine or human), and a target language $l$, the difficulty $d_{m,l}(x)$ is defined as the expected human-assigned translation quality $q$ of $x$ translated by $m$ into $l$. Formally,

$d_{m,l}(x) = q,$

where $q \in [0, 100]$ derives from protocols such as Error Span Annotation (ESA). The modeling goal is to learn a function $m$0, operating only on the source (i.e., source-only or "pre-quality estimation"), such that $m$1 for unseen $m$2 and/or $m$3.

Crucially, this view subsumes and supersedes heuristic definitions of difficulty (e.g., sentence length, syntactic complexity, word rarity) by grounding the task in expected empirical translation quality (Proietti et al., 13 Aug 2025).

2. Evaluation Framework: Difficulty Estimation Correlation (DEC)

Sentinel-src models are evaluated under the Difficulty Estimation Correlation (DEC) framework, which generalizes the notion of difficulty estimation to support cross-system and cross-language calibration. For a set of $m$4 source-texts and corresponding difficulty labels $m$5, predictions $m$6, and using Kendall's $m$7 correlation,

$m$8

where the correlation is computed "Group-by-System": for each translator $m$9 and target language $l$0, the model's source-text-based predictions are compared to the true quality or difficulty ranking across test sources.

This ensures the metric reflects true sentence-level difficulty variability, as experienced by real MT systems or humans, and is not diluted by averaging across sources or translation models (Proietti et al., 13 Aug 2025).

3. Model Architecture and Training Methodology

Sentinel-src models are based on XLM-RoBERTa large, leveraging its multilingual capacity and deep bidirectional transformer layers for featurizing the source sentence:

• Backbone: XLM-RoBERTa large (approx. 550M parameters).
• Output Head: Two dense layers with ReLU activations on the [CLS] token, mapping to a scalar.
• Objective: Mean Squared Error (MSE) between prediction $l$1 and human score $l$2.

Training is conducted in two stages:

1. Direct Assessment (DA) pretraining: Using WMT 2017–2023 DA annotations (and WMT 2024 for later versions), batch size 16, learning rate $l$3, 3 epochs.
2. MQM fine-tuning: On WMT 2020, 2022, 2023 (and WMT 2024 for the most recent models), fine-tuned for one to two epochs.

Each source/difficulty pair is processed independently, regardless of repeated appearances, and all training is source-only. Two model checkpoints—Sentinel-src-24 and Sentinel-src-25—are released, differing in data vintage and target years (Proietti et al., 13 Aug 2025).

4. Comparative Performance Against Baselines

Sentinel-src models are directly compared to multiple families of baselines under identical DEC-based evaluation:

Model Type	DEC (WMT24 ESA Test)
Heuristic: Text Length	0.121
Heuristic: Syntax Height	0.080
Heuristic: Word Rarity	–0.040
PreCOMET_difficulty (XLM-R reg.)	0.153
PreCOMET_diversity	0.142
LLM Judge (GPT-4o src+lang)	0.080
LLM Judge (CommandA src+lang)	0.104
Artificial Crowd (XCOMET→QE)	0.177
Sentinel-src-24	0.182

Sentinel-src-24 achieves the highest correlation among all text-only models. While artificial-crowd approaches that require translating and evaluating with external quality estimation (QE) can match or slightly exceed DEC, they incur substantial computational overhead. Heuristics and LLM-judge baselines are significantly weaker, rarely outperforming mere sentence length (Proietti et al., 13 Aug 2025).

5. Practical Usage and Benchmark Construction

A major application of Sentinel-src is the automatic selection of difficult or challenging subsets for MT evaluation and benchmarking. Formally, given a candidate set $l$4 and a budget $l$5, one selects the top $l$6 sentences according to predicted difficulty:

$l$7

On the WMT 24 testset, the resulting subset has a 5-point lower mean human translation quality (on a 0–100 scale) and a 42.4% reduction in perfectly translated outputs, compared to a random sample. This demonstrates the model's capacity to efficiently winnow large text pools for segments that meaningfully challenge current MT systems (Proietti et al., 13 Aug 2025).

Operational recommendations include using Sentinel-src for initial ranking, optionally clustering on embeddings for domain diversity, and calibrating absolute thresholds based on random selection statistics.

6. Analysis: Strengths, Limitations, and Future Directions

Strengths:

• Substantial improvement over heuristics and LLM-based judges.
• Matches or approaches DA/QE-based "artificial crowd" pipelines at a fraction of computational cost.
• Source-only, enabling efficient deployment at web scale.

Limitations:

• Length Bias: All source-only models show strong negative correlation between predicted difficulty and sentence length ($l$8), potentially over-selecting long but easy sentences.
• Target-Language Agnosticism: Current Sentinel-src variants do not condition on target language, so the same sentence receives the same prediction regardless of translation direction.
• System Relativity: Difficulty labels are model-dependent and may not generalize to radically different architectures or annotation protocols.
• Diversity Trade-off: Strictly ranking by predicted difficulty densifies the test set, reducing domain and surface variety relative to random selection.

Proposed directions include:

• Language-conditioned difficulty estimation by augmenting the encoder inputs.
• Translator-aware or parameterized models to adapt to new or typologically distinct MT systems.
• Debiasing via regularization or adversarial objectives to minimize sentence length confounding.
• Multi-task objectives jointly balancing difficulty and diversity in constructed benchmarks.
• Extension to true human-centric measures of difficulty, such as translation effort or error rate independent of any single MT model (Proietti et al., 13 Aug 2025).

---

Sentinel-src models thus establish the reference standard for automatic, source-text-based translation difficulty estimation, enabling principled and scalable curation of challenging evaluation sets and supporting the next generation of discriminative MT research benchmarks (Proietti et al., 13 Aug 2025).