Translation Difficulty Estimation

• Translation difficulty estimation is the task of predicting how challenging a text is to translate, based on signals such as expected MT quality and human processing effort.
• Researchers utilize information-theoretic metrics, source-side confidence measures, and syntactic complexity analysis to quantify translation challenges.
• These estimation strategies help in creating discriminative MT benchmarks and inform evaluation protocols by identifying texts that are inherently more complex to translate.

Searching arXiv for recent and foundational papers on translation difficulty estimation and closely related MT/QE work. Searching for "Estimating Machine Translation Difficulty" and related benchmarks. Translation difficulty estimation is the task of predicting how hard a source text, segment, or translation instance will be for a translation system, a human translator, or an evaluation workflow. In the machine translation setting, one explicit formulation defines difficulty in terms of the expected quality of translations: a text is difficult if contemporary MT systems tend to produce lower-quality translations of it (Proietti et al., 13 Aug 2025). Closely related lines of work operationalize difficulty through cross-lingual information transfer, source-side confidence, human processing time, sample-size requirements for reliable translation quality evaluation, or shifts in pedagogical complexity such as CEFR level (Bugliarello et al., 2020, Sible et al., 30 Mar 2025, Lim et al., 2023, Gladkoff et al., 2021, Imperial et al., 3 Jun 2026). Across these formulations, difficulty is not absolute: it varies with the target language, translator or MT system, evaluation criterion, and mode of mediation such as written translation versus spoken interpreting (Proietti et al., 13 Aug 2025, Kunilovskaya, 12 Mar 2026).

1. Core definitions and problem formulations

A central recent definition treats translation difficulty estimation as the problem of predicting, from the source text alone, a difficulty score grounded in translation quality. For a source text $x$, translator $m$, and target language $l$, difficulty is defined through the quality score assigned to a translation of $x$ into $l$ produced by $m$; lower translation quality corresponds to higher difficulty (Proietti et al., 13 Aug 2025). The same work emphasizes that difficulty is relative to both the translator and the target language rather than an intrinsic universal property of the source.

An earlier information-theoretic perspective addresses directional MT difficulty rather than instance-level source difficulty. "Cross-mutual information" is proposed as an asymmetric metric intended to estimate translation difficulty while controlling for target-side generation difficulty. Its purpose is to separate the difficulty of translating from the difficulty of generating fluent text in the target language, a distinction that BLEU does not provide across different targets (Bugliarello et al., 2020).

Other work reframes difficulty in operational rather than intrinsic terms. In translation quality evaluation, the practical question becomes how much of a translated document must be inspected in order to make a reliable statement about the quality of the whole text, and how uncertain that statement is (Gladkoff et al., 2021). In simultaneous interpretation, difficulty can be cast as the likelihood that specific source terms will be left untranslated under time pressure, especially numbers and nouns such as proper names, acronyms, and domain-specific terms (Vogler et al., 2019). In human translation process research, difficulty is operationalized as cognitive processing cost, measured through source reading time, target reading time, and production duration (Lim et al., 2023).

These formulations are not interchangeable, but they are compatible. This suggests a broad taxonomy in which translation difficulty may denote at least four related objects: expected MT quality, cross-lingual transfer difficulty, human processing effort, and evaluation burden under uncertainty.

2. Operational signals of difficulty

Several strands of work define difficulty through measurable signals rather than direct human judgments.

One approach uses source-side confidence estimation. Instead of asking how confident a model is in the target sentence, it asks which source words make the translation model uncertain. The proposed score measures how sensitive the probability of the entire output sequence is to changes in a source embedding, using an $L_1$ norm over the gradient. Large values indicate low confidence and likely translation difficulty for that source word in context (Sible et al., 30 Mar 2025). The method is explicitly alignment-free and is proposed for interactive MT settings in which the user knows the source language but cannot judge the target output.

Another approach links difficulty to translationese. Translation task difficulty is decomposed into source-text comprehension difficulty and cross-lingual transfer difficulty, with predictors such as source syntactic complexity, mean dependency distance, number of clauses, MT surprisal, entropy of translation solutions, alignment strength, and teacher-forced pseudo-BLEU (Kunilovskaya, 12 Mar 2026). In that framework, translatedness is the segment-level probability of being a translation, produced by a linear SVM classifier, and difficulty features explain between 6% and 21% of variance in translatedness scores. The strongest model is written English→German with $R^2 = 0.21$, and the strongest predictors are source syntactic complexity and translation-solution entropy (Kunilovskaya, 12 Mar 2026).

A pedagogical-complexity perspective uses CEFR as an ordinal scale. ComplexityMT defines a robustness metric as the Spearman correlation between source CEFR level and reference-free MT quality, with a negative correlation indicating that higher CEFR texts are harder to translate (Imperial et al., 3 Jun 2026). The same benchmark studies whether translation preserves CEFR level under backtranslation, showing that translation quality and CEFR shift are statistically independent, with reported correlations near zero and $|\rho| \le 0.12$ (Imperial et al., 3 Jun 2026).

A dataset-intrinsic perspective for extremely low-resource MT introduces FRED Difficulty Metrics: Fertility Ratio $F$, Retrieval Proxy $m$0, Pre-training Exposure $m$1, and Corpus Diversity $m$2 (Chen et al., 26 Mar 2026). Here, difficulty is associated with weak tokenization coverage, low pre-training overlap, low train–test similarity, and low susceptibility to trivial retrieval. The paper reports that Retrieval Proxy is the strongest predictor of MT performance and argues that much of the variation in reported scores is explained by dataset structure rather than model capability alone (Chen et al., 26 Mar 2026).

Operationalization	Signal	Representative paper
Expected MT quality	Source-only prediction of translation quality	(Proietti et al., 13 Aug 2025)
Transfer difficulty	$m$3 after controlling for target-side generation	(Bugliarello et al., 2020)
Source-word uncertainty	Gradient sensitivity of target probability to source embeddings	(Sible et al., 30 Mar 2025)

3. Information-theoretic and model-based estimation

Information-theoretic methods play a prominent role in current formulations. In directional MT analysis, cross-mutual information is defined as

$m$4

where $m$5 is the cross-entropy of target sentences under a target-side LLM and $m$6 is the cross-entropy of the translation model (Bugliarello et al., 2020). The metric is asymmetric, which is precisely the intended behavior when comparing translation directions. In the controlled Europarl study reported there, translating from English is, on average, slightly easier by XMI than translating into English, even though BLEU often suggests the opposite (Bugliarello et al., 2020).

Surprisal-based predictors also recur in work on human and textual difficulty. In the translationese study, word surprisal is defined as

$m$7

and entropy of translation solutions as

$m$8

The paper further uses average segment surprisal from monolingual GPT-2 models to estimate source comprehension difficulty and NMT-conditioned average surprisal to estimate transfer difficulty (Kunilovskaya, 12 Mar 2026). In written translation, information-theoretic predictors often match or outperform structural predictors, while in spoken mode they offer no advantage (Kunilovskaya, 12 Mar 2026).

Human translation-process modeling adopts closely related quantities. Monolingual surprisal and translation surprisal are extracted from mGPT and NLLB-200 600M distilled, respectively, and evaluated as predictors of source reading time, target reading time, and production duration. Translation surprisal is reported as the single most successful predictor of production duration, while surprisal and attention are complementary predictors overall (Lim et al., 2023). This suggests that model-internal uncertainty is informative not only about MT failure risk but also about human translation effort.

The source-side confidence work provides a different model-based signal. For each source word $m$9, uncertainty is computed as

$l$0

The paper reports that $l$1 is the best norm for gradient reduction and that $l$2 is the best aggregation method over subwords (Sible et al., 30 Mar 2025). In mistranslation detection for English→German, the gradient-based method outperforms MGIZA-based and attention-based projection baselines on F1 and AUC-PR (Sible et al., 30 Mar 2025).

4. Quality estimation, confidence, and statistical uncertainty

A large portion of translation difficulty estimation is inherited from quality estimation. Sentence-level QE predicts a quality score for a source–translation pair without using a reference translation, typically as a proxy for how much post-editing would be needed or how good the translation is (Wu et al., 2021). In the WMT20 sentence-level QE setting, the target is the average of z-standardized Direct Assessment scores, and the low-resource scenario studied for en-de and en-zh uses only 100 QE training sentences (Wu et al., 2021).

The predictor-estimator framework remains a standard architecture in this space. A neural predictor learns token-level contextual features and produces QE feature vectors, which are consumed by a BiLSTM estimator for regression. In the low-resource transfer-learning study, predictor weights pretrained on other language pairs or extra parallel data are transferred and then combined through ridge regression or XGBoost ensembling. The best reported model reaches Pearson correlation $l$3, about 2.54 times higher than the baseline $l$4 on the de/zh test data (Wu et al., 2021). This suggests that translation difficulty signals are partially transferable across languages, but not uniformly.

Other QE work emphasizes confidence and annotator dependence. Sentence-level confidence estimation treats the task as binary classification over good versus needs work, and introduces sentence-level accuracy,

$l$5

as a simple measure of how often the MT system succeeds at the sentence level (Chelba et al., 2020). On the same English-Spanish MT outputs, SACC is reported as about $l$6 under non-expert annotation and about $l$7 under expert annotation, while the CE model shifts from roughly $l$8 Precision with $l$9–$x$0 Recall to $x$1 Precision with $x$2–$x$3 Recall (Chelba et al., 2020). The paper’s explicit conclusion is that annotator proficiency strongly affects apparent translation difficulty.

At the workflow level, statistical uncertainty is itself treated as a central object. Translation errors are modeled as Bernoulli events at sentence level, yielding a binomial model $x$4, confidence intervals of the form

$x$5

and sample-size formulas such as

$x$6

in the simplified case (Gladkoff et al., 2021). The paper supplements Bernoulli Statistical Distribution Modelling with Monte Carlo Sampling Analysis and argues that evaluation reliability depends strongly on sample size, target confidence level, and underlying error density. For normalized post-editing distance, it reports that confidence intervals become unstable below about 200 sentences, roughly 3500 words, and recommends samples above 4000 words for reasonably confident evaluation (Gladkoff et al., 2021). A plausible implication is that evaluation burden can itself serve as a practical index of translation difficulty.

5. Human translation and simultaneous interpreting

Human-centered work broadens translation difficulty estimation beyond MT quality. The largest process-oriented study in the provided set uses the CRITT Translation Process Research Database, covering 17 public studies, 13 language pairs, and 312 translators (Lim et al., 2023). Difficulty is measured through TrtS, TrtT, and Dur, corresponding to source reading time, target reading time, and production duration. Mixed-effects models with language pair and participant random effects show that translation surprisal is the strongest single predictor of production duration, while attention features provide complementary information (Lim et al., 2023).

Simultaneous interpretation introduces a stricter real-time variant of the problem. Predicting untranslated terminology treats difficulty as the probability that the interpreter will fail to render a source term. A source span is an untranslated term when it satisfies termhood, relevance, and interpreter coverage criteria: it consists only of numbers or nouns, a translation appears in the offline reference, and that translation does not appear in the interpreter output (Vogler et al., 2019). On a seven-talk English→Japanese subset of the NAIST Simultaneous Translation Corpus, untranslated term counts are reported as 1,256 for B-rank, 1,206 for A-rank, and 812 for S-rank interpreters, with lower rates for the more experienced interpreters (Vogler et al., 2019).

The modeling choice there is a sliding-window linear SVM tagger with features for elapsed time, word timing, word frequency, and lexical or syntactic characteristics (Vogler et al., 2019). Average precision rises from 45.4, 43.6, and 29.6 for the noun/# baseline to 58.9, 53.5, and 39.1 for the full SVM on B-, A-, and S-rank interpreters, respectively (Vogler et al., 2019). The most useful signals are reported to reflect speech rate, fatigue or temporal load, and lexical rarity.

Difficulty also appears in translationese analyses of spoken mediation. The translationese study finds that the relationship between task difficulty and translatedness differs by mode: in written translation, harder tasks tend to produce more translationese, whereas in spoken translation or interpreting the relationship is weaker and sometimes reversed, suggesting that interpreters may simplify or normalize output under pressure (Kunilovskaya, 12 Mar 2026). This complicates the common assumption that more difficult mediation always yields more literal output.

6. Evaluation, benchmark construction, and open issues

Recent work increasingly treats difficulty estimation as an infrastructure task for MT evaluation. The most direct formulation introduces Difficulty Estimation Correlation, defined as the average Kendall rank correlation $x$7 between predicted and ground-truth difficulty scores over languages and translators (Proietti et al., 13 Aug 2025). The key distinction is that the evaluation is Group-by-System rather than Group-by-Item: the goal is to rank source texts by how challenging they are for the same system, not to compare multiple translations of the same source.

Difficulty estimators are then used to construct harder subsets of benchmark data by selecting the $x$8 texts with the highest estimated difficulty,

$x$9

In the reported experiments, difficult subsets yield lower average human quality scores and fewer perfect translations than random selection, indicating that difficulty estimation can be used to build more discriminative MT benchmarks (Proietti et al., 13 Aug 2025). The same paper reports that dedicated source-only models, Sentinel-src, Sentinel-src-24, and Sentinel-src-25, outperform heuristic baselines such as text length, word rarity, and syntactic complexity, as well as LLM-as-a-judge methods (Proietti et al., 13 Aug 2025).

Difficulty-aware evaluation metrics push the idea further by weighting harder content more heavily. DA-BERTScore estimates token difficulty as

$l$0

then uses this weight inside BERTScore-style precision, recall, and F-score aggregation (Zhan et al., 2021). On WMT19 English↔German metrics shared tasks, the paper reports especially large gains in the top-30% competitive-system setting, where standard metrics struggle to distinguish systems (Zhan et al., 2021). Difficulty, in this case, is explicitly defined relative to the failures of a pool of MT systems rather than to linguistic complexity alone.

Several limitations recur across the literature. Difficulty is often model-specific or benchmark-specific rather than universal (Bugliarello et al., 2020, Proietti et al., 13 Aug 2025). Annotator standards materially change the apparent difficulty of the same translations (Chelba et al., 2020). Corpus properties such as train–test overlap, pre-training exposure, and tokenization coverage can make some low-resource tasks appear easier or harder than they are (Chen et al., 26 Mar 2026). Mode, direction, and language pair also matter: the translationese–difficulty link is strongest in written English→German, weaker in German→English, and qualitatively different in spoken mediation (Kunilovskaya, 12 Mar 2026). This suggests that translation difficulty estimation is best regarded as a family of closely related predictive problems rather than a single stable scalar property.

Taken together, the field has converged on a common premise: difficulty becomes measurable when it is tied to an operational consequence—lower MT quality, greater uncertainty, longer human processing time, higher omission risk, larger CEFR degradation, or greater evaluation sample requirements. The current research agenda is therefore not only to predict which texts are hard, but also to specify for whom, under which conditions, and with respect to which downstream objective (Proietti et al., 13 Aug 2025, Gladkoff et al., 2021).