Sentence-Level Quality Estimation
- Sentence-level QE is a task that estimates the quality of machine-translated sentences by predicting continuous scores or categorical labels from source-hypothesis pairs.
- It employs various methodologies such as regression models, classification techniques, and ensemble approaches using pretrained multilingual encoders and intermediate representations.
- Its practical applications include optimizing post-editing workflows, selecting high-quality training data, and assessing translation adequacy using metrics like Spearman’s rank correlation.
Sentence-level Quality Estimation (QE) is the task of predicting, without access to a reference translation, a sentence-level quality value or label for a source sentence and its machine-translated output. In the literature, it is most commonly formulated as reference-less regression from a source–hypothesis pair to a continuous human score, but it also appears as ordinal or binary classification when the operational question is whether a translation is acceptable as-is or requires post-editing. Typical targets include Direct Assessment (DA), z-normalized DA, MQM-derived sentence scores, and HTER-style post-editing effort, with sentence-level QE positioned between word-level error tagging and document-level assessment in the broader QE taxonomy (Peter et al., 2023, Sindhujan et al., 2023, Chowdhury et al., 2021, Geng et al., 2023, Zhou et al., 2020).
1. Core task formulations
A standard formalization treats sentence-level QE as a real-valued scoring function over a source sentence and a translation hypothesis , written as , where the output is trained to approximate human quality judgments rather than class labels. In data-selection work, these continuous outputs are explicitly used as ranking scores for sentence pairs, with higher scores interpreted as better translations (Peter et al., 2023).
The dominant supervision regime in recent shared tasks is DA-based regression. In the WMT23 Sentence-Level Direct Assessment task for English→Indic translation, human translators rated MT outputs on a 0–100 scale; these scores were z-normalized per annotator and averaged, and the sentence-level QE system was trained to predict the resulting mean z-normalized DA score. This setting uses one real-valued prediction per sentence pair and evaluates how well that prediction tracks human judgments of adequacy and fluency (Sindhujan et al., 2023).
Alternative sentence-level targets coexist with DA. In WMT21 Task 2, the target is HTER, a real-valued estimate of post-editing effort in , so lower values indicate better translations and fewer edits. In WMT23 EN–DE MQM-based work, the target is a sentence-level MQM score derived from error counts and severities, with sentence-level QE again cast as scalar prediction rather than token classification (Chowdhury et al., 2021, Geng et al., 2023).
A parallel line of work reframes sentence-level QE as classification rather than regression. In the “quality classification” view, the key deployment question is whether a sentence is good enough to use as-is, and evaluation emphasizes recall at a fixed precision threshold rather than correlation with a continuous score. This formulation does not replace regression in the literature, but it reorients sentence-level QE toward accept/reject decisions and high-precision filtering (Zhou et al., 2020).
2. Data resources and supervision regimes
Sentence-level QE research depends heavily on benchmark design, because labels may represent human adequacy judgments, post-editing effort, ordinal usability levels, or task-specific acceptability decisions. The best-known multilingual resource in this set is MLQE-PE, which contains eleven language pairs and provides human labels for up to 10,000 translations per language pair in the formats of sentence-level direct assessments, post-editing effort, and word-level good/bad labels; it also includes post-edited sentences, article titles, and the neural MT models used to generate the translations (Fomicheva et al., 2020).
Several later datasets specialize sentence-level QE to particular domains or languages. WMT23 English→Indic DA data covers five language pairs—En–Gu, En–Hi, En–Mr, En–Ta, and En–Te—with roughly 7,000 training, 1,000 development, and 1,000 test sentence pairs per pair, all labeled with mean z-normalized DA scores from three professional translators. Domain-specific work extends this design further: Indic-Domain-QE provides English→Indic sentence-level DA labels across Healthcare, Legal, Tourism, and General, with aggregated train/test counts of 13,280/1,660 for Healthcare, 6,160/770 for Legal, 13,840/1,730 for Tourism, and 18,880/2,360 for General (Sindhujan et al., 2023, Gurav et al., 7 Mar 2026).
A different construction strategy appears in under-resourced English→Hebrew QE. There, the initial corpus contains about 14,000 manually ranked segments on a 1–5 scale, expands to about 200,000 segments through controlled grammatical perturbations and professionally translated anchors, and then grows further through adversarial 0-score mismatched pairs and additional synthetic perturbation. The label space is ordinal rather than continuous: first , later (Siani et al., 12 Mar 2026).
| Dataset or setting | Coverage | Sentence-level supervision |
|---|---|---|
| MLQE-PE | 11 language pairs; up to 10,000 translations per pair | direct assessments and post-editing effort |
| WMT23 English→Indic DA | En–Gu, En–Hi, En–Mr, En–Ta, En–Te | mean z-normalized DA score |
| Indic-Domain-QE | Healthcare, Legal, Tourism, General | DA score $0$–$100$, averaged |
| Semi-synthetic En→He QE | 14k manual core; larger augmented pools | ordinal quality scores $1$–$5$ or 0–1 |
These resources imply two distinct but compatible traditions. One emphasizes human DA-style continuous supervision for multilingual benchmarking; the other engineers large semi-synthetic corpora with controlled error distributions for low-resource conditions. This suggests that sentence-level QE has become as much a data-construction problem as a modeling problem.
3. Model families and representational strategies
One influential architecture is the predictor–estimator family. In the OpenKiwi-style formulation, a predictor is trained as a conditional target-side masked LLM over 2, and sentence-level QE is produced by an estimator that consumes QE feature vectors derived from the predictor’s hidden states. This architecture remains central in transfer-learning work for low-resource QE, where predictors are pretrained on different language pairs or on additional parallel corpora and then combined through sentence-level ensemble regression (Wu et al., 2021).
COMETKIWI extends this lineage with a COMET-style sentence-level head on top of multilingual encoders such as XLM-R, InfoXLM, or RemBERT. Its sentence-level branch uses a scalar mix over encoder layers, extracts the mixed [CLS] representation, and feeds it to a two-layer feed-forward network to predict a single quality score. The same framework can be trained jointly with word-level objectives, and that multi-task coupling is reported to improve sentence-level performance, especially with InfoXLM (Rei et al., 2022).
A simpler but widely used sentence-level pattern is MonoTransQuest. Here the source and target are concatenated as 3 source 4 target, the [CLS] embedding is used as the sentence-pair representation, and a regression head predicts the DA score with mean squared error. Encoder choice matters: in English→Indic WMT23 submission results, MonoTQ-InfoXLM-large was the strongest single model on most language pairs, while XLM-V was especially useful for some low-resource Indic pairs and XLM-R-large was clearly weaker (Sindhujan et al., 2023).
Sentence-level QE has also been implemented through ensembles of multilingual BERT regressors built from different input views. In WMT21 HTER prediction, mBERT models were trained on SRC–MT, MT-only, and MT–MT′ inputs, and their outputs were combined with a GradientBoostingRegressor. The architecture itself remained simple—a [CLS] representation followed by a linear layer and sigmoid—but input diversity supplied complementary adequacy and fluency signals (Chowdhury et al., 2021).
A distinct zero-shot line replaces learned QE heads with metric-like computations over multilingual contextual embeddings. When BERTScore is adapted from reference-based evaluation to QE by replacing the reference with the source sentence, token-level greedy matching produces frequent cross-lingual mismatches. To alleviate this, explicit cross-lingual patterns—word alignments and generation score—are added on top of multilingual similarity, yielding a zero-shot sentence-level QE method that remains label-free at QE training time (Zhou et al., 2020).
LLMs introduce yet another family of sentence-level QE methods. In low-resource English→Indic domain-specific QE, prompt-only zero-shot, few-shot, and guideline-anchored prompting were compared against parameter-efficient adaptation. The ALOPE framework attaches a lightweight regression head to a selected intermediate Transformer layer, fine-tunes LoRA or LoRMA adapters under 4-bit QLoRA, and predicts DA scores with mean squared error. Its central claim is representational rather than purely architectural: intermediate layers, especially around 5 and 6, are more useful for sentence-level QE than the final layer in these settings (Gurav et al., 7 Mar 2026).
4. Evaluation protocols and the relation to word-level evidence
The default evaluation regime for sentence-level QE is correlational. Spearman’s rank correlation and Pearson’s correlation are the standard metrics in recent DA settings, with Spearman often treated as primary because ranking better and worse translations is operationally important. In WMT23 English→Indic experiments, both metrics are computed per domain × language pair and macro-averaged, while in WMT22 QE shared-task systems Spearman is the main sentence-level criterion and Pearson is a secondary measure alongside MAE and RMSE (Sindhujan et al., 2023, Rei et al., 2022).
HTER-based sentence-level QE retains the same scalar-prediction setup but adds calibration-oriented metrics. For WMT21 HTER prediction, systems are evaluated with Pearson’s correlation, MAE, and RMSE, reflecting the fact that sentence-level predictions are sometimes used quantitatively as edit-cost estimates rather than only as rankings (Chowdhury et al., 2021).
Once sentence-level QE is reframed as acceptability classification, the evaluation landscape changes. Classification-based QE studies report F1 under DA-derived thresholds such as 7 or 8, and deployment-oriented work proposes 9, the maximum recall achievable subject to precision at least 0. In this view, a model that preserves the right side of a decision threshold can be practically useful even if its Pearson correlation degrades substantially under compression (Sun et al., 2021, Zhou et al., 2020).
Sentence-level QE also has a strong but nontrivial relation to word-level evidence. One strand argues that successful sentence-level QE models implicitly rely on token-level translation errors and that these can be recovered as rationales. Feature attribution over sentence-level XLM-R-based QE models produces token-level scores that align with human BAD labels, suggesting that sentence-level quality prediction and word-level error localization are partially shared representations rather than separate phenomena (Fomicheva et al., 2021).
5. Empirical patterns and practical uses
A recurring empirical result is that sentence-level QE is highly sensitive to domain, model family, and supervision regime. In English→Indic domain-specific experiments, prompt-only closed-weight LLMs were strong even in zero-shot settings, but prompt-only open-weight LLMs were fragile, especially in high-risk domains such as Legal and Healthcare. Intermediate-layer adaptation with ALOPE consistently improved sentence-level QE, with the best Spearman correlations typically coming from deep intermediate layers rather than the final layer; LoRA yielded the highest absolute correlations, while LoRMA smoothed layer sensitivity (Gurav et al., 7 Mar 2026).
Sentence-level QE is also used as a data-selection mechanism for MT training. One study shows that selecting the highest-quality sentence pairs with QE metrics can improve translation quality while reducing training size by half, and that QE-based filtering captures fine-grained translation deficiencies that noise-oriented filters miss. Another study, however, finds a clear negative result when sentence-level QE is used too aggressively: reducing WMT’23 web-crawled parallel data from roughly 296M sentence pairs to a top-quality ~1M subset via BLEURT-QE filtering led to very poor downstream performance, much worse than training on a similarly sized synthetic NewsPaLM sentence-level dataset. Together, these findings indicate that sentence-level QE is effective for moderate pruning but is not a universal substitute for high-quality synthetic data generation (Peter et al., 2023, Finkelstein et al., 2024).
In translation workflows, sentence-level QE has measurable process effects. In an English–Chinese MT post-editing study using a CAT tool with A/B/C sentence-level QE labels, post-editing time per word dropped from 1.27 s/word without QE to 0.95 s/word with QE, and the main effect of task type was significant. The interaction between QE and MT quality and the interaction between QE and translator expertise were both reported as not significant, while interview data showed that translators also used sentence-level QE for validation and double-checking; at the same time, inaccurate QE was reported as capable of hindering post-editing (Liu et al., 22 Jul 2025).
Sentence-level QE has additionally been treated as a downstream target for auxiliary pre-training. In “metric estimation,” models are first trained to predict reference-based automatic metric values without reference translations and then fine-tuned on human QE labels. In the reported English→German setting, TER-based pre-training improved correlation with human judgments from 19.8% to 22.8%, whereas pre-training on several other metrics did not outperform training from scratch (Zouhar et al., 2023).
6. Debates, low-resource directions, and broadening of the concept
One persistent debate concerns whether sentence-level QE should be optimized as regression or classification. Efficiency-oriented work argues that Pearson correlation over continuous DA scores can over-penalize models whose outputs still preserve the correct accept/reject boundary, and that smaller pruned models can retain most of their F1 while losing much more Pearson correlation. Quality-classification work reaches a similar conclusion from the workflow side: for users who need to know whether a sentence can be used as-is, recall at high precision is more interpretable than correlation with HTER or DA (Sun et al., 2021, Zhou et al., 2020).
Low-resource sentence-level QE has driven increasingly data-centric solutions. In an ensemble-based transfer-learning setup for WMT20, en–de and en–zh were reduced to only 100 labeled training examples each, and a stacked ensemble over pretrained predictor–estimator models achieved a Pearson correlation of 0.298, which was reported as 2.54 times higher than the baseline. Semi-synthetic English→Hebrew QE pushes the same logic further: early models trained on about 14,000 skewed examples collapsed toward predicting score 3, but score-balanced augmentation and larger samples changed the picture sharply—430k uniformly distributed examples yielded Pearson 0.88, while 1M and 4M random samples both reached 0.92. This suggests that sentence-level QE in under-resourced, morphology-rich settings is governed not only by architecture but by corpus size, balance, and the distribution of controlled errors (Wu et al., 2021, Siani et al., 12 Mar 2026).
The concept has also broadened beyond MT-system output. In human translation quality estimation, sentence-level models predict four continuous ATA rubric dimensions—Usefulness, Terminology, Idiomatic Writing, and Target Mechanics—for English–Chinese human translations, using cross-attentive neural architectures rather than post-editing targets. In Wikipedia, WikiSQE reframes sentence-level QE as detection of editor-flagged sentence problems and contributes about 3.4M sentences with 153 quality labels, showing that sentence quality can also be modeled as a family of fine-grained categorical defects rather than a single scalar score. This suggests that “sentence-level QE” now names a wider family of sentence assessment problems united by local quality prediction under incomplete external supervision (Yuan et al., 2020, Ando et al., 2023).