Sqator: Span-Level QA Evaluator

• Sqator is a span-centric quality assurance framework that assesses translation reliability at localized text segments, enhancing error traceability.
• It employs a three-step automated QA workflow: structural auditing, COMET-based quality profiling, and LLM-driven span error diagnostics.
• The architecture integrates global metrics with detailed span-level insights to prioritize human review and improve multilingual benchmark integrity.

Sqator, short for Span-level Quality Assurance EvaluaTOR, denotes a span-centric approach to quality assurance in which reliability is assessed not only at corpus or sequence level but also at the level of localized text segments whose failures can be inspected and acted upon. In the literature summarized here, the concept is anchored by a three-step automated QA workflow for machine-translated benchmarks—structural corpus audit, COMET-based quality profiling, and an LLM-based span-level translation error landscape—and is extended by adjacent work on span-supervised machine translation, quality estimation, and span-level uncertainty modeling (Thellmann et al., 2 Apr 2026). This suggests that Sqator is best understood as a layered QA architecture rather than a single metric: it combines structural validity checks, scalar quality signals, and interpretable span diagnostics so that large multilingual benchmark suites and generated outputs can be reviewed at scale without collapsing all failures into one score.

1. Conceptual scope and intellectual lineage

The immediate problem setting for Sqator is benchmark quality assurance under machine translation. The EU20 study identifies three failure modes that standard releases often leave under-validated: structural corruption of benchmark items, uneven translation quality across tasks and languages, and lack of interpretable diagnostics showing where meaning was lost (Thellmann et al., 2 Apr 2026). In that framing, translation reliability at scale is operationalized along three complementary dimensions: structural integrity at corpus level, item-level quality profiling with COMET, and span-level diagnostic validity with an LLM-as-a-judge setup.

Related work broadens the same span-centric logic. In machine translation training, span-level MQM annotations are treated as direct supervision over local quality rather than as mere metadata, distinguishing error spans from non-error spans and explicitly modeling off-trajectory regions after the first error (Zhang et al., 2024). In quality estimation, sentence-level scores and word-level OK/BAD predictions are unified, after which contiguous non-OK regions are converted into fine-grained error spans with severities (Geng et al., 2023). In LLM generation, span-level uncertainty estimation is formalized as the task of jointly detecting semantically meaningful spans and assigning each a continuous uncertainty score (Zhang et al., 7 Jul 2026).

Work	Span object	Relevance to Sqator
"Diagnosing Translated Benchmarks: An Automated Quality Assurance Study of the EU20 Benchmark Suite" (Thellmann et al., 2 Apr 2026)	Translation error landscape at span level	Defines layered QA over translated benchmarks
"Learning from others' mistakes: Finetuning machine translation models with span-level error annotations" (Zhang et al., 2024)	MQM error spans and non-error spans	Shows span annotations as training signals
"Unify word-level and span-level tasks: NJUNLP's Participation for the WMT2023 Quality Estimation Shared Task" (Geng et al., 2023)	Contiguous word-derived error spans	Connects sentence, word, and span QE
"SpanUQ: Span-Level Uncertainty Quantification for LLM Generation" (Zhang et al., 7 Jul 2026)	Semantically coherent uncertainty spans	Provides native span detection and scoring

Across these works, a span is not merely a window of tokens. Depending on the task, it is either a contiguous erroneous output region, a contiguous block induced from word-level error tags, or a contiguous text segment that conveys a single coherent unit of meaning whose uncertainty can be independently assessed. The common denominator is locality with semantic or evaluative relevance.

2. Layered QA architecture

The canonical Sqator workflow is the three-step automated QA pipeline used to study EU20, a suite of five English benchmarks translated into 20 European languages with DeepL (Thellmann et al., 2 Apr 2026). The first step is a structural corpus audit with targeted fixes. The purpose is not only to confirm that files load, but to detect corruption in item identifiers, fields, answer choices, splits, subsets, and other task-specific formatting constraints that determine whether a benchmark remains valid as a benchmark rather than as raw text.

The second step is quality profiling with COMET. The EU20 study uses both reference-free and reference-based COMET and compares translation services including DeepL, ChatGPT, and Google (Thellmann et al., 2 Apr 2026). In Sqator terms, this component serves as the quality profiler: it produces scalar signals useful for ranking datasets, languages, or systems and for identifying where manual review effort is likely to be most valuable. Reference-based COMET on MMLU against human-edited samples is used as an additional check and points in the same direction as the reference-free findings.

The third step is an LLM-based span-level translation error landscape. This stage is what makes the workflow distinctly span-oriented. Corpus-level averages and sentence-level metrics can indicate that something is wrong, but they do not show what kind of failure occurred, where it occurred, or whether it mainly affects fluency or adequacy. The span layer is designed to expose interpretable local failures, especially accuracy and mistranslation errors that threaten benchmark validity (Thellmann et al., 2 Apr 2026).

The paper’s own design lesson is explicit: automated QA should provide “quantifiable reliability and diagnostic criteria” and “practical, scalable indicators that help prioritize review,” while “complementing, not replacing, human gold standards” (Thellmann et al., 2 Apr 2026). In implementation terms, this maps almost directly to four modules: benchmark auditor, quality profiler, span detector or error classifier, and review prioritizer.

3. Span supervision and the attribution problem

A central rationale for Sqator is that sequence-level supervision suffers from an attribution problem. If an entire translation or generation is assigned a low score, the learner or maintainer still does not know where the error is, which tokens caused it, or which parts remain good and should be preserved (Zhang et al., 2024). The same criticism appears in uncertainty estimation: token-level scores lack semantic coherence, while sequence-level scores fail to localize errors (Zhang et al., 7 Jul 2026).

The TWA framework makes this precise for MT fine-tuning. Professional translators annotate error spans in MT outputs, each associated with an MQM error category and severity, but the learning objective uses chiefly the existence and severity of spans. Error spans receive a span-level unlikelihood loss,

$$\mathcal{L}_\text{TWA}(\text{error span})=-|w|\log(1-p_\text{span}),$$

where

$$p_\text{span}=\exp\left(\sum_{t\in \text{span}}\log p_t\right),$$

and more severe errors incur larger penalties through $|w|$ (Zhang et al., 2024). Non-error spans are trained with ordinary likelihood only if they occur before the first error; later tokens are treated as off-trajectory and receive zero loss. The method therefore rewards good prefixes, penalizes annotated bad spans, and ignores later regions conditioned on already-bad continuations.

This span formulation has two implications for Sqator. First, localization is not merely explanatory; it can be supervisory. Second, local quality cannot always be interpreted independently of prefix state. A token sequence that is locally fluent may still be off-trajectory because it is conditioned on a mistaken earlier span. That distinction matters when Sqator is used not only to diagnose outputs but also to improve systems that generate them.

SpanUQ generalizes the same intuition beyond translation. It defines a span as “a contiguous text segment that conveys a single, coherent unit of meaning whose uncertainty can be independently assessed” and casts the task as jointly detecting such spans and assigning each a continuous uncertainty score (Zhang et al., 7 Jul 2026). This supplies a broader semantic notion of locality: not merely erroneous regions, but independently assessable meaning units.

4. Detection and scoring mechanisms

Two main design patterns for Sqator-style span modeling are visible in the cited literature. One derives spans from token- or word-level predictions. The other predicts spans natively.

The word-to-span route is exemplified by the WMT 2023 QE system from NJUNLP. Source and MT hypothesis are encoded jointly with XLM-R large, sentence-level MQM-derived scores are predicted from pooled target representations, and binary word-level OK/BAD tags are predicted from averaged word representations (Geng et al., 2023). The multitask objective is

$$L_{\text{QE}} = L_{\text{CE}} + \alpha L_{\text{MSE}} + \beta L_{\text{Rank}},$$

with $\alpha=1$, $\beta=1000$, and ranking margin $\epsilon=0.03$ (Geng et al., 2023). For span detection, ensemble-averaged word OK probabilities are thresholded into OK, Minor, or Major labels using $\epsilon_{\text{major}}$ and $\epsilon_{\text{minor}}$; contiguous non-OK words are merged into spans; and each span receives the worst severity found in that block. On English-German, this system achieved MCC 29.7 for word-level QE, F1 28.4 for fine-grained error span detection, and Spearman 47.9 for sentence-level prediction (Geng et al., 2023).

The native span route is exemplified by SpanUQ. It uses a frozen LLM backbone and a lightweight probe with a DETR-style span decoder, jointly predicting span boundaries, validity, and uncertainty (Zhang et al., 7 Jul 2026). Each span’s uncertainty is modeled with a Mixture of Beta distribution, and the reported point estimate is the mixture mean,

$$\hat{u}_k=\sum_j \pi_{k,j}\frac{\alpha_{k,j}}{\alpha_{k,j}+\beta_{k,j}}.$$

Predicted spans are matched to gold spans with the Hungarian algorithm using a cost that combines boundary distance plus GIoU, absolute uncertainty error, and validity BCE (Zhang et al., 7 Jul 2026). On Qwen3-14B, the detector attains precision 0.857, recall 0.970, and F1 0.910; the full framework achieves AUROC 0.939, MAE 0.110, and $$p_\text{span}=\exp\left(\sum_{t\in \text{span}}\log p_t\right),$$0 at span level, and is reported as 10–20x faster than sampling-based methods (Zhang et al., 7 Jul 2026).

For Sqator, the contrast is consequential. Heuristic span induction from word-level probabilities is straightforward and compatible with existing QE pipelines, but its label space and structural fidelity can be limited. Native span detectors are more expressive, especially when overlapping spans or continuous uncertainty scores are required, but they typically demand richer supervision and, in the case of SpanUQ, white-box access to hidden states.

5. Empirical behavior in translated benchmarks

The EU20 study provides the clearest direct evidence for why a Sqator-style system is useful in benchmark maintenance. Its trends are consistent: datasets with lower COMET scores exhibit a higher share of accuracy or mistranslation errors at span level, notably HellaSwag, whereas ARC is comparatively clean (Thellmann et al., 2 Apr 2026). Reference-based COMET on MMLU against human-edited samples points in the same direction. The significance of this pattern is methodological rather than merely descriptive: corpus-level and sentence-level scalar scores appear to correlate with the local error distributions that matter most for benchmark validity.

The practical consequence is prioritization. If lower COMET is associated with a higher proportion of adequacy-threatening span errors, automated QA can identify which translated benchmarks deserve human review first. Conversely, structurally clean datasets with favorable COMET profiles and relatively mild span error landscapes may require less immediate intervention. The EU20 work releases cleaned and corrected versions of the datasets together with code for reproducibility, making the QA process not only diagnostic but also curatorial (Thellmann et al., 2 Apr 2026).

A common misunderstanding is that the scalar metric and the span diagnostic compete. In the EU20 workflow they do not. COMET is used for scalable profiling, while the span-level landscape provides interpretable evidence about error type and location. Sqator therefore does not reject global metrics; it re-situates them as one layer in a larger diagnostic stack.

6. Limitations, misconceptions, and open directions

The most important limitation is conceptual: automated span-level QA is presented as a complement to human gold standards, not a replacement for them (Thellmann et al., 2 Apr 2026). This is especially relevant where benchmark validity depends on task semantics, answer behavior, or domain-specific nuance that may not be captured by automated judges.

A second limitation concerns representation. In the NJUNLP QE system, span outputs are heuristic post-processing over word-level probabilities, predict only Minor and Major severities in the reported conversion rule, and do not explicitly model Critical severity, gap tags, or rich MQM categories at inference time (Geng et al., 2023). Character-level offset conversion is required by the task format but is not described in detail. These constraints make such systems strong baselines for actionable highlighting rather than complete parsers of MQM-style annotation.

A third limitation concerns supervision and access. SpanUQ requires white-box hidden-state access, focuses on factual uncertainty in English, and relies on compute-intensive multi-sample label creation grounded in Wikipedia-supported verification (Zhang et al., 7 Jul 2026). The authors explicitly note that the probe cannot capture uncertainty arising from