--- title: Error Span Annotation in NLG Evaluation url: https://www.emergentmind.com/topics/error-span-annotation-esa type: topic --- # Error Span Annotation in NLG Evaluation Error Span Annotation (ESA) is a methodology for localizing, classifying, and weighting erroneous spans within text, primarily for the evaluation, training, and diagnosis of Natural Language Generation tasks such as Machine Translation (MT) and Large Language Model (LLM) outputs. ESA bridges the granularity of full error taxonomies—such as the MQM standard—and the simplicity of global quality scores, enabling fine-grained signal for both human evaluation and model learning at manageable annotation cost and cognitive load [2406.11580][2410.16509][2504.08697][2010.03260]. ESA encompasses protocol design, formal span schemas, loss functions for model training, and standardized evaluation metrics. ## 1. Formalization, Protocols, and Schemas ESA operates over a sequence or pair of sequences (e.g., source S and translation T) by identifying a set of error spans. Each error span is a contiguous substring in the output (and possibly also in the source), labeled with severity and, optionally, with an error type and rationale. The ESA annotation schema (as formalized by TranslationCorrect and related pipelines) is as follows [2506.18337]: | Field | Type | Description | |--------------------|-------------|--------------------------------------------------------------------| | source_text | string | Original input segment S | | mt_text | string | System output segment T | | error_spans | list | List of error-span records, each with: | | – start_index_trans| integer | 0-based char offset in T (start) | | – end_index_trans | integer | 0-based char offset in T (exclusive end) | | – error_type | string | Category from a fixed set (e.g. Omission, Grammar, Addition, etc.) | | – error_severity | string | “Minor” or “Major” | | – description | string | Brief human-readable explanation or correction hint | Additional fields may capture source alignment, associated spans, or minimal corrections [2503.04232]. Annotation is governed by detailed guidelines: errors are marked as minimal spans covering the mistake, with careful differentiation by severity. Categories are mapped to either coarse-grained tags (as in base ESA [2406.11580]) or more refined MQM-derived taxonomies (Addition, Omission, Mistranslation, etc.) [2506.18337]. For tasks beyond MT, ESA generalizes to span-level error detection in GEC [2010.03260], hallucination identification in data-to-text [2504.08697], and compositional error taxonomies in PLM-generated text [2503.04232]. ## 2. Human and AI-Assisted Annotation Workflows Human ESA protocols are exemplified by Appraise-style interfaces in which annotators: 1. Read source and translation side-by-side. 2. Highlight each erroneous span, marking “Major” or “Minor” severity (or applicable category). 3. Use a special “[MISSING]” tag for omissions. 4. Assign a direct [0,100] quality score after marking errors [2406.11580]. AI-assisted ESA leverages high-recall QE models (e.g., GPT-4 in few-shot mode) to pre-populate candidate spans. Annotators then post-edit these suggestions—removing, resizing, re-categorizing, or confirming as needed. This pipeline, as validated in ESA\textsuperscript{AI}, halves per-span annotation time (71s→31s) and allows for further budget reduction by omitting segments with no predicted errors (≈24% savings) [2406.12419]. Automation bias is minimal, as annotators do not over-rely on imperfect AI hints, and quality-control perturbations show comparable error detection rates to human-only annotation. Large Language Models achieve inter-annotator agreement on par with human crowdworkers and can serve as cost-effective, scalable span annotators in a range of tasks (MT error detection, propaganda, data-to-text) when provided with appropriate zero-shot or few-shot structured-output prompts [2504.08697]. ## 3. ESA in Model Training and Fine-Tuning ESA not only supports evaluation but is a cornerstone of novel fine-tuning objectives for NLG models. The Training with Annotations (TWA) framework integrates span-level error signals into MT model learning [2410.16509]: - **Weights**: Each error span $S_i$ identified in the output is assigned a weight $w_i$ (e.g., $-5$ for major, $-1$ for minor, $-0.1$ for minor punctuation, per MQM mapping). Tokens in $S_i$ inherit $w_i$. Tokens before the first error receive $+1$; tokens after the first error are given weight $0$ as “off-trajectory.” - **Loss Functions**: For error spans ($w < 0$), a weighted unlikelihood loss is applied: $$L_{\mathrm{error}}(S) = -|w| \cdot \log\left[1 - \prod_{t \in S} p(y_t | y_{