GEMBA-MQM: LLM-Based MT Quality Evaluation

• GEMBA-MQM is a reference-free, LLM-driven framework that employs fixed few-shot prompts to extract detailed MQM error annotations from MT outputs.
• It achieves high system-level accuracy (e.g., 96.5% on WMT 2023) by computing severity-weighted quality scores without relying on reference translations.
• Enhancements like prompt compression and batch prompting optimize token usage and cost while maintaining robust evaluation quality across language pairs.

The acronym GEMBA-MQM encompasses a class of methods and metrics that leverage LLMs for fine-grained translation quality evaluation, following the Multidimensional Quality Metrics (MQM) framework. Although its core is in automatic machine translation (MT) assessment, it has rapidly evolved into a central paradigm for high-precision, cost-effective, and scalable evaluation across the MT research spectrum.

1. Definition and Core Methodology

GEMBA-MQM is a prompt-based, reference-free metrics framework that uses LLMs (notably GPT-4 and successors) to extract span-level MQM-style error annotations directly from source-language input and machine-generated translations. Rather than relying on reference translations, it instructs the model to mark and label error spans as dictated by the MQM schema, including error class (e.g., accuracy, fluency), error subtype (e.g., mistranslation, addition), and severity (critical, major, minor). The final quality score is computed as a severity-weighted sum over detected errors: $$\text{Score} = \max(-25, -25 N_{\text{critical}} - 5 N_{\text{major}} - N_{\text{minor}})$$ where $$N_{\text{critical}}, N_{\text{major}}, N_{\text{minor}}$$ are the respective counts detected in a segment (Kocmi et al., 2023, Lu et al., 22 Sep 2024).

GEMBA-MQM typically employs a fixed, few-shot, language-agnostic prompt, containing universal error annotation examples and precise instructions, ensuring applicability beyond specific language pairs or content types. This approach dispenses with language-pair-specific prompt tailoring, a limitation of prior LLM-based error annotation systems.

2. Prompting Strategies and Technical Framework

GEMBA-MQM achieves its span-level error extraction via carefully designed prompts—typically "three-shot," each example illustrating canonical MQM error annotation for structurally distinct errors. The prompt template incorporates:

• Exhaustive error typology and severity definitions
• Three universal prompt examples, reused across language pairs
• Clear output formatting constraints for LLM parsing and downstream score calculation

The methodology is centrally language-agnostic; it avoids prompt customization for new language pairs, sidestepping extensive prompt engineering or chain-of-thought templates (Kocmi et al., 2023).

Prompts guide the LLM (e.g., GPT-4 or other strong models) to output marked spans, each labeled with error class and severity, in a format amenable to automatic parsing and aggregation into segment- or system-level scores.

3. Evaluation Paradigm and Empirical Performance

GEMBA-MQM and its direct assessment variants (GEMBA-DA, GEMBA-SQM) are typically benchmarked against human MQM annotations, using the following main paradigms:

• System-Level Pairwise Accuracy: Agreement with human annotations in the ranking of MT systems, as calculated over all possible system pairs:

$$\text{Accuracy} = \frac{|\text{sign}(\Delta_{\text{metric}}) = \text{sign}(\Delta_{\text{human}})|}{\text{number of system pairs}}$$

• Segment-Level Kendall’s Tau-b: Correlation at the segment level with human rankings, accounting for ties.

Empirical evidence shows that GEMBA-MQM attains the highest system-level pairwise accuracy (e.g., 96.5% on the WMT 2023 blind test set), outperforming state-of-the-art learned and reference-based metrics, notably in high-resource settings (Kocmi et al., 2023). Variants such as GEMBA-DA are also state-of-the-art for both reference-based and reference-free MT evaluation when strong LLMs such as GPT-4 are used (Kocmi et al., 2023).

4. Extensions: Efficiency, Scalability, and Model Integration

Subsequent research tackled token efficiency and cost scaling:

• Prompt Compression: Specialized models (PromptOptMe) are fine-tuned to compress source and translation input, preserving only error-relevant spans for GEMBA-MQM. This results in $\sim$2.4x reduction in token usage, with no loss in system-level or segment-level metric quality (Larionov et al., 20 Dec 2024).
• Batch Prompting: BatchGEMBA-MQM merges multiple translation examples into a single prompt, amortizing instructions and demonstrations across the batch. Batched prompting yields up to 4x reduction in token usage, and, when combined with batching-aware compression, largely preserves—sometimes recovers—evaluation quality lost to batching. For robust LLMs (e.g., GPT-4o), more than 90% of quality is maintained at batch size 4 (Larionov et al., 4 Mar 2025).

These enhancements render GEMBA-MQM scalable for large evaluations: token count, and therefore cost, is minimized while quality is retained. Compression explicitly preserves error spans critical for MQM-style scoring.

5. Error Annotation Quality and Limitations

While GEMBA-MQM excels at system-level ranking, there are important caveats:

• Error Annotation Alignment: The error spans predicted by GEMBA-MQM often exhibit poor overlap with human annotations. Precision for span and "major error" overlap is low (e.g., span precision 9.0%, major error precision 4.9%) (Lu et al., 22 Sep 2024).
• Interpretability and Feedback: The over-detection of errors (especially non-impactful or stylistic) diminishes the utility of span labels for actionable feedback.
• Absence of Impact Validation: GEMBA-MQM does not verify whether correcting a predicted error genuinely improves translation quality. Non-impactful errors persist in metric outputs.

To address this, the MQM-APE extension introduces an LLM-driven automatic post-editing and validation step. Here, each predicted error is "corrected" via LLM, and only those whose correction results in improved translation (validated by LLM or external metrics) are retained. This substantially increases alignment with human span annotations and filtering away spurious feedback (Lu et al., 22 Sep 2024).

6. Practical Impact and Application Constraints

GEMBA-MQM has become the de facto standard for reference-free, fine-grained, and scalable MT system comparison in academic and industrial settings. Its language-agnostic design enables wide coverage without manual prompt engineering.

Recent evidence shows that GEMBA-MQM-based metrics remain limited in the evaluation of literary translations and for distinguishing high-quality human work from strong LLM outputs; here, automatic MQM (including GEMBA-MQM) prefers professional human translations over top LLM systems in only ~9.6% of cases, dramatically lower than holistic or best-worst human judgment methods (Zhang et al., 24 Oct 2024). This indicates an intrinsic limitation of the MQM error-counting paradigm in domains where stylistic or creative fidelity is critical.

For long-context (document-level) QE, GEMBA-MQM-derived methods (e.g., MQM-style EAPrompt) have also performed less robustly than direct assessment (GEMBA-DA) or learned metrics like SLIDE, especially due to aggregation heuristics and lack of output validation routines (Mrozinski et al., 10 Oct 2025).

7. Summary Table: GEMBA-MQM Framework Components

Aspect	GEMBA-MQM
LLM type	GPT-4, or strong open-source LLM
Prompting technique	Fixed, language-agnostic, few-shot
Input	Source, translation (no reference)
Output	Error span, category, severity
Scoring	$\sum_{\text{errors}}$ weighted by severity
Main metric	System-level pairwise accuracy
Efficiency boosters	Prompt compression, batch prompting
Limitations	Weak error span alignment, over-detection, domain sensitivity
Best use	Reference-free system comparison, high-resource MT evaluation

8. Conclusion

GEMBA-MQM and its subsequent variants comprise a robust framework for leveraging LLMs as fine-grained, reference-free judges of machine translation quality, following the well-established MQM annotation schema. While system-level alignment with human judgment reaches or exceeds the prior state of the art, limitations remain in error span annotation fidelity, feedback interpretability, and domain-specific discriminative power. Integrations with prompt compression, batting, and automatic error impact validation are currently the main avenues for efficiency and alignment improvement. GEMBA-MQM now serves as a reference standard for scalable, reliable, and detailed MT quality estimation in automated evaluation pipelines.