BatchGEMBA-MQM: MT Evaluation & MQM Measurement

Updated 23 December 2025

BatchGEMBA-MQM is a dual framework combining LLM-based batched prompting for machine translation quality evaluation with a precise experimental protocol for nuclear magnetic quadrupole moment measurement.
It leverages prompt compression techniques to reduce token usage by up to 2–4× while maintaining high alignment accuracy and strong Pearson correlation in MT evaluations.
In nuclear physics, the framework employs batched Ramsey interferometry and enhanced magnetic gradients to achieve sensitivity improvements exceeding previous MQM measurement limits.

BatchGEMBA-MQM refers to two distinct high-precision frameworks at the intersection of language, computation, and physics. In current academic literature, the term predominantly denotes a token-efficient framework for machine translation (MT) evaluation based on LLM batched prompting and prompt compression. Additionally, the phrase has been employed to describe a protocol for measuring the nuclear magnetic quadrupole moment (MQM) in optically trapped atoms. Both contexts are grounded in rigorous computational or experimental methodologies for large-scale, high-fidelity evaluation. This article provides a comprehensive exposition of both frameworks, their theoretical motivation, workflows, and empirical results.

1. BatchGEMBA-MQM for Machine Translation Quality Evaluation

BatchGEMBA-MQM is an LLM-based metric and protocol for automatic MT quality estimation, integrating batched prompting with the GEMBA-MQM segmentation-based error annotation paradigm. GEMBA-MQM, originally proposed by Kocmi & Federmann, employs GPT-class models to identify error spans and rates their severity for MT output by processing single translation segments per prompt. BatchGEMBA-MQM generalizes this via joint evaluation over multiple translations per prompt, reducing token overhead and cost while preserving alignment and accuracy through explicit output formatting (Larionov et al., 4 Mar 2025, Kocmi et al., 2023).

2. Formal Definitions and Scoring Methods

The single-example GEMBA-MQM quality score for source $x$ , machine translation output $y$ , and reference $r$ is computed as follows:

$Q_i = w_c \cdot e^c_i + w_m \cdot e^m_i + w_k \cdot e^k_i,$

where $e^c_i, e^m_i, e^k_i$ denote the counts of error spans at critical, major, and minor levels, respectively, with severity weights $w_c > w_m > w_k$ (typically 3, 2, 1).

BatchGEMBA-MQM evaluates a batch $B = \{(x_i, y_i, r_i)\}_{i=1}^n$ with a single LLM call, yielding per-example scores $Q_i$ . The batch-aggregated score is the mean:

$S_{\text{batch}}(B) = \frac{1}{n} \sum_{i=1}^n Q(x_i, y_i, r_i)$

Alternative aggregations (e.g. weighted, min, max) are possible, but empirical studies typically use the average (Larionov et al., 4 Mar 2025).

3. Batched Prompting and Output Schema

Batched prompt construction concatenates a common instruction ( $P_\text{instr}$ ), fixed in-context demonstrations $D_1, D_2, D_3$ , and the test translations $T_1$ through $T_n$ :

$P(B) = P_\text{instr} \;\Vert\; D_1 \;\Vert\; D_2 \;\Vert\; D_3 \;\Vert\; T_1 \;\Vert\; ... \;\Vert\; T_n$

The output is required in a JSON schema, with each "evaluation" object indexed by translation ID:

{
  "evaluation": [
    { "translation_id": 0, "response": { "critical": [...], "major": [...], "minor": [...] }},
    ...
  ]
}

This explicit alignment prevents misassignment of outputs when batch size increases or prompts are compressed (Larionov et al., 4 Mar 2025).

4. Prompt Compression: Architecture and Training

BatchGEMBA-MQM introduces a batching-aware prompt compression model to further reduce token count and to mitigate batching-induced quality degradation. The model is based on fine-tuning Llama 3.2 3B using LoRA adapters (r=64, α=16; 97M trainable parameters), in a two-stage procedure:

Data-Driven Compression: Random token-level ablations preserving human-annotated error spans, maximizing likelihood (cross-entropy loss).
Preference Optimization: Dataset of compressed prompts per batch, scored by a target LLM; preference ranking via ORPO, encouraging (a) JSON conformity and (b) high MQM correlation.

The compression model outputs $\hat{P}(B)$ , a shorter prompt that matches the semantic and formatting requirements of the original (Larionov et al., 4 Mar 2025).

5. Empirical Results: Token Efficiency and Correlation Retention

Token Usage & Savings

Across languages and LLMs, BatchGEMBA-MQM achieves 2–4× reduction in token usage relative to single-example prompting; prompt compression further reduces tokens by 13–15% on average. For instance, with GPT-4o:

Model	Batch 1	Batch 2	Batch 4
GPT-4o (orig)	5.4M	4.1M	2.8M
GPT-4o (comp)	4.7M	3.8M	2.6M

Reduction Factor $R(b)$ : $R(2) \approx 1.32$ , $R(4) \approx 1.93$ Compression Savings $\Delta \approx$ 13–15%

Quality Retention

Batching without compression degrades Pearson $r$ (human-system correlation) by 24–44% for GPT-4o at $b=2$ ; compression recovers 20–30 absolute percentage points of $r$ , achieving up to 90% retention at $b=4$ for GPT-4o. Open-source LLMs degrade more heavily with batching and are less robust to compression.

Model	$q_{\text{orig}}(2)$	$q_{\text{comp}}(2)$	$q_{\text{orig}}(4)$	$q_{\text{comp}}(4)$
GPT-4o	55.5%	82.6%	62.7%	90.9%
GPT-4o-mini	75.2%	66.0%	—	—
Mistral	27.7%	35.8%	—	—

Error rates remain ≤0.5% except for CommandR7B at $b=1$ (Larionov et al., 4 Mar 2025).

6. Analysis, Best Practices, and Caveats

Batch Size Selection: For GPT-4o/mini, batching at $b \in \{2,4\}$ with $\sim$ 15% compression is optimal for 2× token efficiency and ≥80% quality.
Prompt Compression Use: Particularly effective for robustness at higher batch sizes, likely due to reduced inter-example interference.
Model-Dependent Behavior: Less robust models (Mistral, Phi4) require lower batch sizes and conservative compression (≤10%) to avoid formatting errors.
JSON Conformity: Verifying output schema integrity is critical; if misformatting exceeds 1%, reduce batch size or increase guidance in prompts.
Monitoring: Latency, throughput, cost, and reproducibility depend on LLM model class and API.

BatchGEMBA-MQM substantially improves the cost-quality tradeoff for large-scale MT evaluations, making it practical for industrial benchmarks and meta-evaluation studies (Larionov et al., 4 Mar 2025, Kocmi et al., 2023).

7. BatchGEMBA-MQM in Nuclear Physics: MQM Measurement Protocol

In nuclear physics, "BatchGEMBA-MQM" denotes a precision measurement protocol for detecting the nuclear magnetic quadrupole moment (MQM) in optically trapped $^{173}$ Yb atoms (Sunaga et al., 2023). The method leverages:

Enhanced magnetic-field gradients in the $^3P_2$ state.
Ultracold atom techniques for long coherence times.
Ramsey interferometry with batched experimental runs to average quantum projection noise.

Formalism: The MQM-electron interaction Hamiltonian,

$\hat H_{\rm MQM} = -\frac{1}{6}\sum_{i,j}\mathcal{M}_{ij}\,(\nabla B)_{ij}$

generates an energy shift observable by phase accumulation in Ramsey sequences. The experimental cycle uses batch processing to reduce statistical uncertainty.

Projected Sensitivity: The combination of large electronic enhancement factors and low-noise Ramsey protocol yields projected MQM sensitivity $\delta\mathcal M \sim 3 \times 10^{-8}\, \mu_N\,\text{fm}$ , exceeding previous limits by over an order of magnitude (Sunaga et al., 2023).

8. Outlook and Future Directions

BatchGEMBA-MQM frameworks in both computational linguistics and fundamental physics exemplify the convergence of batched protocol optimization and error-controlled measurement. In MT, further advances may optimize compression algorithms, prompt strategies, and adapt to evolving LLM architectures. For experimental physics, increased atom numbers, improved coherence control, and advanced readout will extend MQM sensitivity and test deeper aspects of fundamental symmetry violation. Public releases of code and models (see https://github.com/NL2G/batchgemba) promote reproducibility and facilitate benchmarking across domains (Larionov et al., 4 Mar 2025).