Rubric-as-Experts: Case-Specific MQM Rubrics for Translation Quality Evaluation

Abstract: LLMs have shown strong potential in fine-grained translation quality evaluation (QE), yet existing MQM-based approaches typically rely on fixed rubric configurations shared across all translation samples. However, translation instances often differ substantially in error complexity, ambiguity, and required evaluation granularity, making static rubric allocation suboptimal for span-level error detection. We find that larger MQM subtype spaces improve error coverage but also introduce more false positives, while different translation instances prefer different rubric granularities, suggesting that evaluation spaces should be allocated dynamically for each case. Motivated by these observations, we propose a case-specific dynamic rubric framework that adaptively constructs MQM evaluation spaces for individual translation instances. Unlike fully free-form rubric generation methods, our framework remains grounded in the predefined MQM taxonomy while dynamically selecting suitable subtype spaces and evaluation granularity for different cases. Experiments on WMT span-level QE benchmarks across multiple model scales demonstrate that the proposed framework consistently improves MCC and produces cleaner span-level error localization compared with static rubric settings. Our results suggest that combining structured MQM rubrics with case-specific adaptive allocation is an effective strategy for fine-grained LLM-based translation evaluation.

• The paper introduces a dynamic, case-specific MQM rubric allocation system for adaptive span-level translation quality evaluation.
• It employs a cascaded gating mechanism with LoRA-based classifiers to adjust rubric granularity based on error density and translation complexity.
• Empirical results on WMT23 benchmarks demonstrate significant improvements in F1 and MCC metrics over static rubric methods.

Case-Specific Rubric Allocation for Span-Level Machine Translation Quality Evaluation

Motivation and Context

Current LLM-based span-level translation quality evaluation (QE) under the MQM rubric taxonomy often treats evaluation criteria as static, exposing each translation instance to an identical rubric configuration. This is suboptimal given the significant heterogeneity observed across translation cases in error density, ambiguity, and the necessary granularity for error localization. Recent findings highlight that expanding MQM subtype spaces increases recall, but at the expense of false positives, and distinct translation instances reveal variable preferences for rubric granularity. This motivates the "Rubric-as-Experts" framework, which implements case-wise adaptive MQM rubric allocation for span-level error detection, departing from both fully free-form and rigid static approaches.

Figure 1: Static rubric exposure introduces noisy span predictions, whereas dynamic instance-level rubric allocation enables more focused and accurate span-level evaluation.

Framework Design

The dynamic rubric allocation system operates as a cascaded evaluation pipeline. Each translation instance first passes through a correctness gate, which filters out cases predicted to be error-free, minimizing unnecessary computation and avoiding search space expansion on simple inputs. For cases flagged as error-prone or uncertain, an expansion gate determines whether compact rubric coverage suffices or whether broader MQM subtype exploration is required. Depending on instance complexity, the framework selects between small, medium, or comprehensive rubric views, each with distinct MQM type coverage and granularity.

Figure 2: The cascaded MQM evaluation framework routes each translation instance through correctness and expansion gates, adaptively exposing rubric granularity for span-level error detection.

MQM rubric subtypes are organized hierarchically, with each major category containing multiple flattened subtypes relevant to diverse translation errors. The framework utilizes lightweight LoRA-based classifier modules atop Qwen3 backbones for major-category routing and rubric granularity prediction, allowing efficient scaling to large datasets.

Empirical Performance and Routing Effects

Dynamic rubric allocation is empirically validated on WMT23 Zh-En and En-De span-level QE benchmarks. The framework is compared against vanilla LLM baselines, static rubric-guided prompting, and prior competitive QE systems. Results indicate:

• Superior span-level MCC and F1: For Zh-En, Ours-8B achieves 33.78 F1 and 32.40 MCC, outperforming DCSQE and CometKiwi by significant margins. On En-De, Ours-8B gains 2.33 F1 and 2.46 MCC points over DCSQE.
• Strong recall at competitive precision: Dynamic routing models reach recall close to 70% for Zh-En, a substantial improvement over baseline Qwen3 models, while maintaining precision stability.

Ablation studies dissect routing module contributions, revealing that unrestricted rubric expansion delivers higher recall but reduces precision. The correctness and expansion gates selectively enlarge search spaces only for difficult cases, achieving optimal balance.

Rubric Granularity vs. Translation Complexity

Analysis demonstrates that required rubric granularity is strongly coupled to translation complexity:

• Instances with higher error-span counts and longer source sentences increasingly demand comprehensive rubric views for adequate exploration.
• Simpler cases are suitably handled with compact rubrics, avoiding unnecessary span predictions and reducing false positives.

  Figure 3: Required rubric tier distribution rises with error span count and source sentence length.

  

  Figure 4: Bucketed rubric tier statistics further reveal complexity-driven search space allocation.

These patterns persist across language pairs and model scales. Expanded and comprehensive tiers become more prevalent in error-heavy or syntactically dense translations, while expanded tier serves as an intermediate regime.

Rubric Space as Exploration Frontier

The study reveals that MQM rubrics function not only as explicit criteria but also as search space frontiers that shape exploration dynamics. Restricting the rubric space suppresses span discovery and reduces recall; broader rubrics encourage the evaluator to inspect translations from multiple perspectives, including adequacy, fluency, terminology, and stylistic considerations. This interaction between rubric exposure and evaluator behavior explains empirical gains in recall and supports dynamic search space allocation as a principled strategy.

Practical Implications and Future Directions

Adopting case-specific adaptive rubric allocation transforms span-level QE into an instance-wise optimization problem. The approach:

• Improves overall effectiveness in error localization tasks, yielding cleaner and more reliable span predictions across heterogeneous translation samples.
• Offers a lightweight adaptation layer that can be integrated into other LLM evaluators and extended to multi-task or multilingual settings, provided MQM taxonomy coverage is available.
• Invites further investigation into routing dynamics, exploration-aware decoding, and rubric-space design, potentially generalizing to other structured evaluation tasks (e.g., code evaluation or factuality assessment).

The theoretical implication is a re-conception of rubrics as dynamic reasoning constraints, impacting both model interpretability and evaluation robustness under diverse annotation schemas.

Conclusion

The "Rubric-as-Experts" framework establishes that structured, case-specific MQM rubric allocation significantly advances LLM-based span-level translation quality estimation, achieving stronger MCC and F1 than both direct prediction and static rubric settings. The study’s quantitative analysis and qualitative evidence substantiate that dynamic rubric routing preserves exploration efficiency and error localization accuracy by aligning rubric granularity with translation complexity. This paradigm provides a foundation for adaptive evaluation-space control in future AI systems, supporting more nuanced and effective assessments of translation and other structured outputs.