MENLO: From Preferences to Proficiency -- Evaluating and Modeling Native-like Quality Across 47 Languages (2509.26601v1)

Abstract: Ensuring native-like quality of LLM responses across many languages is challenging. To address this, we introduce MENLO, a framework that operationalizes the evaluation of native-like response quality based on audience design-inspired mechanisms. Using MENLO, we create a dataset of 6,423 human-annotated prompt-response preference pairs covering four quality dimensions with high inter-annotator agreement in 47 language varieties. Our evaluation reveals that zero-shot LLM judges benefit significantly from pairwise evaluation and our structured annotation rubrics, yet they still underperform human annotators on our dataset. We demonstrate substantial improvements through fine-tuning with reinforcement learning, reward shaping, and multi-task learning approaches. Additionally, we show that RL-trained judges can serve as generative reward models to enhance LLMs' multilingual proficiency, though discrepancies with human judgment remain. Our findings suggest promising directions for scalable multilingual evaluation and preference alignment. We release our dataset and evaluation framework to support further research in multilingual LLM evaluation.

• The paper presents MENLO, a comprehensive framework that evaluates native-like LLM responses across 47 languages by integrating localized prompt design and sociolinguistic theory.
• The study demonstrates that pairwise RL-trained judges significantly improve preference accuracy by up to 18.0% and ensure high inter-annotator agreement.
• Methodologically, MENLO leverages human-annotated response pairs and reward shaping to optimize LLM evaluations, setting a new standard for multilingual proficiency alignment.

MENLO: Evaluating and Modeling Native-like Quality Across 47 Languages

Introduction and Motivation

MENLO introduces a comprehensive framework for evaluating and improving native-like response quality in LLMs across 47 language varieties. The framework is grounded in sociolinguistic theory, specifically audience design, and operationalizes native-like quality along four dimensions: fluency, tone, localized tone, and localized factuality. MENLO addresses the limitations of prior multilingual evaluation datasets, which often rely on English-centric prompts, translated responses, and narrow dimensions such as linguistic acceptability or naturalness. Instead, MENLO provides localized prompts, high inter-annotator agreement ($\alpha = 0.84$), and a focus on contextually appropriate, native-like output.

MENLO Dataset Construction

MENLO comprises 6,423 annotated prompt–response preference pairs, totaling 81,014 human annotations. Prompts are human-written parametric templates, translated and localized by native speakers for each language variety. Annotation guidelines are meticulously crafted to reduce subjectivity, breaking down each dimension into concrete rubrics and signals. Each response pair is rated by at least three annotators on a 1–5 Likert scale, with majority vote aggregation. The dataset covers a typologically diverse set of languages, including major regional variants and romanized forms of non-Latin scripts.

MENLO achieves higher inter-annotator agreement and broader language coverage than previous datasets such as Pariksha, Recon, M-RewardBench, and MM-Eval. The annotation process incorporates expert annotators for quality control and a custom annotation tool for efficient data collection and screening.

Evaluation of LLM Judges

MENLO systematically benchmarks LLMs as automatic judges of native-like quality. The evaluation setup contrasts pointwise (single response grading) and pairwise (simultaneous grading of two responses) paradigms, with and without detailed grading rubrics. Results demonstrate that pairwise evaluation yields substantial improvements over pointwise, with up to $+12.4\%$ Macro-F1 and $+18.0\%$ Preference accuracy gains. The presence of explicit rubrics further enhances judge reliability, especially in pointwise setups.

Pairwise evaluation anchors judgments, reducing variance and improving alignment with human annotators. The effect is robust across open-source and API models, including Qwen3-4B, Qwen3-32B, Llama-3.1-8B, Llama-3.3-70B, Llama4-Scout, o3, gpt-4o, and gpt-4.1.

Training LLM Judges: SFT vs. RL

MENLO explores supervised fine-tuning (SFT) and reinforcement learning (RL) for judge training, using Qwen3-4B and Llama4-Scout as representative architectures. RL-trained judges consistently outperform SFT, with Qwen3-4B RL achieving $+4.0\%$ Macro-F1 and $+2.9\%$ Preference accuracy over zero-shot. Reward shaping—combining binary accuracy, smoothing for near-miss predictions, and preference bonuses—further boosts performance.

For reasoning-oriented models, SFT without chain-of-thought supervision degrades performance, while RL incentivizes explicit reasoning and robust score assignment. Multi-task RL (joint training across all dimensions) matches or exceeds single-task optimization, except for localized factuality, which remains challenging and may require retrieval or tool augmentation.

Cross-Language and Per-Dimension Analysis

MENLO provides granular analysis of judge performance across languages and dimensions. Preference accuracy varies widely by language, with tr_TR (Turkish) at $82.1\%$ and bn_BD (Bengali) at $37.9\%$. Gains do not strictly correlate with resource levels, highlighting the importance of localized prompt and annotation design. English-only training fails to generalize, underscoring the necessity of multilingual data for robust cross-lingual alignment.

Tone and localized tone dimensions achieve the highest Macro-F1 and Preference accuracy, while localized factuality lags, indicating persistent challenges in grounding responses in local context.

Figure 1: Preference accuracy per language for pairwise RL-trained Qwen3-4B, illustrating substantial cross-lingual variance and the impact of localized training.

Generative Reward Models and Policy Improvement

MENLO demonstrates that RL-trained judges can serve as generative reward models (RMs) for policy optimization. Post-training Qwen3-4B with a pairwise RL-trained judge yields consistent quality gains across LLM and human evaluators. LLM judges report average score improvements of $+0.80$ to $+1.16$ (up to $+36.1\%$), while human raters observe a more conservative $+10.9\%$ gain. This reveals a systematic tendency for LLM judges to overestimate improvements relative to human annotators, a critical consideration for future reward modeling.

The two-stage evaluation strategy—automated LLM judging across all languages and human validation on high-resource subsets—confirms the effectiveness of MENLO's unified framework for scalable multilingual proficiency alignment.

Implementation Considerations

MENLO's framework is designed for practical deployment:

• Annotation Pipeline: Requires native speaker recruitment, localized prompt engineering, and robust annotation tooling.
• Judge Training: RL with reward shaping is preferred, leveraging pairwise templates and explicit reasoning. Multi-task training is efficient and effective for most dimensions.
• Policy Optimization: Generative RMs can be integrated into RLHF pipelines for multilingual post-training, with careful calibration against human raters to avoid overestimation.
• Scaling: MENLO's approach is scalable to new languages and domains, provided sufficient native annotation resources and localized prompt design.

Implications and Future Directions

MENLO sets a new standard for multilingual LLM evaluation and alignment, emphasizing native-like quality over mere acceptability or translationese. The framework's sociolinguistic grounding and rigorous annotation methodology enable nuanced assessment and improvement of LLMs in real-world conversational settings.

Persistent challenges remain in modeling localized factuality and mitigating LLM judge overconfidence. Future work should explore retrieval-augmented evaluation, pluralistic alignment, and automated rubric generation. MENLO's dataset and framework provide a foundation for research into cross-lingual transfer, cultural adaptation, and scalable human-in-the-loop evaluation.

Conclusion

MENLO operationalizes native-like quality evaluation and improvement for LLMs across 47 languages, integrating sociolinguistic theory, localized prompt design, high-quality annotation, and RL-based judge training. Pairwise evaluation and reward shaping are critical for reliable automated judging, and generative RMs enable direct policy optimization. MENLO's unified framework advances scalable multilingual proficiency alignment, with ongoing challenges in localized factuality and LLM-human agreement warranting further investigation.