GlobalMMLU: A Multilingual Evaluation Benchmark

• GlobalMMLU is a multilingual evaluation benchmark that extends the English MMLU to 42 languages, explicitly addressing translation artefacts and cultural bias.
• It combines machine translation with professional and community verification, supplementing the test set with annotations for culturally-sensitive and culturally-agnostic items.
• Studies using GlobalMMLU employ metrics like κp and TIR to assess cross-lingual functional similarity and reasoning alignment, highlighting impacts of model scale and script differences.

Searching arXiv for papers on GlobalMMLU and closely related multilingual evaluation work. GlobalMMLU is a multilingual evaluation benchmark derived from MMLU and designed to make cross-lingual assessment less dependent on naïve translation and less opaque to cultural bias. In its 2024 release, it extends the original English MMLU test set to 42 languages, combines machine translation with professional and community verification, and adds metadata that separates culturally-sensitive and culturally-agnostic items. Subsequent work has used GlobalMMLU configurations to study two distinct properties of multilingual LLM behavior: functional similarity across languages via the chance-corrected agreement metric $\kappa_p$, and reasoning–answer misalignment across languages via Trace Inconsistency Rate (TIR) (Singh et al., 2024, Mishra et al., 4 Sep 2025, Ovalle et al., 27 Dec 2025).

1. Definition and benchmark configurations

GlobalMMLU was introduced to address two limitations of multilingual evaluation: translated benchmarks can preserve the cultural assumptions of the English source, and translation itself can introduce artefacts that alter meaning or clarity. The benchmark therefore aims not only to broaden language coverage but also to make bias and translation quality explicit evaluation variables rather than hidden confounders (Singh et al., 2024).

The 42-language release covers Amharic, Arabic, Bengali, Chinese, Czech, Dutch, English, Filipino, French, German, Greek, Hausa, Hebrew, Hindi, Igbo, Indonesian, Italian, Japanese, Korean, Kyrgyz, Lithuanian, Malagasy, Malay, Nepali, Nyanja, Persian, Polish, Portuguese, Romanian, Russian, Serbian, Sinhala, Somali, Shona, Spanish, Swahili, Swedish, Telugu, Turkish, Ukrainian, Vietnamese, and Yoruba. Later studies operate on narrower GlobalMMLU configurations optimized for particular analyses rather than reproducing the full 42-language release verbatim.

Configuration	Coverage	Primary use
Global-MMLU release	42 languages; culturally-sensitive and culturally-agnostic subsets	Bias-aware multilingual evaluation
GlobalMMLU in the $\kappa_p$ study	20 languages, 47 subjects, 13,844 questions per language	Cross-lingual functional similarity
GlobalMMLU in the TIR study	6 languages, 66,615 total questions, roughly 11,100 per language	Reasoning–answer alignment

This multi-configuration usage is important for interpreting reported numbers. A result on 20-language fully parallel multiple-choice evaluation, a result on 42-language bias annotation, and a result on 6-language chain-of-thought auditing all concern GlobalMMLU, but they characterize different benchmark slices and different failure modes (Mishra et al., 4 Sep 2025, Ovalle et al., 27 Dec 2025).

2. Construction, translation verification, and annotation

The 42-language release begins with machine translation of all English MMLU questions and answers via the Google Translate API. Translation quality is then improved through several human-in-the-loop pathways. Four high-resource languages—Arabic, French, Hindi, and Spanish—form a professional “Gold Set” that was post-edited by paid professional translators. Fourteen languages were incorporated from OpenAI’s professionally translated MMMLU. For 11 mid- and low-resource languages—Amharic, Czech, Malay, Persian, Romanian, Russian, Sinhala, Telugu, Turkish, Ukrainian, and Vietnamese—native speakers reviewed and edited translations via Argilla. The final mix contains 16 fully machine-translated languages, 14 MMMLU human translations, 4 professional post-edits, and 8 community-edited languages with at least 50 human-edited samples each (Singh et al., 2024).

GlobalMMLU also introduces a dedicated cultural-sensitivity annotation layer. A random sample of 2,850 English MMLU items, corresponding to 50 items per subject area across 57 subjects, was independently reviewed by 200 annotators. Each item was labeled for whether correct answering required cultural knowledge, geographic knowledge, or English-dialect knowledge; each item received at least 3 annotations and often up to 10, and majority vote determined the final label. An item is designated Culturally-Sensitive (CS) if any of those criteria apply, and Culturally-Agnostic (CA) otherwise (Singh et al., 2024).

This design distinguishes GlobalMMLU from evaluation sets produced by direct translation alone. The benchmark is not merely multilingual in surface form; it is also annotated for the kinds of world knowledge and cultural framing that can systematically distort measured model quality. A plausible implication is that GlobalMMLU should be read as both a dataset and a diagnostic instrument for multilingual evaluation methodology.

3. Cultural sensitivity, bias metrics, and ranking distortion

The 2,850-item annotation exercise quantifies the extent of cultural skew in the source benchmark. Of the annotated items, 28.0% are CS. Within the CS subset, the tag breakdown is Geographic 54.7%, Cultural 32.7%, Dialect 0.5%, with overlaps such as geo + cul accounting for 10.6%. Within the Geographic tags, 64.5% concern North America and 20.4% concern Europe, so North America + Europe account for 84.9%. Within the Cultural tags, 86.5% concern Western Culture, while South Asian, Latin American, and African tags account for 4.0%, 1.3%, and 1.1%, respectively (Singh et al., 2024).

The benchmark formalizes this structure with simple diagnostic quantities:

$$B=\frac{N_{\mathrm{sensitive}}}{N_{\mathrm{total}}},$$

$$G_r=\frac{N_{\mathrm{region}\,r}}{N_{\mathrm{geo\;questions}}},$$

$\Delta A=A_{\mathrm{CA}}-A_{\mathrm{CS}}.$

Accuracy itself is reported as

$$\mathrm{Accuracy}=\frac{\#\;\mathrm{correct}}{\#\;\mathrm{total}}\times 100\%.$$

In large-scale evaluation over 14 systems, average accuracy is higher on CS than on CA across all models, because CS items are concentrated in Social Sciences and Humanities where performance is higher. At the same time, the standard deviation across languages is larger for CS than for CA for every model. The reported resource-level standard deviations are CA $\sigma \approx 3.21$, CS $\sigma \approx 3.86$ for high-resource languages; CA $\sigma \approx 3.42$, CS $\sigma \approx 4.60$ for mid-resource languages; and CA $\kappa_p$0, CS $\kappa_p$1 for low-resource languages. Model rankings also shift materially when evaluation moves from the full annotated portion to CA or CS subsets: the average number of models whose rank changes is 3.4 on CA and 5.7 on CS, with average position shifts of 3.7 and 7.3, respectively (Singh et al., 2024).

These results support two methodological conclusions. First, translated MMLU scores are not invariant to the cultural composition of the evaluation slice. Second, headline multilingual rankings can conceal sensitivity to Western-centric content and translation quality. A common misconception is that broad language coverage alone yields globally representative evaluation; GlobalMMLU was designed precisely to show that this is insufficient.

4. Cross-lingual functional similarity and $\kappa_p$2

A later study uses a fully parallel GlobalMMLU setting to measure whether a model behaves similarly when asked the same question in different languages. In that setup, GlobalMMLU contains 20 languages—am, ar, bn, zh, en, fr, de, he, hi, id, it, ja, ko, fa, ru, es, sw, te, tr, vi—47 sub-subjects partitioned into STEM, Humanities, Social Sciences, and Other, and 13,844 questions per language. All languages share the same item IDs; only the natural-language wording differs. Prompts follow the LM-Eval-Harness multiple-choice template, with zero-shot Q–A and options A–D, no in-language fine-tuning, no translation-based augmentation, temperature $\kappa_p$3, and max-tokens set just high enough to output a single letter choice (Mishra et al., 4 Sep 2025).

The study applies the chance-corrected agreement metric $\kappa_p$4 to two prediction streams, such as a model’s outputs in language A and language B. Observed agreement is the empirical rate of identical choices across items, expected agreement is the chance agreement induced by the streams’ marginal choice frequencies, and

$\kappa_p$5

Its intended interpretation is explicit: if two runs make the same mistakes beyond what their individual accuracies predict, $\kappa_p$6 is high; if agreement is largely due to both runs being mostly correct, $\kappa_p$7 is down-weighted (Mishra et al., 4 Sep 2025).

Quantitatively, mean $\kappa_p$8 over all 190 language pairs grows with model scale. For Qwen-3, the reported averages are $\kappa_p$9 at 1.7B, $$B=\frac{N_{\mathrm{sensitive}}}{N_{\mathrm{total}}},$$0 at 4B, $$B=\frac{N_{\mathrm{sensitive}}}{N_{\mathrm{total}}},$$1 at 8B, and $$B=\frac{N_{\mathrm{sensitive}}}{N_{\mathrm{total}}},$$2 at 14B. For Gemma, the study reports $$B=\frac{N_{\mathrm{sensitive}}}{N_{\mathrm{total}}},$$3 for Gemma-3-1B, $$B=\frac{N_{\mathrm{sensitive}}}{N_{\mathrm{total}}},$$4 for the older Gemma-7B non-it baseline, and $$B=\frac{N_{\mathrm{sensitive}}}{N_{\mathrm{total}}},$$5 for Gemma-3-12B. The Pearson correlation between model size and $$B=\frac{N_{\mathrm{sensitive}}}{N_{\mathrm{total}}},$$6 is $$B=\frac{N_{\mathrm{sensitive}}}{N_{\mathrm{total}}},$$7 with $$B=\frac{N_{\mathrm{sensitive}}}{N_{\mathrm{total}}},$$8, and the correlation between average accuracy and $$B=\frac{N_{\mathrm{sensitive}}}{N_{\mathrm{total}}},$$9 is $$G_r=\frac{N_{\mathrm{region}\,r}}{N_{\mathrm{geo\;questions}}},$$0 with $$G_r=\frac{N_{\mathrm{region}\,r}}{N_{\mathrm{geo\;questions}}},$$1. On Qwen3-14B, the domain averages are STEM $$G_r=\frac{N_{\mathrm{region}\,r}}{N_{\mathrm{geo\;questions}}},$$2, Humanities $$G_r=\frac{N_{\mathrm{region}\,r}}{N_{\mathrm{geo\;questions}}},$$3, Social Sciences $$G_r=\frac{N_{\mathrm{region}\,r}}{N_{\mathrm{geo\;questions}}},$$4, and Other $$G_r=\frac{N_{\mathrm{region}\,r}}{N_{\mathrm{geo\;questions}}},$$5 (Mishra et al., 4 Sep 2025).

The same work contrasts intra-model and inter-model agreement. Intra-model agreement computes $$G_r=\frac{N_{\mathrm{region}\,r}}{N_{\mathrm{geo\;questions}}},$$6 over all $$G_r=\frac{N_{\mathrm{region}\,r}}{N_{\mathrm{geo\;questions}}},$$7 language pairs for a single model; inter-model agreement computes $$G_r=\frac{N_{\mathrm{region}\,r}}{N_{\mathrm{geo\;questions}}},$$8 between pairs of models within the same language, yielding 28 model pairs per language for the 8-model pool. For every model, intra-model $$G_r=\frac{N_{\mathrm{region}\,r}}{N_{\mathrm{geo\;questions}}},$$9 is significantly higher than inter-model $\Delta A=A_{\mathrm{CA}}-A_{\mathrm{CS}}.$0 in the same language. For Qwen3-1.7B, the Mann–Whitney $\Delta A=A_{\mathrm{CA}}-A_{\mathrm{CS}}.$1 statistic is $\Delta A=A_{\mathrm{CA}}-A_{\mathrm{CS}}.$2 with $\Delta A=A_{\mathrm{CA}}-A_{\mathrm{CS}}.$3, and analogous significance holds across all model sizes with all $\Delta A=A_{\mathrm{CA}}-A_{\mathrm{CS}}.$4. High-resource languages such as English and Spanish also show higher inter-model $\Delta A=A_{\mathrm{CA}}-A_{\mathrm{CS}}.$5 than low-resource languages such as Amharic and Telugu, with correlation to log number of Wikipedia articles reported as $\Delta A=A_{\mathrm{CA}}-A_{\mathrm{CS}}.$6 with $\Delta A=A_{\mathrm{CA}}-A_{\mathrm{CS}}.$7 (Mishra et al., 4 Sep 2025).

Taken together, these results characterize GlobalMMLU as a substrate for measuring multilingual reliability rather than only multilingual accuracy. They suggest that larger models do not merely answer more questions correctly; they also converge toward more language-invariant decision patterns.

5. Reasoning–answer misalignment and Trace Inconsistency Rate

Another line of work uses GlobalMMLU to test whether chain-of-thought traces actually support the final answer selected by the model. The evaluated GlobalMMLU slice consists of 66,615 four-way multiple-choice questions across 6 languages—English, Spanish, Hindi, Arabic, Ukrainian, and Korean—with roughly 11,100 questions per language. The subject categories are global-facts, european-history, us-history, world-history, government-and-politics, and prehistory (Ovalle et al., 27 Dec 2025).

The evaluation framework isolates reasoning–answer alignment from task correctness. For each output in language $\Delta A=A_{\mathrm{CA}}-A_{\mathrm{CS}}.$8, the chain-of-thought trace $\Delta A=A_{\mathrm{CA}}-A_{\mathrm{CS}}.$9 and final answer $$\mathrm{Accuracy}=\frac{\#\;\mathrm{correct}}{\#\;\mathrm{total}}\times 100\%.$$0 are obtained; non-English traces are back-translated to English; the final answer is removed to form $$\mathrm{Accuracy}=\frac{\#\;\mathrm{correct}}{\#\;\mathrm{total}}\times 100\%.$$1; and annotators or an automated judge inspect $$\mathrm{Accuracy}=\frac{\#\;\mathrm{correct}}{\#\;\mathrm{total}}\times 100\%.$$2 together with the original question and options. They determine which option is supported by the trace, whether the trace is logically coherent, whether it contains sufficient information, and whether it includes problematic content. The core metric is Trace Inconsistency Rate:

$$\mathrm{Accuracy}=\frac{\#\;\mathrm{correct}}{\#\;\mathrm{total}}\times 100\%.$$3

The study also reports inter-annotator agreement: for Phase 2 human annotation, Cohen’s $$\mathrm{Accuracy}=\frac{\#\;\mathrm{correct}}{\#\;\mathrm{total}}\times 100\%.$$4 with 80% raw agreement on answer selection, $$\mathrm{Accuracy}=\frac{\#\;\mathrm{correct}}{\#\;\mathrm{total}}\times 100\%.$$5 with 69.7% agreement on logical coherence, and $$\mathrm{Accuracy}=\frac{\#\;\mathrm{correct}}{\#\;\mathrm{total}}\times 100\%.$$6 with 79% agreement on information sufficiency. Automated judge versus human consensus on $$\mathrm{Accuracy}=\frac{\#\;\mathrm{correct}}{\#\;\mathrm{total}}\times 100\%.$$7 gives $$\mathrm{Accuracy}=\frac{\#\;\mathrm{correct}}{\#\;\mathrm{total}}\times 100\%.$$8 and 62.5% agreement for answer selection, $$\mathrm{Accuracy}=\frac{\#\;\mathrm{correct}}{\#\;\mathrm{total}}\times 100\%.$$9 and 70.8% agreement for logical coherence, and $\sigma \approx 3.21$0 and 70.8% agreement for information sufficiency (Ovalle et al., 27 Dec 2025).

The principal empirical result is a script-level asymmetry. Latin-script languages, English and Spanish, have mean accuracy of approximately 86–88% and mean TIR of approximately 2.2–4.2%. Non-Latin scripts—Hindi, Arabic, Ukrainian, and Korean—show lower mean accuracy, approximately 78–82%, and higher mean TIR, approximately 5.8–12.6%. Across all six models, the Latin-script versus non-Latin-script gap is approximately 4–6 percentage points in accuracy and approximately 4–6 percentage points in TIR, with non-Latin scripts exhibiting at least twice as much misalignment. The study further reports that Qwen3-32B-thinking has the lowest TIR across all languages, with English TIR = 0.91% and Korean TIR = 8.50%, and that even when the final answer is incorrect, TIR often exceeds 10–25% in non-Latin scripts (Ovalle et al., 27 Dec 2025).

Human annotation yields an error taxonomy consisting of Illogical Leap, Logical Contradiction, Multiple Answers, Conflicting Facts, Unsupported Claims, Ambiguous Facts, Linguistic/Translation, and Irrelevant/Excessive. The authors state that failures stem primarily from evidential errors—unsupported claims and ambiguous facts—followed by illogical reasoning steps. This reframes what GlobalMMLU can measure: high answer accuracy does not guarantee that multilingual reasoning traces are internally supportive of the chosen answer. A plausible implication is that multilingual evaluation pipelines that report only final-choice accuracy are incomplete for chain-of-thought-capable models.

6. Limitations, methodological implications, and related benchmark development

GlobalMMLU-based studies identify several limitations. In the $\sigma \approx 3.21$1 framework, the metric applies only to multiple-choice tasks; free-form generation does not yet have an analogous accuracy-adjusted agreement metric. The $\sigma \approx 3.21$2 study is correlational and does not determine why larger models converge across languages, leaving open hypotheses involving more data, architecture, or parallel-corpus effects. Dataset coverage is also a constraint: the study explicitly names MMLU-ProX as an example of a benchmark that could broaden multilingual evaluation beyond current GlobalMMLU settings (Mishra et al., 4 Sep 2025).

The TIR study adds a different limitation: standard multilingual evaluation can be “right for the wrong reason.” Its recommended practices are to report both task accuracy and a reasoning consistency metric such as TIR, to use human-validated or strong automated judges for misalignment detection, to analyze script- and resource-level biases separately, and to investigate interventions such as reasoning-aware fine-tuning, self-verification, consistency regularization, and early stopping on incoherent traces (Ovalle et al., 27 Dec 2025).

Related benchmark development reinforces these directions. MMLU-ProX introduces a 29-language benchmark built on an English benchmark, with 11,829 identical questions per language in the full version and a 658-question lite version. It emphasizes strict parallelism, expert-reviewed multilingual translation, and direct cross-linguistic gap reporting through $\sigma \approx 3.21$3. Its stated goal is to realize a multilingual benchmark with rigorous construction, uniform protocol, and fine-grained analysis of language-family effects; the paper explicitly describes this as realizing what a truly “global” MMLU benchmark must look like, what the original Global MMLU set out to do at scale (Xuan et al., 13 Mar 2025).

Across these lines of work, GlobalMMLU occupies a dual role. It is, first, a multilingual benchmark with explicit translation curation and bias metadata; it is, second, a research platform for studying whether multilingual models preserve decision structure and reasoning quality across languages. The combined evidence suggests that multilingual evaluation must distinguish at least three axes—accuracy, cultural sensitivity, and cross-lingual functional consistency—if it is to provide a faithful account of model behavior.