MultiLoKo: Multilingual Local Knowledge Benchmark

• MultiLoKo is a multilingual benchmark comprising native and translated QA datasets derived from locally sourced questions on top Wikipedia pages across 31 languages.
• It differentiates between in-language and translated data to assess model performance, parity, and translation-induced evaluation artifacts using metrics like Exact Match and consistency.
• The benchmark reveals significant performance disparities, translation drops, and cross-lingual transfer challenges, highlighting the limitations of current multilingual LLMs.

MultiLoKo is a multilingual benchmark for evaluating the local-knowledge capabilities of LLMs across 31 languages. It combines native, locally sourced question–answer data with parallel translations in both directions, produced by humans and by machine, and is organized into a public dev split and a blind, out-of-distribution test split. The benchmark is designed not only to measure multilingual performance, but also to support analysis of performance parity across languages, the dependence of answers on question language, and the consequences of using translated rather than native evaluation data (Hupkes et al., 14 Apr 2025).

1. Definition and scope

MultiLoKo is centered on “local knowledge”: closed-form, short-answer questions that are locally relevant to a specific language community rather than globally salient facts. In the main partition, each language contributes 500 questions written from scratch about paragraphs sampled from the 6 000 most visited Wikipedia pages in that locale and judged locally relevant. The benchmark also includes translated counterparts in both directions between English and the 30 non-English languages, with both human-authored and machine-authored versions, enabling controlled analysis of multilingual transfer and benchmark-construction choices (Hupkes et al., 14 Apr 2025).

The benchmark covers the following 31 languages: Arabic, Bengali, Chinese (Traditional), Czech, Dutch, English, Farsi (Persian), French, German, Greek, Hebrew, Hindi, Hungarian, Indonesian, Italian, Japanese, Korean, Malay, Marathi, Polish, Portuguese, Romanian, Russian, Spanish, Swedish, Tagalog, Thai, Turkish, Ukrainian, Urdu, and Vietnamese. The language inventory spans high-resource and lower-resource settings and supports comparisons across different multilingual operating regimes.

A central design feature is the separation between in-language, locally authored data and translated data. This makes MultiLoKo suitable both for standard multilingual benchmarking and for meta-evaluation of multilingual benchmarks themselves, including whether English-translated evaluation over- or under-estimates true local-language performance (Hupkes et al., 14 Apr 2025).

2. Corpus composition and partitioning

The dataset is divided into one main partition and four translated partitions. The main partition contains 31 × 500 = 15 500 examples. Each translated partition contains 30 × 500 = 15 000 examples, since English serves as the anchor language in both translation directions (Hupkes et al., 14 Apr 2025).

Partition	Description	Size
Main	Locally sourced native questions for each language	15,500
Human-translated-to-English	30 non-English native questions translated into English	15,000
Human-translated-from-English	English native questions translated into each of the 30 other languages	15,000
Machine-translated-to-English	Same direction as above via Google Translate	15,000
Machine-translated-from-English	English-to-other via Google Translate	15,000

The data is equally distributed over two splits. The dev split contains 250 examples per language and is public. The test split contains 250 examples per language and is held out and blind. The split is explicitly designed to be out-of-distribution: the most frequently visited Wikipedia pages are placed in dev, and the least frequently visited pages are placed in test (Hupkes et al., 14 Apr 2025).

This partition structure serves two distinct purposes. First, it supports model evaluation on native, locally grounded questions. Second, it permits controlled comparisons between native and translated question formulations, as well as between human and machine translation regimes. A plausible implication is that MultiLoKo functions simultaneously as a task benchmark and as an instrument for benchmarking methodology.

3. Data construction pipeline

Question sourcing begins with paragraph selection. For each language, 6 000 top-viewed Wikipedia pages from 2016–2021 are sampled, and approximately 3 000-character snippets are extracted around a random word. Native annotators score each snippet on a 1–5 locality scale, where 1 denotes very local or obscure content and 5 denotes globally known content; snippets scoring above 3 are discarded (Hupkes et al., 14 Apr 2025).

Question generation proceeds in two stages. A pilot round of 50 questions per language is used to refine instructions. Annotators then produce 500 additional questions per language. These questions must be closed-form, require a single short answer, require synthesis across the snippet, and avoid yes/no or time-varying answers. Annotators also provide a longer explanation rationale and label the answer type, such as number, name, or date (Hupkes et al., 14 Apr 2025).

The review and validation stages are designed to enforce unambiguity and answerability. New annotators verify clarity, local relevance, correct answer sets, and correct language. Minor fixes, including alternative answer forms or time scopes, may be applied immediately; faulty questions are replaced. In a further validation stage, two independent annotators answer each question. If either annotator flags ambiguity or disagrees with the author’s answer list, the question is discarded. Validators may also add semantically equivalent answers such as “four” and “4” (Hupkes et al., 14 Apr 2025).

Translation is carried out in both directions. Human translations are commissioned from annotators, while machine translations are produced sentence-by-sentence with the Google Translate API. This yields four translated partitions and makes it possible to study both cross-linguistic knowledge transfer and translation-induced evaluation artifacts (Hupkes et al., 14 Apr 2025).

4. Scoring methodology

MultiLoKo uses Exact Match (EM) as its primary per-language scoring rule. For each language $\ell$ and partition $P$, EM is computed as the percentage of model outputs exactly matching one of the reference answers after minimal post-processing: lowercasing, punctuation stripping, and removal of a leading “Answer:” prefix (Hupkes et al., 14 Apr 2025).

Several aggregate and comparative statistics are then defined. Let $L = 31$ be the number of languages and let $M$ denote a model. The benchmark reports average EM across languages,

$$EM_{avg}(M) = \frac{1}{L}\sum_{\ell=1}^{L} EM_{\ell}(M),$$

and a parity statistic,

$$Gap = \max_{\ell} EM_{\ell} - \min_{\ell} EM_{\ell}.$$

The Gap measures dispersion between the best- and worst-performing languages for the same model (Hupkes et al., 14 Apr 2025).

To study language dependence more directly, MultiLoKo defines the Mother Tongue Effect (MTE) for non-English languages:

$$MTE_{\ell}(M) = EM_{\ell}^{local\text{-}in\text{-}\ell}(M) - EM_{\ell}^{same\text{-}questions\text{-}asked\text{-}in\text{-}English}(M).$$

A positive MTE means that a model answers local questions better in the local language than in English. The benchmark also defines the Locality Effect (LE),

$$LE_{\ell}(M) = EM_{\ell}^{English\text{-}translated}(M) - EM_{\ell}^{locally\text{-}sourced}(M),$$

where a positive value indicates that English-translated data overestimates true local performance (Hupkes et al., 14 Apr 2025).

Cross-lingual answer alignment is quantified with a consistency score on the locally sourced partition:

$$Consistency(M,\ell) = \frac{|\{i: correct_i(\ell) \wedge correct_i(EN)\}|}{|\{i: correct_i(\ell) \vee correct_i(EN)\}|}.$$

This statistic measures the overlap between the sets of questions answered correctly in the local language and in English. It is designed to distinguish mere aggregate accuracy from stable multilingual retrieval or transfer of the same underlying fact (Hupkes et al., 14 Apr 2025).

5. Evaluation on multilingual LLMs

The benchmark paper evaluates 11 base and chat models marketed as multilingual: Llama 3.1 (70B and 405B, base and chat), Mixtral 8x22B (base and instruct), Qwen 2.5 72B (base and instruct), Gemini 2.0 Flash, GPT4-o, and Claude 3.5 Sonnet. All runs use temperature $= 0$; base models are prompted with 5-shot prompts and chat models with 0-shot prompts (Hupkes et al., 14 Apr 2025).

On the dev split of the locally sourced partition, the highest average EM scores are Gemini 2.0 Flash at $P$0, Llama 3.1 405B at $P$1, and GPT4-o at $P$2. All models remain below $P$3 $P$4, and the paper emphasizes that even the strongest systems leave more than $P$5 of local-knowledge questions unanswered correctly. Language-level disparity is also substantial: the reported Gaps range from approximately 29 points for Llama 3.1 70B base to approximately 49 points for GPT4-o (Hupkes et al., 14 Apr 2025).

The Mother Tongue Effect is generally positive. Most models show an average non-English MTE of approximately 3 to 6 points, indicating better performance when local questions are asked in the native language than when the same questions are asked in English. At the same time, MTE varies markedly by model and language; for Gemini 2.0, the distribution ranges from $P$6 to $P$7 across languages (Hupkes et al., 14 Apr 2025).

Consistency is limited. The benchmark reports that even when models answer correctly in English roughly $P$8 of the time and in the local language roughly $P$9 of the time, only roughly $L = 31$0 of questions answered correctly in one language are answered correctly in both. The top models achieve at best approximately $L = 31$1 consistency, which the paper interprets as evidence of suboptimal cross-lingual knowledge alignment (Hupkes et al., 14 Apr 2025).

Language-wise difficulty is uneven. The easiest languages by $L = 31$2 are reported as Tagalog, French, Spanish, and English, while the hardest are reported as Farsi, Khmer, Malay, and other mid/low-resource languages. When locally sourced data is translated into English, native-language difficulty correlates with translated difficulty at approximately $L = 31$3, but some languages, including Bengali, Urdu, and Hindi, become much easier in English, indicating language-proficiency effects in addition to factual knowledge effects (Hupkes et al., 14 Apr 2025).

6. Translation effects, benchmark interpretation, and subsequent use

One of MultiLoKo’s principal methodological findings concerns the difference between local and translated evaluation. Per-language scores differ by up to $L = 31$4 points between the locally sourced and English-translated partitions, and the average absolute LE magnitude is approximately 15 points across models. The language-difficulty rank correlation between the two regimes ranges from 0.54 to 0.88 across models, indicating that translation can substantially reorder which languages appear difficult. By contrast, model rankings by $L = 31$5 are reported as virtually unchanged, with rank correlation approximately 1.0. This establishes a distinction between model comparison and per-language diagnosis: translated data may be sufficient for the former, but is risky for the latter (Hupkes et al., 14 Apr 2025).

Human versus machine translation yields a related but distinct pattern. All models experience systematic EM drops on machine-translated compared with human-translated data for the same questions. The average drop per model ranges from 2% up to 34% of the human-translated score, and maximum per-language drops exceed 20 points in some cases. Nevertheless, language-difficulty ranks remain highly correlated at approximately $L = 31$6, and model rankings are stable. The paper therefore characterizes machine translation as introducing a consistent under-estimation rather than radically reshuffling leaderboards, while noting that some languages, including Russian, Swedish, and Urdu, occasionally show gains under machine translation (Hupkes et al., 14 Apr 2025).

MultiLoKo has also been used as an analytic substrate in later work on cross-lingual knowledge transfer. A 2026 study of reasoning LLMs analyzes MultiLoKo alongside ECLeKTic and concludes that script match, rather than language family or exact source–target language match, is the primary predictor of transfer failure after controlling for model capability and question difficulty. In an Ordinary Least Squares analysis on MultiLoKo, the estimated coefficient for script match is $L = 31$7 with $L = 31$8, corresponding to a $L = 31$9 percentage-point boost in answer accuracy for same-script cases; family match and language match are reported as not significant. The same study further reports that prefixing questions with key entities in their source language disproportionately improves cross-script performance and that LoRA SFT targeted at transliteration-aware reasoning reduces the cross-script gap (Bandarkar et al., 17 Mar 2026).

Taken together, these findings position MultiLoKo as both a benchmark and a research instrument. The benchmark documents that contemporary multilingual LLMs remain weak on locally relevant factual questions, with low average EM and large inter-language disparities. At the same time, its paired native and translated partitions make it possible to isolate language effects, translation artifacts, and cross-script transfer failures with considerably more precision than monolingual or translation-only evaluation setups (Hupkes et al., 14 Apr 2025).