---
title: 'MultiWikiQA: Multilingual Extractive QA Benchmark'
url: https://www.emergentmind.com/topics/multiwikiqa
type: topic
---

# MultiWikiQA: Multilingual Extractive QA Benchmark

MultiWikiQA is a multilingual extractive reading comprehension benchmark in 306 languages whose contexts come from Wikipedia articles, whose questions are generated by an LLM, and whose answers appear verbatim in the Wikipedia context [2509.04111]. It was introduced to address the limited language coverage of prior multilingual reading comprehension datasets and to provide a span-grounded benchmark for document-level reading comprehension and retrieval-augmented generation pipelines across typologically diverse and very low-resource languages [2509.04111]. Subsequent work has used the dataset as the backbone for multilingual faithfulness hallucination evaluation, exploiting its extractive structure for token-level grounding [2605.02504].

## 1. Scope and conceptual positioning

MultiWikiQA was created to fill a gap in multilingual reading comprehension evaluation: existing multilingual RC datasets typically span between 3 and 26 languages, while Belebele covers 122 languages but uses short multiple-choice passages, is abstractive, and has only 900 samples per language [2509.04111]. MultiWikiQA instead uses an extractive QA format with answers guaranteed to appear verbatim in the context and scales to 306 languages, which makes it suitable for span-based readers and for RC components of RAG pipelines [2509.04111].

The benchmark is explicitly grounded in Wikipedia. The source pool comprises 315 Wikipedia language editions from the 20231101 snapshot, with special handling for Mandarin and Portuguese: Mandarin is split into `zh-cn` and `zh-tw` using HanzIdentifier, and Portuguese is split into `pt-br` and `pt-pt` using the PtVid classifier [2509.04111]. The resulting language inventory spans Latin, Cyrillic, Arabic, Hebrew, Devanagari and other Indic scripts, Southeast Asian scripts such as Thai, Lao, and Burmese, Ethiopic, Georgian, Armenian, Kana/Kanji, and logographic Han characters, and includes many low-resource languages such as Bashkir, Kara-Kalpak, Faroese, Ossetian, Udmurt, and Icelandic [2509.04111].

A central property of MultiWikiQA is that it is extractive rather than generative. In later faithfulness work, each instance is described as a language-specific context, a question, and a gold answer that appears verbatim in the context, which is crucial for precise token-level grounding and hallucination detection [2605.02504]. The multilingual structure is language-local rather than parallel: cross-language comparability comes from using the same task structure and splits per language rather than translated or aligned prompts [2605.02504].

## 2. Construction pipeline

The context source is the Wikipedia 20231101 dump from `https://hf.co/datasets/wikimedia/wikipedia`, with question generation proceeding per language edition until 5,000 context–question–answer triples are produced or the available articles are exhausted [2509.04111]. Question generation uses Gemini-1.5-pro with `temperature = 1.0` and up to 1,000 generated tokens per call [2509.04111]. The model is prompted to produce 2–10 question-answer pairs per article in strict JSON format, under several constraints: all questions and answers must be in the target language, questions must be about the given article, each answer must appear exactly as written in the article including casing, answers must be as short as possible, answers must be different from each other, and output must be a JSON object with key `"results"` where each entry has `"question"` and `"answer"` [2509.04111].

A post-generation filter verifies the presence of the `"question"` and `"answer"` keys and checks that the answer string appears verbatim in the context document [2509.04111]. No synonym expansion or normalization of the answer is permitted, and casing must match the source [2509.04111]. This design enforces exact span grounding and avoids ambiguity about whether an answer is actually supported by the context.

To reduce direct wording overlap between question and context, the same LLM is prompted to rephrase each question as much as possible without access to the context while preserving meaning through synonyms, paraphrases, or different valid word order [2509.04111]. The output is JSON with a single `"question"` key [2509.04111]. This rephrasing step is motivated by a common criticism of reading comprehension datasets in which direct lexical overlap enables superficial matching rather than genuine reading comprehension [2509.04111].

The benchmark follows a SQuAD-style schema. Each example contains a context, a question, and an answer that appears verbatim in the context; in SQuAD-style datasets, answers typically include character offsets and text, and MultiWikiQA adheres to this format [2509.04111].

## 3. Coverage, scale, and data regime

MultiWikiQA attempts to cap each language at 5,000 context–question–answer triples [2509.04111]. For 101 languages, the pipeline ran out of articles before reaching the cap; for the remaining languages, the cap was reached [2509.04111]. The paper provides exact counts for all languages below 5,000. Examples from the upper end of the “small subsets” include `lld: 4,745`, `tn: 4,744`, `pcm: 4,623`, `gcr: 4,590`, `fat: 4,539`, `om: 4,458`, `av: 4,375`, `se: 4,257`, `tum: 4,252`, and `gpe: 4,242`; at the lower end are `cu: 443`, `za: 427`, `ki: 416`, `tpi: 397`, `ti: 385`, `got: 383`, `ady: 380`, `lbe: 378`, `ve: 369`, `srn: 321`, `kg: 261`, `arc: 251`, `chr: 185`, `bi: 149`, `iu: 148`, `ch: 135`, `ty: 129`, `bug: 119`, `sg: 83`, `pi: 79`, `ik: 67`, `cr: 33`, and `chy: 25` [2509.04111].

The theoretical upper bound on total size is \(306 \times 5{,}000 = 1{,}530{,}000\) QA pairs, but the actual aggregate count is lower because 101 languages did not reach the cap, and the exact aggregate total is not reported [2509.04111]. Average context length and answer length are also not reported [2509.04111].

For model evaluation, languages are included only if the dataset provides at least 1,024 training samples, 32 validation samples, and 128 test samples [2509.04111]. The text states that 264 languages meet this criterion, while the evaluation tables report results for 261 languages [2509.04111]. This thresholded evaluation regime preserves very broad coverage while excluding the smallest Wikipedias from the main benchmark tables.

The language distribution is therefore intentionally broad but not uniform. A plausible implication is that the benchmark evaluates both mainstream multilingual capability and robustness under data scarcity, because it combines high-resource Wikipedias with extremely small ones within a single extractive QA framework.

## 4. Metrics, baselines, and empirical difficulty

The primary reported metric is token-level F1 [2509.04111]. The paper also provides standard extractive QA definitions:

\[
EM = \frac{1}{N} \sum_{i=1}^{N} 1\{ \text{normalize}(\hat{y}_i) = \text{normalize}(y_i) \},
\]

\[
P_i = \frac{|A_i \cap \hat{A}_i|}{|\hat{A}_i|}, \quad
R_i = \frac{|A_i \cap \hat{A}_i|}{|A_i|}, \quad
F1_i = \frac{2 P_i R_i}{P_i + R_i},
\]

\[
F1_{\text{macro}} = \frac{1}{L} \sum_{\ell=1}^{L} F1_\ell,
\]

\[
F1_{\text{micro}} = \frac{2 \sum_i |A_i \cap \hat{A}_i|}{\sum_i |A_i| + \sum_i |\hat{A}_i|}.
\]

The paper does not specify tokenization details, and for languages without whitespace it does not detail whether EuroEval uses character-based scoring [2509.04111].

Evaluation is conducted with EuroEval [2509.04111]. Decoder models are evaluated in a 2-shot setting using few-shot examples drawn from the training split, while encoder models are fine-tuned on the training split with early stopping on the validation split and reported on the test split [2509.04111]. Six models are evaluated.

| Model | Type | Mean F1 |
|---|---|---|
| Mistral-Small-3.1-24B-Instruct-2503 | Decoder, instruction-tuned | 55.83% ± 1.09% |
| Mistral-Small-3.1-24B-Base-2503 | Decoder, base | 54.71% ± 1.20% |
| Llama-3.1-8B-Instruct | Decoder, instruction-tuned | 52.38% ± 0.91% |
| Llama-3.1-8B | Decoder, base | 47.26% ± 1.22% |
| Multilingual-E5-large | Encoder | 23.82% ± 0.65% |
| XLM-RoBERTa-large | Encoder | 20.23% ± 0.69% |

The reported pattern is consistent across model classes: instruction-tuned decoders outperform base decoders, and both substantially outperform fine-tuned encoders on this benchmark [2509.04111]. The benchmark is described as challenging and far from saturated across languages [2509.04111].

Large language discrepancies are a defining empirical result. Best-performing languages include Welsh, where decoders reach roughly 90% F1, Scandinavian and other Germanic languages such as Swedish, Norwegian Nynorsk, Norwegian, Dutch, and German at roughly 70–80% F1, and several Romance languages such as Galician, Catalan, Spanish, Italian, and Romanian at roughly 65–78% F1 [2509.04111]. Lower-performing languages include Dzongkha at roughly 1–6%, Tibetan at roughly 0–4%, Meitei at roughly 1–3%, N’Ko at roughly 2–3%, Santali at roughly 4–5%, Burmese/Shan-related varieties such as Mon and Shan at roughly 5–21% and roughly 3–16%, and some African languages such as Wolof, Ganda, and Tsonga [2509.04111]. The paper does not formally analyze the causes, but hypothesizes script segmentation challenges, morphology, limited tokenizer coverage, scarcity and noisiness of Wikipedia content, longer contexts, and mismatch between pretraining corpora and target languages [2509.04111].

## 5. Human evaluation, quality assurance, and stated limitations

Question quality is evaluated through crowdsourced fluency surveys conducted via Microsoft Forms [2509.04111]. Each language-specific survey contains a random sample of 50 generated questions, and annotators rate fluency on a 3-star scale where 1 means “not natural,” 2 means “mostly natural (some part wrong),” and 3 means “natural” [2509.04111]. The survey preamble explicitly focuses on fluency rather than answerability [2509.04111].

The study reports 156 responses across 30 different languages [2509.04111]. Although numerical per-language means are not tabulated, the reported result is that mean fluency ratings exceed 2.0, corresponding to “mostly natural,” even for very low-resource languages such as Bashkir, Kara-Kalpak, Faroese, Ossetian, Udmurt, and Icelandic [2509.04111]. The key conclusion is that LLM-generated questions are generally judged to be fluent across a diverse subset of languages [2509.04111].

The paper also states several limitations. Human fluency evaluation covers 30 languages, approximately 10% of the full 306-language benchmark, so generalization to the remaining languages is plausible but not guaranteed [2509.04111]. Inter-annotator agreement and confidence intervals are not reported [2509.04111]. The paper additionally lists several likely failure modes, given the results and the pipeline, though these are not exhaustively analyzed: long contexts and answer span localization under limited in-context examples; ambiguity in rephrased questions; tokenization issues for languages without whitespace or with complex orthographies; normalization mismatches in scoring; and low-resource Wikipedias yielding shorter, less informative contexts or limited domain coverage [2509.04111].

The dataset and survey evaluations are described as freely available, although repository URLs are redacted in the paper version summarized here [2509.04111].

## 6. MultiWikiQA in multilingual faithfulness and hallucination research

MultiWikiQA serves as the backbone of MultiWikiQHalluA, a multilingual hallucination benchmark that uses the extractive reading comprehension setting for faithfulness evaluation [2605.02504]. In this usage, each instance comprises a language-specific context from Wikipedia, a question, and a gold answer that appears verbatim in the context [2605.02504]. The extractive property is essential because it permits precise token-level grounding and token-level hallucination classification [2605.02504].

MultiWikiQHalluA uses the full language support of MultiWikiQA, namely 306 languages, with a uniform split per language of 4,000 context–question–answer triples for training and 1,000 for testing [2605.02504]. The benchmark plugs these instances into LettuceDetect’s generation pipeline, replacing RagTruth with MultiWikiQA inputs and using RAGFactChecker to rewrite the gold answer into a false-but-plausible answer according to a sampled hallucination intensity [2605.02504]. The intensity \(I\) is drawn from a beta distribution with mean 0.2 and standard deviation 0.15, skewing toward subtle errors [2605.02504]. The paper defines five intensity-dependent regimes: \(I \le 0.2\) for very subtle errors that are hard to detect, \(I \le 0.4\) for moderate errors that are noticeable but plausible, \(I \le 0.6\) for clear errors that are obviously incorrect, \(I \le 0.8\) for strong errors that significantly change meaning, and \(I > 0.8\) for extreme errors that completely contradict the original [2605.02504]. The adopted error types are factual, temporal, and numerical [2605.02504].

The synthetic generation process returns both the rewritten text and span annotations of changed regions as character intervals \([(s_1,e_1),(s_2,e_2),\ldots]\), which are projected to subword tokens [2605.02504]. If \(S=\{(s_i,e_i)\}\) is the set of hallucinated spans and token \(t\) covers \([a_t,b_t)\), then the token label is defined as

\[
h_t = 1 \text{ if } \exists (s_i,e_i)\in S \text{ such that } \max(a_t,s_i) < \min(b_t,e_i); \text{ otherwise } h_t = 0.
\]

This yields, per language, 5,000 hallucinated samples, split into 4,000 training and 1,000 testing examples [2605.02504]. These are paired with the original MultiWikiQA samples, so every instance appears twice—one true and one hallucinated—giving a balanced supervision signal for binary token-level classification [2605.02504].

For classifier selection, the authors fine-tuned three multilingual encoders on Danish and German and found that mmBERT-small (140M) performed best in both languages [2605.02504]. Its reported test-set results on 1,000 samples are: Danish Supported-F1 0.9143, Unsupported-F1 0.8689, Accuracy 0.8963; German Supported-F1 0.9147, Unsupported-F1 0.8627, Accuracy 0.8948 [2605.02504]. It was then used to train token-level hallucination classifiers for 30 European languages [2605.02504].

The benchmark defines two principal hallucination metrics:

\[
HR_\ell = \frac{1}{T_\ell} \sum_{t=1}^{T_\ell} h_t,
\]

where \(T_\ell\) is the total number of tokens across all answers in language \(\ell\), and

\[
AR_\ell = \frac{1}{N_\ell} \sum_{j=1}^{N_\ell} \mathbf{1}[\exists t \in A_j : h_t = 1],
\]

where \(N_\ell\) is the number of answers in the test set for language \(\ell\) [2605.02504].

When five LLMs are evaluated on English, Danish, German, and Icelandic, the reported rates show consistent language and model-size effects [2605.02504]. For token-level hallucination rate \(HR\), Qwen3-0.6B records EN 0.03, DA 0.17, DE 0.09, IS 0.36; Qwen3-14B records EN 0.01, DA 0.08, DE 0.03, IS 0.17; Gemma-3-12B-IT records EN 0.02, DA 0.08, DE 0.05, IS 0.20; Cogito-Qwen-32B records EN 0.01, DA 0.07, DE 0.05, IS 0.18; and Cogito-Llama-70B records EN 0.02, DA 0.07, DE 0.05, IS 0.15 [2605.02504]. For answer-level rate \(AR\), the same order yields EN 0.07, DA 0.52, DE 0.17, IS 0.60; EN 0.02, DA 0.12, DE 0.04, IS 0.26; EN 0.03, DA 0.13, DE 0.06, IS 0.27; EN 0.01, DA 0.09, DE 0.06, IS 0.18; and EN 0.03, DA 0.08, DE 0.06, IS 0.19 [2605.02504]. The most prominent result is the Icelandic peak for Qwen3-0.6B, where up to 60% of answers contain at least one hallucination [2605.02504]. The paper interprets this as evidence that hallucination rates are consistently higher for lower-resource languages, particularly Icelandic, while larger models generally hallucinate less [2605.02504].

## 7. Relation to MegaWika and multilingual KGQA

MultiWikiQA occupies the text-based extractive QA end of multilingual wiki-derived question answering, but it sits within a broader ecosystem that includes source-grounded QA over Wikipedia citations and structured KGQA over wiki-derived knowledge graphs. MegaWika introduces 13 million Wikipedia articles in 50 diverse languages together with 71 million referenced source materials, aligned at the sentence level as “reports” paired with citations and harvested sources [2307.07049]. Although the exact term “MultiWikiQA” is not used there, MegaWika is explicitly positioned for cross-lingual QA and related knowledge-intensive tasks through sentence-level claim–citation–source triplets, English translations for non-English content, and a QA baseline comparison section [2307.07049]. Its natural QA use cases include retrieving relevant context from article text or cited sources, extracting or generating answers, and transferring QA models between languages [2307.07049]. A plausible implication is that MegaWika extends the evidence space beyond the Wikipedia passage itself, whereas MultiWikiQA constrains the task to extractive reading over Wikipedia context.

QALD-9-plus represents a structurally different strand of multilingual wiki-based QA: knowledge graph question answering over DBpedia and Wikidata [2202.00120]. It extends QALD-9 by translating English questions into eight additional languages by native speakers and transferring the original SPARQL queries from DBpedia to Wikidata, yielding a benchmark with 558 unique English questions, 4,930 new multilingual question translations, and 507 new Wikidata queries [2202.00120]. The supported languages are English, German, French, Russian, Ukrainian, Lithuanian, Belarusian, Bashkir, and Armenian [2202.00120]. Unlike MultiWikiQA, which requires extracting verbatim answers from unstructured Wikipedia text, QALD-9-plus evaluates mapping natural-language questions to SPARQL over structured wiki-derived knowledge graphs [2202.00120].

This distinction is methodologically important. Text-based QA resources such as MultiWikiQA, MLQA, MKQA, TyDi QA, and XQuAD operate over unstructured text and answer spans, whereas KGQA benchmarks such as QALD-9-plus require semantic parsing into executable graph queries [2202.00120]. MegaWika, meanwhile, introduces cited external sources as a second evidence layer beyond Wikipedia itself [2307.07049]. Taken together, these resources define complementary formulations of multilingual wiki-centered QA: extractive reading comprehension over article text, source-grounded evidence retrieval and answering over article-plus-citation graphs, and multilingual KGQA over DBpedia and Wikidata.

Source: https://www.emergentmind.com/topics/multiwikiqa