MultiWikiQA: Multilingual Extractive QA Benchmark

• MultiWikiQA is a multilingual extractive reading comprehension benchmark that leverages verbatim Wikipedia contexts across 306 languages.
• It uses an LLM-driven pipeline to generate language-specific questions with strict JSON formatting and extractive answer matching.
• The benchmark supports diverse evaluations, including token-level F1 metrics and multilingual hallucination analysis, enhancing cross-lingual QA robustness.

MultiWikiQA is a multilingual extractive reading comprehension benchmark in 306 languages whose contexts come from Wikipedia articles, whose questions are generated by an LLM, and whose answers appear verbatim in the Wikipedia context (Smart, 4 Sep 2025). It was introduced to address the limited language coverage of prior multilingual reading comprehension datasets and to provide a span-grounded benchmark for document-level reading comprehension and retrieval-augmented generation pipelines across typologically diverse and very low-resource languages (Smart, 4 Sep 2025). Subsequent work has used the dataset as the backbone for multilingual faithfulness hallucination evaluation, exploiting its extractive structure for token-level grounding (Thoresen et al., 4 May 2026).

1. Scope and conceptual positioning

MultiWikiQA was created to fill a gap in multilingual reading comprehension evaluation: existing multilingual RC datasets typically span between 3 and 26 languages, while Belebele covers 122 languages but uses short multiple-choice passages, is abstractive, and has only 900 samples per language (Smart, 4 Sep 2025). MultiWikiQA instead uses an extractive QA format with answers guaranteed to appear verbatim in the context and scales to 306 languages, which makes it suitable for span-based readers and for RC components of RAG pipelines (Smart, 4 Sep 2025).

The benchmark is explicitly grounded in Wikipedia. The source pool comprises 315 Wikipedia language editions from the 20231101 snapshot, with special handling for Mandarin and Portuguese: Mandarin is split into zh-cn and zh-tw using HanzIdentifier, and Portuguese is split into pt-br and pt-pt using the PtVid classifier (Smart, 4 Sep 2025). The resulting language inventory spans Latin, Cyrillic, Arabic, Hebrew, Devanagari and other Indic scripts, Southeast Asian scripts such as Thai, Lao, and Burmese, Ethiopic, Georgian, Armenian, Kana/Kanji, and logographic Han characters, and includes many low-resource languages such as Bashkir, Kara-Kalpak, Faroese, Ossetian, Udmurt, and Icelandic (Smart, 4 Sep 2025).

A central property of MultiWikiQA is that it is extractive rather than generative. In later faithfulness work, each instance is described as a language-specific context, a question, and a gold answer that appears verbatim in the context, which is crucial for precise token-level grounding and hallucination detection (Thoresen et al., 4 May 2026). The multilingual structure is language-local rather than parallel: cross-language comparability comes from using the same task structure and splits per language rather than translated or aligned prompts (Thoresen et al., 4 May 2026).

2. Construction pipeline

The context source is the Wikipedia 20231101 dump from https://hf.co/datasets/wikimedia/wikipedia, with question generation proceeding per language edition until 5,000 context–question–answer triples are produced or the available articles are exhausted (Smart, 4 Sep 2025). Question generation uses Gemini-1.5-pro with temperature = 1.0 and up to 1,000 generated tokens per call (Smart, 4 Sep 2025). The model is prompted to produce 2–10 question-answer pairs per article in strict JSON format, under several constraints: all questions and answers must be in the target language, questions must be about the given article, each answer must appear exactly as written in the article including casing, answers must be as short as possible, answers must be different from each other, and output must be a JSON object with key "results" where each entry has "question" and "answer" (Smart, 4 Sep 2025).

A post-generation filter verifies the presence of the "question" and "answer" keys and checks that the answer string appears verbatim in the context document (Smart, 4 Sep 2025). No synonym expansion or normalization of the answer is permitted, and casing must match the source (Smart, 4 Sep 2025). This design enforces exact span grounding and avoids ambiguity about whether an answer is actually supported by the context.

To reduce direct wording overlap between question and context, the same LLM is prompted to rephrase each question as much as possible without access to the context while preserving meaning through synonyms, paraphrases, or different valid word order (Smart, 4 Sep 2025). The output is JSON with a single "question" key (Smart, 4 Sep 2025). This rephrasing step is motivated by a common criticism of reading comprehension datasets in which direct lexical overlap enables superficial matching rather than genuine reading comprehension (Smart, 4 Sep 2025).

The benchmark follows a SQuAD-style schema. Each example contains a context, a question, and an answer that appears verbatim in the context; in SQuAD-style datasets, answers typically include character offsets and text, and MultiWikiQA adheres to this format (Smart, 4 Sep 2025).

3. Coverage, scale, and data regime

MultiWikiQA attempts to cap each language at 5,000 context–question–answer triples (Smart, 4 Sep 2025). For 101 languages, the pipeline ran out of articles before reaching the cap; for the remaining languages, the cap was reached (Smart, 4 Sep 2025). The paper provides exact counts for all languages below 5,000. Examples from the upper end of the “small subsets” include lld: 4,745, tn: 4,744, pcm: 4,623, gcr: 4,590, fat: 4,539, om: 4,458, av: 4,375, se: 4,257, tum: 4,252, and gpe: 4,242; at the lower end are cu: 443, za: 427, ki: 416, tpi: 397, ti: 385, got: 383, ady: 380, lbe: 378, ve: 369, srn: 321, kg: 261, arc: 251, chr: 185, bi: 149, iu: 148, ch: 135, ty: 129, bug: 119, sg: 83, pi: 79, ik: 67, cr: 33, and chy: 25 (Smart, 4 Sep 2025).

The theoretical upper bound on total size is $306 \times 5{,}000 = 1{,}530{,}000$ QA pairs, but the actual aggregate count is lower because 101 languages did not reach the cap, and the exact aggregate total is not reported (Smart, 4 Sep 2025). Average context length and answer length are also not reported (Smart, 4 Sep 2025).

For model evaluation, languages are included only if the dataset provides at least 1,024 training samples, 32 validation samples, and 128 test samples (Smart, 4 Sep 2025). The text states that 264 languages meet this criterion, while the evaluation tables report results for 261 languages (Smart, 4 Sep 2025). This thresholded evaluation regime preserves very broad coverage while excluding the smallest Wikipedias from the main benchmark tables.

The language distribution is therefore intentionally broad but not uniform. A plausible implication is that the benchmark evaluates both mainstream multilingual capability and robustness under data scarcity, because it combines high-resource Wikipedias with extremely small ones within a single extractive QA framework.

4. Metrics, baselines, and empirical difficulty

The primary reported metric is token-level F1 (Smart, 4 Sep 2025). The paper also provides standard extractive QA definitions:

$$EM = \frac{1}{N} \sum_{i=1}^{N} 1\{ \text{normalize}(\hat{y}_i) = \text{normalize}(y_i) \},$$

$$P_i = \frac{|A_i \cap \hat{A}_i|}{|\hat{A}_i|}, \quad R_i = \frac{|A_i \cap \hat{A}_i|}{|A_i|}, \quad F1_i = \frac{2 P_i R_i}{P_i + R_i},$$

$$F1_{\text{macro}} = \frac{1}{L} \sum_{\ell=1}^{L} F1_\ell,$$

$$F1_{\text{micro}} = \frac{2 \sum_i |A_i \cap \hat{A}_i|}{\sum_i |A_i| + \sum_i |\hat{A}_i|}.$$

The paper does not specify tokenization details, and for languages without whitespace it does not detail whether EuroEval uses character-based scoring (Smart, 4 Sep 2025).

Evaluation is conducted with EuroEval (Smart, 4 Sep 2025). Decoder models are evaluated in a 2-shot setting using few-shot examples drawn from the training split, while encoder models are fine-tuned on the training split with early stopping on the validation split and reported on the test split (Smart, 4 Sep 2025). Six models are evaluated.

Model	Type	Mean F1
Mistral-Small-3.1-24B-Instruct-2503	Decoder, instruction-tuned	55.83% ± 1.09%
Mistral-Small-3.1-24B-Base-2503	Decoder, base	54.71% ± 1.20%
Llama-3.1-8B-Instruct	Decoder, instruction-tuned	52.38% ± 0.91%
Llama-3.1-8B	Decoder, base	47.26% ± 1.22%
Multilingual-E5-large	Encoder	23.82% ± 0.65%
XLM-RoBERTa-large	Encoder	20.23% ± 0.69%

The reported pattern is consistent across model classes: instruction-tuned decoders outperform base decoders, and both substantially outperform fine-tuned encoders on this benchmark (Smart, 4 Sep 2025). The benchmark is described as challenging and far from saturated across languages (Smart, 4 Sep 2025).

Large language discrepancies are a defining empirical result. Best-performing languages include Welsh, where decoders reach roughly 90% F1, Scandinavian and other Germanic languages such as Swedish, Norwegian Nynorsk, Norwegian, Dutch, and German at roughly 70–80% F1, and several Romance languages such as Galician, Catalan, Spanish, Italian, and Romanian at roughly 65–78% F1 (Smart, 4 Sep 2025). Lower-performing languages include Dzongkha at roughly 1–6%, Tibetan at roughly 0–4%, Meitei at roughly 1–3%, N’Ko at roughly 2–3%, Santali at roughly 4–5%, Burmese/Shan-related varieties such as Mon and Shan at roughly 5–21% and roughly 3–16%, and some African languages such as Wolof, Ganda, and Tsonga (Smart, 4 Sep 2025). The paper does not formally analyze the causes, but hypothesizes script segmentation challenges, morphology, limited tokenizer coverage, scarcity and noisiness of Wikipedia content, longer contexts, and mismatch between pretraining corpora and target languages (Smart, 4 Sep 2025).

5. Human evaluation, quality assurance, and stated limitations

Question quality is evaluated through crowdsourced fluency surveys conducted via Microsoft Forms (Smart, 4 Sep 2025). Each language-specific survey contains a random sample of 50 generated questions, and annotators rate fluency on a 3-star scale where 1 means “not natural,” 2 means “mostly natural (some part wrong),” and 3 means “natural” (Smart, 4 Sep 2025). The survey preamble explicitly focuses on fluency rather than answerability (Smart, 4 Sep 2025).

The study reports 156 responses across 30 different languages (Smart, 4 Sep 2025). Although numerical per-language means are not tabulated, the reported result is that mean fluency ratings exceed 2.0, corresponding to “mostly natural,” even for very low-resource languages such as Bashkir, Kara-Kalpak, Faroese, Ossetian, Udmurt, and Icelandic (Smart, 4 Sep 2025). The key conclusion is that LLM-generated questions are generally judged to be fluent across a diverse subset of languages (Smart, 4 Sep 2025).

The paper also states several limitations. Human fluency evaluation covers 30 languages, approximately 10% of the full 306-language benchmark, so generalization to the remaining languages is plausible but not guaranteed (Smart, 4 Sep 2025). Inter-annotator agreement and confidence intervals are not reported (Smart, 4 Sep 2025). The paper additionally lists several likely failure modes, given the results and the pipeline, though these are not exhaustively analyzed: long contexts and answer span localization under limited in-context examples; ambiguity in rephrased questions; tokenization issues for languages without whitespace or with complex orthographies; normalization mismatches in scoring; and low-resource Wikipedias yielding shorter, less informative contexts or limited domain coverage (Smart, 4 Sep 2025).

The dataset and survey evaluations are described as freely available, although repository URLs are redacted in the paper version summarized here (Smart, 4 Sep 2025).

6. MultiWikiQA in multilingual faithfulness and hallucination research

MultiWikiQA serves as the backbone of MultiWikiQHalluA, a multilingual hallucination benchmark that uses the extractive reading comprehension setting for faithfulness evaluation (Thoresen et al., 4 May 2026). In this usage, each instance comprises a language-specific context from Wikipedia, a question, and a gold answer that appears verbatim in the context (Thoresen et al., 4 May 2026). The extractive property is essential because it permits precise token-level grounding and token-level hallucination classification (Thoresen et al., 4 May 2026).

MultiWikiQHalluA uses the full language support of MultiWikiQA, namely 306 languages, with a uniform split per language of 4,000 context–question–answer triples for training and 1,000 for testing (Thoresen et al., 4 May 2026). The benchmark plugs these instances into LettuceDetect’s generation pipeline, replacing RagTruth with MultiWikiQA inputs and using RAGFactChecker to rewrite the gold answer into a false-but-plausible answer according to a sampled hallucination intensity (Thoresen et al., 4 May 2026). The intensity $I$ is drawn from a beta distribution with mean 0.2 and standard deviation 0.15, skewing toward subtle errors (Thoresen et al., 4 May 2026). The paper defines five intensity-dependent regimes: $I \le 0.2$ for very subtle errors that are hard to detect, $I \le 0.4$ for moderate errors that are noticeable but plausible, $I \le 0.6$ for clear errors that are obviously incorrect, $I \le 0.8$ for strong errors that significantly change meaning, and $$EM = \frac{1}{N} \sum_{i=1}^{N} 1\{ \text{normalize}(\hat{y}_i) = \text{normalize}(y_i) \},$$0 for extreme errors that completely contradict the original (Thoresen et al., 4 May 2026). The adopted error types are factual, temporal, and numerical (Thoresen et al., 4 May 2026).

The synthetic generation process returns both the rewritten text and span annotations of changed regions as character intervals $$EM = \frac{1}{N} \sum_{i=1}^{N} 1\{ \text{normalize}(\hat{y}_i) = \text{normalize}(y_i) \},$$1, which are projected to subword tokens (Thoresen et al., 4 May 2026). If $$EM = \frac{1}{N} \sum_{i=1}^{N} 1\{ \text{normalize}(\hat{y}_i) = \text{normalize}(y_i) \},$$2 is the set of hallucinated spans and token $$EM = \frac{1}{N} \sum_{i=1}^{N} 1\{ \text{normalize}(\hat{y}_i) = \text{normalize}(y_i) \},$$3 covers $$EM = \frac{1}{N} \sum_{i=1}^{N} 1\{ \text{normalize}(\hat{y}_i) = \text{normalize}(y_i) \},$$4, then the token label is defined as

$$EM = \frac{1}{N} \sum_{i=1}^{N} 1\{ \text{normalize}(\hat{y}_i) = \text{normalize}(y_i) \},$$5

This yields, per language, 5,000 hallucinated samples, split into 4,000 training and 1,000 testing examples (Thoresen et al., 4 May 2026). These are paired with the original MultiWikiQA samples, so every instance appears twice—one true and one hallucinated—giving a balanced supervision signal for binary token-level classification (Thoresen et al., 4 May 2026).

For classifier selection, the authors fine-tuned three multilingual encoders on Danish and German and found that mmBERT-small (140M) performed best in both languages (Thoresen et al., 4 May 2026). Its reported test-set results on 1,000 samples are: Danish Supported-F1 0.9143, Unsupported-F1 0.8689, Accuracy 0.8963; German Supported-F1 0.9147, Unsupported-F1 0.8627, Accuracy 0.8948 (Thoresen et al., 4 May 2026). It was then used to train token-level hallucination classifiers for 30 European languages (Thoresen et al., 4 May 2026).

The benchmark defines two principal hallucination metrics:

$$EM = \frac{1}{N} \sum_{i=1}^{N} 1\{ \text{normalize}(\hat{y}_i) = \text{normalize}(y_i) \},$$6

where $$EM = \frac{1}{N} \sum_{i=1}^{N} 1\{ \text{normalize}(\hat{y}_i) = \text{normalize}(y_i) \},$$7 is the total number of tokens across all answers in language $$EM = \frac{1}{N} \sum_{i=1}^{N} 1\{ \text{normalize}(\hat{y}_i) = \text{normalize}(y_i) \},$$8, and

$$EM = \frac{1}{N} \sum_{i=1}^{N} 1\{ \text{normalize}(\hat{y}_i) = \text{normalize}(y_i) \},$$9

where $$P_i = \frac{|A_i \cap \hat{A}_i|}{|\hat{A}_i|}, \quad R_i = \frac{|A_i \cap \hat{A}_i|}{|A_i|}, \quad F1_i = \frac{2 P_i R_i}{P_i + R_i},$$0 is the number of answers in the test set for language $$P_i = \frac{|A_i \cap \hat{A}_i|}{|\hat{A}_i|}, \quad R_i = \frac{|A_i \cap \hat{A}_i|}{|A_i|}, \quad F1_i = \frac{2 P_i R_i}{P_i + R_i},$$1 (Thoresen et al., 4 May 2026).

When five LLMs are evaluated on English, Danish, German, and Icelandic, the reported rates show consistent language and model-size effects (Thoresen et al., 4 May 2026). For token-level hallucination rate $$P_i = \frac{|A_i \cap \hat{A}_i|}{|\hat{A}_i|}, \quad R_i = \frac{|A_i \cap \hat{A}_i|}{|A_i|}, \quad F1_i = \frac{2 P_i R_i}{P_i + R_i},$$2, Qwen3-0.6B records EN 0.03, DA 0.17, DE 0.09, IS 0.36; Qwen3-14B records EN 0.01, DA 0.08, DE 0.03, IS 0.17; Gemma-3-12B-IT records EN 0.02, DA 0.08, DE 0.05, IS 0.20; Cogito-Qwen-32B records EN 0.01, DA 0.07, DE 0.05, IS 0.18; and Cogito-Llama-70B records EN 0.02, DA 0.07, DE 0.05, IS 0.15 (Thoresen et al., 4 May 2026). For answer-level rate $$P_i = \frac{|A_i \cap \hat{A}_i|}{|\hat{A}_i|}, \quad R_i = \frac{|A_i \cap \hat{A}_i|}{|A_i|}, \quad F1_i = \frac{2 P_i R_i}{P_i + R_i},$$3, the same order yields EN 0.07, DA 0.52, DE 0.17, IS 0.60; EN 0.02, DA 0.12, DE 0.04, IS 0.26; EN 0.03, DA 0.13, DE 0.06, IS 0.27; EN 0.01, DA 0.09, DE 0.06, IS 0.18; and EN 0.03, DA 0.08, DE 0.06, IS 0.19 (Thoresen et al., 4 May 2026). The most prominent result is the Icelandic peak for Qwen3-0.6B, where up to 60% of answers contain at least one hallucination (Thoresen et al., 4 May 2026). The paper interprets this as evidence that hallucination rates are consistently higher for lower-resource languages, particularly Icelandic, while larger models generally hallucinate less (Thoresen et al., 4 May 2026).

7. Relation to MegaWika and multilingual KGQA

MultiWikiQA occupies the text-based extractive QA end of multilingual wiki-derived question answering, but it sits within a broader ecosystem that includes source-grounded QA over Wikipedia citations and structured KGQA over wiki-derived knowledge graphs. MegaWika introduces 13 million Wikipedia articles in 50 diverse languages together with 71 million referenced source materials, aligned at the sentence level as “reports” paired with citations and harvested sources (Barham et al., 2023). Although the exact term “MultiWikiQA” is not used there, MegaWika is explicitly positioned for cross-lingual QA and related knowledge-intensive tasks through sentence-level claim–citation–source triplets, English translations for non-English content, and a QA baseline comparison section (Barham et al., 2023). Its natural QA use cases include retrieving relevant context from article text or cited sources, extracting or generating answers, and transferring QA models between languages (Barham et al., 2023). A plausible implication is that MegaWika extends the evidence space beyond the Wikipedia passage itself, whereas MultiWikiQA constrains the task to extractive reading over Wikipedia context.

QALD-9-plus represents a structurally different strand of multilingual wiki-based QA: knowledge graph question answering over DBpedia and Wikidata (Perevalov et al., 2022). It extends QALD-9 by translating English questions into eight additional languages by native speakers and transferring the original SPARQL queries from DBpedia to Wikidata, yielding a benchmark with 558 unique English questions, 4,930 new multilingual question translations, and 507 new Wikidata queries (Perevalov et al., 2022). The supported languages are English, German, French, Russian, Ukrainian, Lithuanian, Belarusian, Bashkir, and Armenian (Perevalov et al., 2022). Unlike MultiWikiQA, which requires extracting verbatim answers from unstructured Wikipedia text, QALD-9-plus evaluates mapping natural-language questions to SPARQL over structured wiki-derived knowledge graphs (Perevalov et al., 2022).

This distinction is methodologically important. Text-based QA resources such as MultiWikiQA, MLQA, MKQA, TyDi QA, and XQuAD operate over unstructured text and answer spans, whereas KGQA benchmarks such as QALD-9-plus require semantic parsing into executable graph queries (Perevalov et al., 2022). MegaWika, meanwhile, introduces cited external sources as a second evidence layer beyond Wikipedia itself (Barham et al., 2023). Taken together, these resources define complementary formulations of multilingual wiki-centered QA: extractive reading comprehension over article text, source-grounded evidence retrieval and answering over article-plus-citation graphs, and multilingual KGQA over DBpedia and Wikidata.