TyDi QA-WANA: WANA Multilingual QA Benchmark
- TyDi QA-WANA is a multilingual benchmark featuring 28,197 question-article pairs in 10 language varieties that emphasize native question elicitation and full-document answers.
- It employs a three-stage pipeline—question elicitation, full Wikipedia article retrieval, and minimal answer span labeling—to ensure cultural relevance and long-context evaluation.
- Baseline models using Gemini architectures highlight challenges in byte-index recovery and handling high NULL proportions across diverse languages.
TyDi QA-WANA is a multilingual extractive question answering benchmark for information-seeking questions in languages of West Asia and North Africa, introduced as “TyDi QA-WANA: A Benchmark for Information-Seeking Question Answering in Languages of West Asia and North Africa” (Riley et al., 23 Jul 2025). It contains 28,197 question–article pairs across 10 language varieties, pairs each question with an entire Wikipedia article that may or may not contain the answer, and evaluates a Minimal Answer Span task whose outputs are byte-index spans, YES/NO, or NULL. The benchmark inherits the translation-free, information-seeking design philosophy of TyDi QA while narrowing the scope to WANA languages and to long-context full-document QA (Clark et al., 2020).
1. Design rationale and scope
TyDi QA-WANA is motivated by three concerns stated explicitly in the benchmark design: information-seeking question elicitation, coverage of related but underrepresented languages and language varieties from West Asia and North Africa, and evaluation over long article-length contexts rather than short preselected passages (Riley et al., 23 Jul 2025). The authors position the benchmark against multilingual reading-comprehension datasets such as SQuAD 2.0, XQuAD, MLQA, M2QA, and LAReQA, which they characterize as datasets where each question was written after reading the paired passage. By contrast, TyDi QA-WANA follows the information-seeking paradigm associated with Natural Questions, MKQA, and TyDi QA.
A central methodological choice is native collection rather than translation. Questions are elicited directly in each language variety “to avoid issues of cultural relevance” (Riley et al., 23 Jul 2025). This places the benchmark in opposition to parallel multilingual QA datasets built by translating a fixed question set across languages. The underlying claim is not merely linguistic authenticity; it is that topic salience, underspecification, and information needs differ across language communities, so direct elicitation better matches realistic use.
The benchmark is also explicitly long-context. Each example is paired with the full text of a retrieved Wikipedia article, and the abstract stresses that the “relatively large size of the articles results in a task suitable for evaluating models' abilities to utilize large text contexts in answering questions” (Riley et al., 23 Jul 2025). In this respect, TyDi QA-WANA is closer to TyDi QA’s article-level setting than to passage-only reading comprehension.
2. Dataset composition and language coverage
The dataset contains 28,197 questions divided into 16,200 training examples, 5,995 development examples, and 6,002 test examples (Riley et al., 23 Jul 2025). Training examples have 1 annotation each; development and test examples have 3 annotations each. The benchmark covers 10 language varieties across 3 language families:
- Algerian Arabic (
arq) - Egyptian Arabic (
arz) - Jordanian Arabic (
apc-JO) - Iraqi Arabic (
acm) - Armenian (
hy) - Azerbaijani (
az) - Hebrew (
he) - Farsi (
fa) - Tajik (
tg) - Turkish (
tr)
For the Arabic varieties, the questions are in regional Arabic varieties, but the paired articles come from Arabic Wikipedia and are therefore in Modern Standard Arabic (MSA) (Riley et al., 23 Jul 2025). The paper treats this as related to cross-lingual QA, while emphasizing that prior datasets do not evaluate cross-variety QA specifically.
| Language variety | Train / Dev / Test | % With NULL Consensus |
|---|---|---|
| Algerian Arabic | 1306 / 649 / 647 | 69.6% |
| Egyptian Arabic | 1519 / 754 / 755 | 49.2% |
| Jordanian Arabic | 1386 / 688 / 690 | 50.7% |
| Iraqi Arabic | 1320 / 658 / 657 | 66.2% |
| Armenian | 1673 / 823 / 824 | 76.7% |
| Azerbaijani | 1469 / 724 / 724 | 68.1% |
| Hebrew | 1517 / 760 / 759 | 48.8% |
| Farsi | 3275 / 105 / 104 | 47.9% |
| Tajik | 1114 / 86 / 88 | 82.7% |
| Turkish | 1624 / 749 / 754 | 51.4% |
The benchmark also reports substantial variation in question length, article size, and answer length. Average question length ranges from 5.2 whitespace-delimited tokens in Turkish to 8.1 in Farsi. Average article length ranges from 36K bytes in Turkish to 131K bytes in Egyptian Arabic. Average answer length among non-NULL examples ranges from 18 bytes in Azerbaijani to 145 bytes in Algerian Arabic (Riley et al., 23 Jul 2025).
The high NULL-consensus rates are a defining property of the resource. The paper attributes them partly to smaller Wikipedias, fewer articles, and shorter articles in some languages, which increase the chance that the top retrieved article is irrelevant or incomplete (Riley et al., 23 Jul 2025).
3. Data collection and annotation pipeline
TyDi QA-WANA adopts a three-stage pipeline: question elicitation, article retrieval, and answer labeling (Riley et al., 23 Jul 2025).
Question elicitation begins from short prompts consisting of the first 150 characters of Wikipedia articles in the target language variety. Annotators are asked to write questions that are not directly answered by the prompt and that they are curious about. The generated question need not pertain closely to the prompt. This setup is designed to elicit information-seeking questions rather than answer-aware questions.
Article retrieval is performed with Google search restricted to the Wikipedia domain for the corresponding language variety, using the first result if any is found; if no result is found, the question is discarded (Riley et al., 23 Jul 2025). Retrieved text is preprocessed to remove tables, long lists, and infoboxes, focusing the benchmark on natural text rather than semi-structured content.
Answer labeling is hierarchical. For each retained question–article pair, annotators first decide whether some paragraph contains an answer; if so, they then select a minimal answer span, defined as the shortest character span that still satisfactorily answers the question. Development and test examples are labeled by 3 human annotators; training examples are labeled by 1 human annotator (Riley et al., 23 Jul 2025). The paper gives “George Washington” for “Who was the first President of the United States?” as the canonical illustration of a minimal answer, while noting that some answers may require most of a sentence.
The quality-control pipeline is comparatively elaborate. Raters first elicited a small number of English questions, received feedback, then elicited in-language questions that were reviewed by trusted in-house native speakers to verify that each rater was a native speaker of the relevant variety. For answer labeling, raters reviewed instructions and took a multiple-choice quiz on 21 example English question/article pairs, repeating the instruction-review cycle until scoring greater than 90%. They then labeled a small number of English examples, which researchers manually reviewed with feedback; this process was repeated until the researchers were satisfied with the annotators’ understanding. The paper also notes that raters were paid fair market rates, including training time (Riley et al., 23 Jul 2025).
4. Task definition and evaluation protocol
TyDi QA-WANA adopts TyDi QA’s Minimal Answer Span Task (MinSpan) (Riley et al., 23 Jul 2025). Given the full text of an article and a question, the system must produce one of three outputs:
- the start and end byte indices of the minimal span that completely answers the question, if such a span exists within the article;
- YES or NO if the question requires a yes/no answer and such a conclusion can be drawn from the article;
- NULL if neither of the above is possible.
This task definition makes the benchmark extractive in its formal output space, but not restricted to spans only. The byte-index representation is inherited from TyDi QA’s evaluation practice (Clark et al., 2020).
Evaluation uses a consensus-based NULL procedure. For development and test, where three annotations are available, the rule is: if fewer than two annotations are NULL, discard NULL annotations; if at least two are NULL, discard non-NULL annotations (Riley et al., 23 Jul 2025). This operationalizes answerability by majority vote and prevents a model from receiving credit merely because one annotator judged an example unanswerable.
The primary metric is average per-example F1, with Exact Match (EM) reported as a secondary metric. For YES, NO, or NULL outputs, F1 is 1.0 if any non-discarded annotation matches and 0.0 otherwise. For span outputs, the predicted byte range is compared against each annotated minimal span; the single-annotation score is the harmonic mean of byte-level precision and recall, and the example’s score is the maximum across retained annotations (Riley et al., 23 Jul 2025). EM follows the same structure but gives no partial credit for span overlap.
Two properties of this evaluation are especially important. First, the benchmark is fundamentally answerability-aware, because NULL is part of the formal label space rather than an auxiliary flag. Second, it is full-document rather than open-retrieval at runtime: the system is given the article and must find an answer within it.
5. Baseline models and reported performance
The paper reports two zero-shot long-context baselines: Gemini 1.5 Pro and Gemini 2.0 Flash (Riley et al., 23 Jul 2025). Neither model is finetuned for the task. Instead, each example is handled by showing the entire article to the model at once, preceded by a preamble and one exemplar of each question type for in-context learning. Decoding uses greedy sampling with maximum generation length 1024.
Because the benchmark formally requires byte ranges but the LLMs output answer text, the baseline recovers a span by locating the first occurrence of the generated text in the article (Riley et al., 23 Jul 2025). When the output is neither a valid article substring nor one of “yes”, “no”, or “no answer”, the pipeline invokes a second-pass no-answer critic to decide whether the generation semantically indicates that no answer was found.
| Language | Gemini 1.5 Pro EM / F1 | Gemini 2.0 Flash EM / F1 |
|---|---|---|
| Algerian Arabic | 45.0 / 46.9 | 45.7 / 47.7 |
| Egyptian Arabic | 53.0 / 58.4 | 53.1 / 58.0 |
| Iraqi Arabic | 60.6 / 62.9 | 61.5 / 64.3 |
| Jordanian Arabic | 46.8 / 56.3 | 48.1 / 57.6 |
| Armenian | 71.8 / 72.7 | 71.1 / 72.2 |
| Azerbaijani | 74.9 / 76.2 | 73.6 / 75.1 |
| Hebrew | 53.4 / 57.3 | 52.6 / 56.3 |
| Farsi | 34.6 / 38.7 | 45.2 / 50.4 |
| Tajik | 68.2 / 68.2 | 56.8 / 57.6 |
| Turkish | 53.3 / 55.1 | 53.8 / 55.3 |
The authors emphasize that cross-language comparisons should be treated cautiously because the question sets and NULL rates differ across language varieties (Riley et al., 23 Jul 2025). Their recommendation is to compare models within a language variety rather than interpreting score differences as direct evidence of absolute language ability.
A prominent pattern is the asymmetry between NULL and non-NULL cases. On development and test, the models usually perform better on NULL-consensus examples than on answerable ones. The paper notes one clear exception: Farsi, especially for Gemini 2.0 Flash, where test F1 is 46.7 on NULL examples and 51.0 on non-NULL examples (Riley et al., 23 Jul 2025). For Algerian Arabic, by contrast, the gap is severe: test F1 for Gemini 1.5 Pro is 55.9 on NULL examples and 16.1 on non-NULL examples, while Gemini 2.0 Flash shows 56.9 and 16.3 respectively. The authors therefore recommend reporting NULL / non-NULL breakdowns alongside aggregate scores.
6. Position within the TyDi ecosystem
TyDi QA-WANA is best understood as a regional, long-context continuation of the TyDi QA line rather than as an open-retrieval benchmark. TyDi QA introduced the underlying design philosophy: translation-free, native question elicitation, article-level evidence, and support for unanswerable and yes/no cases (Clark et al., 2020). Subsequent work on TyDi QA showed that answerability prediction is a major source of headroom in information-seeking QA, with substantial gains under oracle answer-type conditions (Asai et al., 2020). That finding is directly relevant to TyDi QA-WANA because NULL is part of the formal task and because many WANA subsets have high NULL-consensus rates.
The benchmark is also closely related to the long-document modeling literature. “Poolingformer: Long Document Modeling with Pooling Attention” evaluated long-context XLM-R models on TyDi QA using 4096-token spans and reported state-of-the-art single-model results on passage and minimal-answer tasks (Zhang et al., 2021). This supports the view that TyDi QA-WANA is not only a multilingual benchmark but also a stress test for long-context architectures.
By contrast, Mr. TyDi recasts TyDi QA into a mono-lingual retrieval benchmark over full Wikipedias (Zhang et al., 2021), and XOR QA constructs a cross-lingual open-retrieval QA task from TyDi questions that lacked same-language answers (Asai et al., 2020). A later MIA shared-task system for XOR-TyDi QA used multilingual dense retrieval, same-language sparse retrieval, and Fusion-in-Decoder generation (Tu et al., 2022). These resources are complementary rather than interchangeable: TyDi QA-WANA evaluates answer finding within a provided full article, whereas Mr. TyDi and XOR-style work evaluate the upstream retrieval problem.
This placement also clarifies TyDi QA-WANA’s Arabic design. The benchmark’s dialect-question/MSA-article configuration is related to cross-lingual QA, but it remains a provided-context task, not a retrieval challenge over multilingual corpora (Riley et al., 23 Jul 2025).
7. Limitations, interpretation, and likely research directions
Several limitations are explicit in the benchmark description. The resource has high NULL proportions in several languages, uneven development and test sizes—especially Farsi (105 / 104) and Tajik (86 / 88)—and relies on only the first Wikipedia search result during example construction (Riley et al., 23 Jul 2025). It is also limited to Wikipedia, so it does not evaluate broader web QA or domain-specific evidence access. For Arabic, the benchmark is deliberately cross-varietal rather than monovarietal, since questions are dialectal but articles are in MSA.
The baseline methodology introduces its own limitations. Byte-span recovery is heuristic because the models generate answer text and the evaluation pipeline maps it to the first occurrence in the article. The need for a no-answer critic further indicates that output-format compliance is nontrivial even when the model’s underlying judgment may be correct (Riley et al., 23 Jul 2025).
The broader TyDi literature suggests two especially salient extensions. First, answerability remains a major modeling challenge in information-seeking QA (Asai et al., 2020). This suggests that TyDi QA-WANA is likely to reward explicit evidence-sufficiency and abstention modeling rather than span extraction alone. Second, later work on TyDi boolean questions added sentence-level evidence spans, graded evidence strength, and weak-evidence reason codes (Rosenthal et al., 2021). A plausible implication is that TyDi QA-WANA could eventually be enriched with rationale-aware supervision, though the current benchmark does not provide such annotations.
As released, TyDi QA-WANA is most usefully interpreted as a benchmark for culturally grounded, long-context, answerability-aware QA in underrepresented WANA language varieties. Its main contribution is not a new modeling architecture but a carefully delimited evaluation setting: native information-seeking questions, full Wikipedia articles, explicit NULL handling, and language varieties that were largely absent from earlier multilingual QA benchmarks (Riley et al., 23 Jul 2025).
