---
title: 'TyDi QA-WANA: WANA Multilingual QA Benchmark'
url: https://www.emergentmind.com/topics/tydi-qa-wana
type: topic
---

# TyDi QA-WANA: WANA Multilingual QA Benchmark

TyDi QA-WANA is a multilingual extractive question answering benchmark for information-seeking questions in languages of West Asia and North Africa, introduced as “TyDi QA-WANA: A Benchmark for Information-Seeking Question Answering in Languages of West Asia and North Africa” [2507.17709]. It contains 28,197 question–article pairs across 10 language varieties, pairs each question with an entire Wikipedia article that may or may not contain the answer, and evaluates a Minimal Answer Span task whose outputs are byte-index spans, YES/NO, or NULL. The benchmark inherits the translation-free, information-seeking design philosophy of TyDi QA while narrowing the scope to WANA languages and to long-context full-document QA [2003.05002].

## 1. Design rationale and scope

TyDi QA-WANA is motivated by three concerns stated explicitly in the benchmark design: information-seeking question elicitation, coverage of related but underrepresented languages and language varieties from West Asia and North Africa, and evaluation over long article-length contexts rather than short preselected passages [2507.17709]. The authors position the benchmark against multilingual reading-comprehension datasets such as SQuAD 2.0, XQuAD, MLQA, M2QA, and LAReQA, which they characterize as datasets where each question was written after reading the paired passage. By contrast, TyDi QA-WANA follows the information-seeking paradigm associated with Natural Questions, MKQA, and TyDi QA.

A central methodological choice is native collection rather than translation. Questions are elicited directly in each language variety “to avoid issues of cultural relevance” [2507.17709]. This places the benchmark in opposition to parallel multilingual QA datasets built by translating a fixed question set across languages. The underlying claim is not merely linguistic authenticity; it is that topic salience, underspecification, and information needs differ across language communities, so direct elicitation better matches realistic use.

The benchmark is also explicitly long-context. Each example is paired with the full text of a retrieved Wikipedia article, and the abstract stresses that the “relatively large size of the articles results in a task suitable for evaluating models' abilities to utilize large text contexts in answering questions” [2507.17709]. In this respect, TyDi QA-WANA is closer to TyDi QA’s article-level setting than to passage-only reading comprehension.

## 2. Dataset composition and language coverage

The dataset contains **28,197 questions** divided into **16,200 training examples**, **5,995 development examples**, and **6,002 test examples** [2507.17709]. Training examples have **1 annotation** each; development and test examples have **3 annotations** each. The benchmark covers **10 language varieties** across **3 language families**:

- Algerian Arabic (`arq`)
- Egyptian Arabic (`arz`)
- Jordanian Arabic (`apc-JO`)
- Iraqi Arabic (`acm`)
- Armenian (`hy`)
- Azerbaijani (`az`)
- Hebrew (`he`)
- Farsi (`fa`)
- Tajik (`tg`)
- Turkish (`tr`)

For the Arabic varieties, the questions are in regional Arabic varieties, but the paired articles come from Arabic Wikipedia and are therefore in **Modern Standard Arabic (MSA)** [2507.17709]. The paper treats this as related to cross-lingual QA, while emphasizing that prior datasets do not evaluate cross-variety QA specifically.

| Language variety | Train / Dev / Test | % With NULL Consensus |
|---|---:|---:|
| Algerian Arabic | 1306 / 649 / 647 | 69.6% |
| Egyptian Arabic | 1519 / 754 / 755 | 49.2% |
| Jordanian Arabic | 1386 / 688 / 690 | 50.7% |
| Iraqi Arabic | 1320 / 658 / 657 | 66.2% |
| Armenian | 1673 / 823 / 824 | 76.7% |
| Azerbaijani | 1469 / 724 / 724 | 68.1% |
| Hebrew | 1517 / 760 / 759 | 48.8% |
| Farsi | 3275 / 105 / 104 | 47.9% |
| Tajik | 1114 / 86 / 88 | 82.7% |
| Turkish | 1624 / 749 / 754 | 51.4% |

The benchmark also reports substantial variation in question length, article size, and answer length. Average question length ranges from **5.2 whitespace-delimited tokens** in Turkish to **8.1** in Farsi. Average article length ranges from **36K bytes** in Turkish to **131K bytes** in Egyptian Arabic. Average answer length among non-NULL examples ranges from **18 bytes** in Azerbaijani to **145 bytes** in Algerian Arabic [2507.17709].

The high NULL-consensus rates are a defining property of the resource. The paper attributes them partly to smaller Wikipedias, fewer articles, and shorter articles in some languages, which increase the chance that the top retrieved article is irrelevant or incomplete [2507.17709].

## 3. Data collection and annotation pipeline

TyDi QA-WANA adopts a three-stage pipeline: **question elicitation**, **article retrieval**, and **answer labeling** [2507.17709].

Question elicitation begins from short prompts consisting of the **first 150 characters** of Wikipedia articles in the target language variety. Annotators are asked to write questions that are **not directly answered by the prompt** and that they are **curious about**. The generated question need not pertain closely to the prompt. This setup is designed to elicit information-seeking questions rather than answer-aware questions.

Article retrieval is performed with **Google search restricted to the Wikipedia domain** for the corresponding language variety, using the **first result** if any is found; if no result is found, the question is discarded [2507.17709]. Retrieved text is preprocessed to remove **tables**, **long lists**, and **infoboxes**, focusing the benchmark on natural text rather than semi-structured content.

Answer labeling is hierarchical. For each retained question–article pair, annotators first decide whether some paragraph contains an answer; if so, they then select a **minimal answer span**, defined as the shortest character span that still satisfactorily answers the question. Development and test examples are labeled by **3 human annotators**; training examples are labeled by **1 human annotator** [2507.17709]. The paper gives “George Washington” for “Who was the first President of the United States?” as the canonical illustration of a minimal answer, while noting that some answers may require most of a sentence.

The quality-control pipeline is comparatively elaborate. Raters first elicited a small number of **English questions**, received feedback, then elicited in-language questions that were reviewed by trusted in-house native speakers to verify that each rater was a native speaker of the relevant variety. For answer labeling, raters reviewed instructions and took a **multiple-choice quiz on 21 example English question/article pairs**, repeating the instruction-review cycle until scoring **greater than 90%**. They then labeled a small number of English examples, which researchers manually reviewed with feedback; this process was repeated until the researchers were satisfied with the annotators’ understanding. The paper also notes that raters were paid **fair market rates**, including training time [2507.17709].

## 4. Task definition and evaluation protocol

TyDi QA-WANA adopts TyDi QA’s **Minimal Answer Span Task (MinSpan)** [2507.17709]. Given the full text of an article and a question, the system must produce one of three outputs:

1. the **start and end byte indices** of the minimal span that completely answers the question, if such a span exists within the article;
2. **YES or NO** if the question requires a yes/no answer and such a conclusion can be drawn from the article;
3. **NULL** if neither of the above is possible.

This task definition makes the benchmark extractive in its formal output space, but not restricted to spans only. The byte-index representation is inherited from TyDi QA’s evaluation practice [2003.05002].

Evaluation uses a consensus-based NULL procedure. For development and test, where three annotations are available, the rule is: if **fewer than two annotations are NULL**, discard NULL annotations; if **at least two are NULL**, discard non-NULL annotations [2507.17709]. This operationalizes answerability by majority vote and prevents a model from receiving credit merely because one annotator judged an example unanswerable.

The primary metric is **average per-example F1**, with **Exact Match (EM)** reported as a secondary metric. For YES, NO, or NULL outputs, F1 is **1.0** if any non-discarded annotation matches and **0.0** otherwise. For span outputs, the predicted byte range is compared against each annotated minimal span; the single-annotation score is the harmonic mean of byte-level precision and recall, and the example’s score is the **maximum** across retained annotations [2507.17709]. EM follows the same structure but gives no partial credit for span overlap.

Two properties of this evaluation are especially important. First, the benchmark is fundamentally **answerability-aware**, because NULL is part of the formal label space rather than an auxiliary flag. Second, it is **full-document** rather than open-retrieval at runtime: the system is given the article and must find an answer within it.

## 5. Baseline models and reported performance

The paper reports two zero-shot long-context baselines: **Gemini 1.5 Pro** and **Gemini 2.0 Flash** [2507.17709]. Neither model is finetuned for the task. Instead, each example is handled by showing the **entire article** to the model at once, preceded by a preamble and **one exemplar of each question type** for in-context learning. Decoding uses **greedy sampling** with **maximum generation length 1024**.

Because the benchmark formally requires byte ranges but the LLMs output answer text, the baseline recovers a span by locating the **first occurrence** of the generated text in the article [2507.17709]. When the output is neither a valid article substring nor one of “yes”, “no”, or “no answer”, the pipeline invokes a second-pass **no-answer critic** to decide whether the generation semantically indicates that no answer was found.

| Language | Gemini 1.5 Pro EM / F1 | Gemini 2.0 Flash EM / F1 |
|---|---:|---:|
| Algerian Arabic | 45.0 / 46.9 | 45.7 / 47.7 |
| Egyptian Arabic | 53.0 / 58.4 | 53.1 / 58.0 |
| Iraqi Arabic | 60.6 / 62.9 | 61.5 / 64.3 |
| Jordanian Arabic | 46.8 / 56.3 | 48.1 / 57.6 |
| Armenian | 71.8 / 72.7 | 71.1 / 72.2 |
| Azerbaijani | 74.9 / 76.2 | 73.6 / 75.1 |
| Hebrew | 53.4 / 57.3 | 52.6 / 56.3 |
| Farsi | 34.6 / 38.7 | 45.2 / 50.4 |
| Tajik | 68.2 / 68.2 | 56.8 / 57.6 |
| Turkish | 53.3 / 55.1 | 53.8 / 55.3 |

The authors emphasize that cross-language comparisons should be treated cautiously because the question sets and NULL rates differ across language varieties [2507.17709]. Their recommendation is to compare models **within** a language variety rather than interpreting score differences as direct evidence of absolute language ability.

A prominent pattern is the asymmetry between NULL and non-NULL cases. On development and test, the models usually perform better on **NULL-consensus** examples than on answerable ones. The paper notes one clear exception: **Farsi**, especially for Gemini 2.0 Flash, where test F1 is **46.7** on NULL examples and **51.0** on non-NULL examples [2507.17709]. For Algerian Arabic, by contrast, the gap is severe: test F1 for Gemini 1.5 Pro is **55.9** on NULL examples and **16.1** on non-NULL examples, while Gemini 2.0 Flash shows **56.9** and **16.3** respectively. The authors therefore recommend reporting NULL / non-NULL breakdowns alongside aggregate scores.

## 6. Position within the TyDi ecosystem

TyDi QA-WANA is best understood as a regional, long-context continuation of the TyDi QA line rather than as an open-retrieval benchmark. TyDi QA introduced the underlying design philosophy: translation-free, native question elicitation, article-level evidence, and support for unanswerable and yes/no cases [2003.05002]. Subsequent work on TyDi QA showed that answerability prediction is a major source of headroom in information-seeking QA, with substantial gains under oracle answer-type conditions [2010.11915]. That finding is directly relevant to TyDi QA-WANA because NULL is part of the formal task and because many WANA subsets have high NULL-consensus rates.

The benchmark is also closely related to the long-document modeling literature. “Poolingformer: Long Document Modeling with Pooling Attention” evaluated long-context XLM-R models on TyDi QA using **4096-token** spans and reported state-of-the-art single-model results on passage and minimal-answer tasks [2105.04371]. This supports the view that TyDi QA-WANA is not only a multilingual benchmark but also a stress test for long-context architectures.

By contrast, Mr. TyDi recasts TyDi QA into a **mono-lingual retrieval** benchmark over full Wikipedias [2108.08787], and XOR QA constructs a **cross-lingual open-retrieval QA** task from TyDi questions that lacked same-language answers [2010.11856]. A later MIA shared-task system for XOR-TyDi QA used multilingual dense retrieval, same-language sparse retrieval, and Fusion-in-Decoder generation [2207.01940]. These resources are complementary rather than interchangeable: TyDi QA-WANA evaluates answer finding within a provided full article, whereas Mr. TyDi and XOR-style work evaluate the upstream retrieval problem.

This placement also clarifies TyDi QA-WANA’s Arabic design. The benchmark’s dialect-question/MSA-article configuration is related to cross-lingual QA, but it remains a **provided-context** task, not a retrieval challenge over multilingual corpora [2507.17709].

## 7. Limitations, interpretation, and likely research directions

Several limitations are explicit in the benchmark description. The resource has **high NULL proportions** in several languages, **uneven development and test sizes**—especially **Farsi** (**105 / 104**) and **Tajik** (**86 / 88**)—and relies on only the **first** Wikipedia search result during example construction [2507.17709]. It is also limited to **Wikipedia**, so it does not evaluate broader web QA or domain-specific evidence access. For Arabic, the benchmark is deliberately cross-varietal rather than monovarietal, since questions are dialectal but articles are in MSA.

The baseline methodology introduces its own limitations. Byte-span recovery is heuristic because the models generate answer text and the evaluation pipeline maps it to the **first occurrence** in the article. The need for a no-answer critic further indicates that output-format compliance is nontrivial even when the model’s underlying judgment may be correct [2507.17709].

The broader TyDi literature suggests two especially salient extensions. First, answerability remains a major modeling challenge in information-seeking QA [2010.11915]. This suggests that TyDi QA-WANA is likely to reward explicit evidence-sufficiency and abstention modeling rather than span extraction alone. Second, later work on TyDi boolean questions added sentence-level evidence spans, graded evidence strength, and weak-evidence reason codes [2112.07772]. A plausible implication is that TyDi QA-WANA could eventually be enriched with rationale-aware supervision, though the current benchmark does not provide such annotations.

As released, TyDi QA-WANA is most usefully interpreted as a benchmark for **culturally grounded, long-context, answerability-aware QA** in underrepresented WANA language varieties. Its main contribution is not a new modeling architecture but a carefully delimited evaluation setting: native information-seeking questions, full Wikipedia articles, explicit NULL handling, and language varieties that were largely absent from earlier multilingual QA benchmarks [2507.17709].

Source: https://www.emergentmind.com/topics/tydi-qa-wana