---
title: 'XLQA: Locale-Aware Multilingual QA Benchmark'
url: https://www.emergentmind.com/topics/xlqa
type: topic
---

# XLQA: Locale-Aware Multilingual QA Benchmark

XLQA is a benchmark for locale-aware multilingual open-domain question answering (ODQA) that was introduced to address a specific limitation in multilingual evaluation: most prior benchmarks focus on English or on parallel translations of English QA pairs and assume locale-invariant answers across languages. XLQA instead distinguishes locale-invariant and locale-sensitive cases, using human-verified annotations to capture questions whose correct answer legitimately varies with cultural or regional context. The benchmark contains 3,000 English seed questions expanded to eight languages and is designed to expose failures of multilingual large language models on locale-sensitive ODQA [2508.16139].

## 1. Motivation and problem setting

Open-domain question answering research to date has overwhelmingly focused on English, or else on parallel translations of English QA pairs such as MLQA, XQuAD, TyDiQA, and MKQA. These benchmarks are described as “locale-agnostic”: they assume that the meaning of a question—and its one “correct” answer—remain identical across all languages. XLQA was created to challenge that assumption by focusing on cases where cultural or regional context affects question understanding and answer selection [2508.16139].

The motivating observation is that many everyday questions are locale-sensitive. The paper’s example is “Twin Towers”: the dominant referent in English is the World Trade Center, whereas Korean speakers may think of the LG Twin Towers, and Chinese speakers of the Tianjin IFC. By treating all languages as if they share the same background knowledge, existing multilingual ODQA benchmarks risk evaluation bias against systems that answer correctly from a given locale’s perspective. XLQA is designed to expose and quantify this gap [2508.16139].

The benchmark therefore reframes multilingual ODQA evaluation around locale-grounding knowledge rather than translation equivalence alone. A plausible implication is that multilingual competence and locale-awareness are separable capacities: a system may be fluent in a target language yet still fail because its background knowledge remains anchored to English-centric distributions.

## 2. Dataset construction pipeline

XLQA’s construction pipeline has three main stages: seed selection and multilingual question generation, locale-aware answer generation, and human verification and annotation [2508.16139].

The first stage begins with 3,000 high-quality English seed questions drawn from existing ODQA test sets: MKQA, MLQA, and HotpotQA. Duplicates, defined by exact match on question or answer, and unanswerable items are removed; 28.4 % are filtered out at this step. GPT-4.1 is then used to translate each seed into seven target languages—Arabic, Hebrew, Japanese, Korean, Russian, Simplified Chinese, and Vietnamese—producing 24,000 candidate multilingual questions. A back-translation filtering step follows: each translation is back-translated to English and judged, via an LLM-as-judge prompt, for semantic equivalence. Any seed whose translation fails in any language is discarded; 10.8 % of instances are removed in this stage [2508.16139].

The second stage generates locale-aware answers. For each semantically consistent multilingual question, the pipeline runs a retrieval-augmented generation system using GPT-4.1 plus web search, prioritizing Wikipedia, to produce an answer in the target language grounded in a real URL and a supporting evidence passage or URL that must contain “wikipedia” or “news.” If no reliable source is found or the URL is missing, the pair is discarded [2508.16139].

The third stage performs human verification and sensitivity annotation. Any question for which the answer in at least one language differs semantically from the English reference is flagged as a conflict candidate. Three multilingual annotators then verify two properties for each instance: correctness, meaning whether the answer is factually supported by the evidence, and locale-sensitivity, meaning whether the question legitimately allows multiple correct answers across locales. Only triples for which at least two annotators agree on both labels are retained. Overall human-agreement rates average 97.4 % for correctness (2/3) and 95.3 % for sensitivity (2/3) [2508.16139].

## 3. Corpus composition and annotation structure

XLQA contains 24,000 QA instances, consisting of 3,000 English questions and 21,000 translations, across the languages en, ar, he, ja, ko, ru, zh_cn, and vi [2508.16139].

The corpus statistics indicate relatively short ODQA instances. Average question length is 17–40 tokens, and average answer length is 4–6 tokens. The central annotation distinction is between locale-sensitive and locale-invariant questions. Among the 3,000 English seeds, 2,356, or 73.9 %, exhibit at least one cross-lingual answer conflict and are labeled locale-sensitive [2508.16139].

Conflict rates vary by language. Table 5 reports Arabic at 46.2 %, Hebrew at 44.3 %, Chinese at 39.0 %, Korean at 37.3 %, Japanese at 32.8 %, Russian at 30.2 %, and Vietnamese at 28.5 %. These figures quantify the extent to which answer divergence emerges when the same English seed question is localized across languages [2508.16139].

The paper groups answer conflicts into four categories. “Entity Conflict” covers cases such as “Who sang ‘Come and Get Your Love’?” where different locales privilege different referents. “Factual Conflict” includes historical dates or statistics. “Cultural Reference” covers cases such as award winners or media popularity. “Ambiguous Question” includes open-ended norms such as “national dish” [2508.16139]. This taxonomy suggests that locale-sensitivity is not reducible to named-entity ambiguity; it also arises in factual framing, cultural salience, and normatively underspecified questions.

## 4. Formal framework and evaluation protocol

XLQA defines a systematic framework for semantic consistency filtering, conflict detection, and multilingual evaluation [2508.16139].

For semantic consistency, the paper formalizes back-translation as
$$
T_l(q_{en}) \to q_l \to q'_{en}.
$$
A JudgeLM prompt asks whether $q'_{en}$ is semantically equivalent to $q_{en}$ with a Yes/No judgment, and the procedure discards any $q_{en}$ if there exists a language $l$ such that the judgment is No. This mechanism is intended to prevent apparent locale effects from being artifacts of mistranslation [2508.16139].

For conflict detection, the framework collects answers $a_{i,l}$ for each question $i$ and language $l$, normalizes strings, and computes embedding similarity $\mathrm{sim}(a_{i,en}, a_{i,l})$. A conflict is marked if $\mathrm{sim} < \tau$ or if the surface form differs in a way that implies a different referent. A question is labeled locale-sensitive if there exists at least one language with a conflict [2508.16139].

The evaluation protocol uses five state-of-the-art multilingual LLMs: GPT-4.1, Qwen-3 (14B), Gemma-3 (12B), LLaMA-3.1 (8B), and Exaone (7.8B). Experiments are zero-shot QA with temperature = 0, and prompting is performed in each target language with a simple “Answer the following question” template. Evaluation is reported separately on locale-invariant (“non-conflict”) and locale-sensitive (“conflict”) subsets [2508.16139].

The metrics are Exact Match and token-level F1:
$$
\mathrm{EM} = \frac{1}{N}\sum_{i=1}^N \mathbb{1}\{\hat y_i = y_i^*\}.
$$
For token overlap,
$$
\mathrm{Precision} = \frac{|\mathrm{Prediction} \cap \mathrm{Reference}|}{|\mathrm{Prediction}|}, \qquad
\mathrm{Recall} = \frac{|\mathrm{Prediction} \cap \mathrm{Reference}|}{|\mathrm{Reference}|},
$$
and
$$
\mathrm{F1} = \frac{2 \cdot \mathrm{Precision} \cdot \mathrm{Recall}}{\mathrm{Precision} + \mathrm{Recall}}.
$$
The paper also restates instance-level forms:
$$
\mathrm{EM}_i = 1 \text{ if } \hat y_i = y_i^* \text{ else } 0,
$$
$$
\mathrm{Precision}_i = \frac{|\hat y_i \cap y_i^*|}{|\hat y_i|}, \quad
\mathrm{Recall}_i = \frac{|\hat y_i \cap y_i^*|}{|y_i^*|}, \quad
\mathrm{F1}_i = \frac{2 \cdot \mathrm{Precision}_i \cdot \mathrm{Recall}_i}{\mathrm{Precision}_i + \mathrm{Recall}_i}.
$$

## 5. Empirical findings

XLQA’s evaluation shows a marked asymmetry between English and non-English performance and a systematic degradation on locale-sensitive questions [2508.16139].

Across models, English EM/F1 averages approximately 88.4 % / 90.8 %. Outside English, performance drops substantially: Qwen-3 reaches 52.7 % F1, Gemma-3 49.4 %, LLaMA-3.1 50.6 %, and Exaone 39.6 %. The lowest scores occur on Arabic and Hebrew, with F1 as low as 6–24 % [2508.16139].

The most important result is the gap between locale-invariant and locale-sensitive subsets. Across all models, locale-sensitive questions incur a 10–30 point F1 drop relative to locale-invariant ones. The paper gives the example of Gemma-3 on Japanese, where non-conflict F1 is 67.0 % and the corresponding contrasting score is 37.4 % [2508.16139]. This result directly supports the benchmark’s central claim that multilingual QA failures cannot be explained solely by translation difficulty.

Prompting with explicit locale information partially mitigates the problem. Adding a locale condition such as “Please answer as if you are in [locale]” improves Qwen-3 F1 by up to +25 points in Japanese and +17 in Chinese, although gains vary by language [2508.16139]. This indicates that some models possess latent locale-relevant knowledge but do not reliably activate it under standard prompts.

The paper attributes part of the gap to training data distribution. Models with stronger regional pretraining, such as Exaone on Korean and Qwen-3 on Chinese, show relatively smaller drops on locale-sensitive questions. The uneven distribution of locale-specific knowledge in pretraining data leads to both language-proficiency gaps and lack of locale-grounding [2508.16139].

## 6. Relation to prior multilingual QA and the term “XLQA”

XLQA is positioned against a line of multilingual benchmarks that are valuable but fundamentally locale-agnostic, including MLQA, XQuAD, TyDiQA, and MKQA [2508.16139]. Its novelty lies not in multilinguality alone but in the explicit annotation of answer divergence across locales.

The term “XLQA” also has an earlier usage in the literature. In the MLQA paper, “cross-lingual extractive question answering (XLQA)” denotes the task of locating answer spans in documents written in one language for questions posed in another [1910.07475]. MLQA was created to benchmark that task under realistic conditions using naturally-written contexts in many diverse languages, professionally-translated questions, and high-quality answer annotations. It contains QA instances in seven languages and supports both conventional cross-lingual transfer and generalized transfer, where question language and context language differ [1910.07475].

The distinction is methodologically important. MLQA addresses extractive QA with answer spans inside provided documents, whereas XLQA in [2508.16139] addresses open-domain QA and locale-sensitive answer validity. A plausible implication is that the two uses of “XLQA” correspond to different axes of multilingual difficulty: cross-lingual alignment in the earlier extractive setting and locale-aware grounding in the later ODQA setting. The benchmarks are therefore complementary rather than interchangeable.

## 7. Significance, misconceptions, and future directions

The main significance of XLQA is its demonstration that a large majority of everyday ODQA questions can admit valid, locale-dependent answers across languages. The benchmark reports that approximately 74 % of seeds are locale-sensitive, and it shows that state-of-the-art multilingual LLMs suffer significant performance drops on such items [2508.16139]. This challenges the common practice of treating multilingual evaluation as a translation problem with a single universal answer.

A common misconception is that multilingual QA benchmarks can be made sufficiently rigorous by translating English questions into other languages and preserving the English answer as the gold reference. XLQA directly argues against that assumption: multilingual benchmarks must go beyond mere translation equivalence, because correct answers may vary with regional background knowledge even when the translated question is semantically consistent [2508.16139].

The paper advocates three directions for future work: benchmark designs that explicitly distinguish locale-invariant and locale-sensitive questions; pretraining and instruction-tuning with richer, regionally grounded knowledge sources; and further expansion of locales and cultures, including low-resource languages, with careful human-authored or human-verified data [2508.16139]. This suggests a broader research program in which multilingual QA is evaluated not only for linguistic transfer but also for fidelity to the cultural context in which a question is posed.

Source: https://www.emergentmind.com/topics/xlqa