LocQA: Locale-Ambiguous QA Benchmark

• LocQA is a benchmark for locale-ambiguous multilingual QA that challenges models to deliver locale-specific answers despite the absence of explicit regional cues.
• It features a dataset of 2,156 question–language–locale triples across 49 regions, enabling detailed evaluation of both inter-lingual and intra-lingual biases.
• LocQA reveals that multilingual fluency does not guarantee accurate localization, highlighting prevalent US-centric biases and discrepancies linked to regional population scales.

Searching arXiv for the specified LocQA paper and closely related benchmarks to ground the article in current research. LocQA is a test set for locale-ambiguous multilingual question answering in which the query omits explicit locale cues and only the querying language remains, allowing model outputs to reveal implicit priors about locale-dependent facts such as laws, dates, and measurements. Introduced in "Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMs" (Mor-Lan et al., 21 Apr 2026), it frames locale dependence as a structural evaluation problem: multilingual fluency does not guarantee correct localization. In a related trajectory, XLQA states that it lays groundwork for “LocQA” research by separating locale-invariant from locale-sensitive open-domain questions (Roh et al., 22 Aug 2025).

1. Conceptual basis

LocQA is built around a specific failure mode of multilingual LLMs: a model may answer fluently in many languages while still defaulting to the wrong locale. The benchmark therefore targets questions whose surface form is semantically invariant but whose correct answer changes with locale. Its design removes explicit country names or regional markers, so the only cue available to the model is the language of the query itself (Mor-Lan et al., 21 Apr 2026).

The underlying problem is not mere translation ambiguity. Rather, identical linguistic forms can correspond to different factual referents across regions. The benchmark’s examples make this explicit: in Spanish, “¿Cuál es la moneda nacional?” can correctly map to “Euro (EUR)” for Spain, “Peso (MXN)” for Mexico, or “Sol (PEN)” for Peru; in French, “Quand commence l’exercice fiscal ?” is answered by “1 er Janvier” in France or Belgium, “1 er Avril” in Canada, and “1 er Octobre” in Haiti (Mor-Lan et al., 21 Apr 2026). Because the query never specifies which country is intended, every LocQA item is intrinsically ambiguous.

This construction reorients multilingual QA evaluation away from the assumption of a single language-independent gold answer. A plausible implication is that LocQA is best understood as a diagnostic of implicit priors, not as a conventional accuracy-only benchmark.

2. Dataset construction and coverage

The dataset was created from 44 semantically invariant question templates, including prompts such as “What is the legal drinking age?”, “When does the fiscal year start?”, and “What is my international telephone country code?” Bilingual annotators translated each template into twelve target languages—English, Spanish, French, German, Hebrew, Hindi, Indonesian, Italian, Japanese, Korean, Portuguese, and Chinese—and provided the ground-truth answer for every country in which that language is spoken, limiting coverage to countries with at least $1$ M native speakers (Mor-Lan et al., 21 Apr 2026).

After cross-validation and author review, the final dataset contains $2{,}156$ distinct question–language–locale triples covering $49$ regions in total. The unit of analysis is therefore not simply a question or a language, but a triple linking a locale-ambiguous query form, a language, and a region-specific gold answer (Mor-Lan et al., 21 Apr 2026).

This design distinguishes LocQA from multilingual benchmarks that rely on parallel translation but still collapse correctness to a single universal answer. It also makes locale ambiguity measurable at two levels: across languages and within the multiple locales associated with a single language.

3. Evaluation protocol and bias metrics

LocQA evaluates models in a zero-shot, open-ended setting. Responses are compared against all gold answers by an LLM-as-Judge pipeline that semantically matches outputs to any valid locale-specific answer, including the US answer, and also detects whether the United States is used as an explicit conceptual anchor even when its answer is not listed (Mor-Lan et al., 21 Apr 2026).

To quantify inter-lingual or “global” bias, the benchmark defines the US-bias of a model on language $L$ as

$B_{US}(L) = P_{obs}(A_{US}) - P_{exp}(A_{US}),$

where

$$P_{obs}(A_{US}) = \frac{1}{|Q|}\sum_{q \in Q} \mathbf{1}[\text{model’s answers to } (q,L) \text{ include the US answer}],$$

and

$$P_{exp}(A_{US}) = \frac{1}{|Q|\cdot|C_L|}\sum_{q\in Q}\sum_{c\in C_L} \mathbf{1}[\text{US answer} \equiv \text{gold answer for locale } c].$$

In plain terms, $B_{US}$ measures how much more often a model proposes the US answer than would be expected from chance collisions among locale-specific gold answers (Mor-Lan et al., 21 Apr 2026).

To quantify intra-lingual or “regional” bias, LocQA computes locale-wise collision-aware counts,

$$N_{model}(c) = \# \text{ of questions whose model reply matches any gold answer held by locale } c,$$

$$N_{data}(c) = \# \text{ of questions whose gold answers are valid for locale } c,$$

and converts them into probability distributions

$2{,}156$0

The regional lift

$2{,}156$1

indicates over-representation when $2{,}156$2 and under-representation when $2{,}156$3. Regional fairness in a language is summarized by the mean absolute deviation $2{,}156$4 over locales and then averaged across multi-locale languages (Mor-Lan et al., 21 Apr 2026).

The automated judge was checked against human annotation and reached $2{,}156$5 agreement on a random sample. The authors also report that empty or off-topic answers, which constitute $2{,}156$6 of cases, do not drive the main bias signals: $2{,}156$7 rankings remain stable when conditioning on at least one matched answer, with Spearman $2{,}156$8, and $2{,}156$9 is inherently unaffected by non-matches (Mor-Lan et al., 21 Apr 2026).

4. Empirical findings

LocQA was used to evaluate $49$0 models, ranging from $49$1 B open-weight backbones to the largest proprietary systems. The main inter-lingual result is a strong and pervasive US-centric default even on non-English queries. On average, US answers appear in $49$2 of responses to non-English questions, whereas the expected collision rate is $49$3, yielding an average $49$4 (Mor-Lan et al., 21 Apr 2026).

The paper further decomposes this behavior into three error modes. “Intrusion,” in which the model provides the correct local answer but gratuitously adds the US answer, occurs in $49$5 of replies. “Selective preference for the US option in multi-locale questions” occurs in $49$6. “Complete erasure of local reality in favor of US norms” occurs in $49$7 (Mor-Lan et al., 21 Apr 2026). These categories are important because they distinguish between additive US anchoring and outright substitution of local facts.

The main intra-lingual result is that when multiple locales share a language, model outputs track the order of magnitude of speaking populations. The reported logarithmic fit between mean $49$8 and $49$9 is strong, with $L$0, $L$1, and $L$2. In Spanish, the USA, Spain, Mexico, and Argentina are over-represented, while Honduras, Bolivia, and Nicaragua are under-represented; similar hierarchies appear in French and English (Mor-Lan et al., 21 Apr 2026).

Instruction tuning produces a mixed effect. It reduces average regional distortion, since mean $L$3 decreases, but it simultaneously amplifies US bias, with $L$4 for every model family. The paper attributes this to learned “answer multiplicity”: aligned models list more valid answers per question, up by approximately $L$5, which dilutes regional dominance while still including the US as an anchor option, with a reported correlation of $L$6 between increased multiplicity and rising $L$7 (Mor-Lan et al., 21 Apr 2026). A common misconception is therefore that alignment uniformly improves localization; LocQA shows that it can improve one fairness dimension while worsening another.

5. Relation to adjacent locale-aware QA resources

LocQA sits within a broader cluster of benchmarks and frameworks that shift QA evaluation from purely linguistic transfer to cultural, regional, and local grounding. XLQA is the closest conceptual neighbor: it introduces a benchmark for locale-sensitive multilingual open-domain QA, starting from $L$8 English seed questions expanded to eight languages, with human-verified labels distinguishing locale-invariant from locale-sensitive cases; the final resource contains $L$9 QA triples, and $B_{US}(L) = P_{obs}(A_{US}) - P_{exp}(A_{US}),$0 are labeled locale-sensitive (Roh et al., 22 Aug 2025). Whereas LocQA makes every item ambiguous by design, XLQA separates invariant from sensitive cases and studies performance degradation on the sensitive subset.

NativQA addresses the data-generation problem rather than the diagnostic problem. It is a framework for constructing culturally and regionally aligned QA resources from seed queries and search-engine APIs, evaluated across $B_{US}(L) = P_{obs}(A_{US}) - P_{exp}(A_{US}),$1 locations in $B_{US}(L) = P_{obs}(A_{US}) - P_{exp}(A_{US}),$2 countries and $B_{US}(L) = P_{obs}(A_{US}) - P_{exp}(A_{US}),$3 languages, resulting in more than $B_{US}(L) = P_{obs}(A_{US}) - P_{exp}(A_{US}),$4 QA pairs (Alam et al., 8 Apr 2025). Its emphasis on “native knowledge” and “locality” complements LocQA’s focus on latent model priors.

CUS-QA extends the regional-knowledge agenda into open-ended and multimodal evaluation. It contains manually curated textual and visual questions grounded in Wikipedia for Czechia, Slovakia, and Ukraine, with English translations, totaling $B_{US}(L) = P_{obs}(A_{US}) - P_{exp}(A_{US}),$5 English QA pairs. Its results show a significant gap in regional knowledge among current LLMs and show that, apart from LLM-based evaluation, automated metrics correlate only minimally with human judgment (Libovický et al., 30 Jul 2025).

Taken together, these resources indicate a shift from multilingual QA as translation-consistent fact retrieval toward multilingual QA as locale-conditioned knowledge access. This suggests that “multilingual” and “multi-regional” should not be treated as interchangeable categories.

6. Broader usage of the term and research implications

Recent literature uses “LocQA” in more than one sense. In OCC-RAG, the term refers to localized, context-grounded QA defined by three capabilities: multi-hop inference and commonsense bridging across context chunks, strict faithfulness, and calibrated abstention on unanswerable or insufficient-evidence queries (Savkin et al., 30 May 2026). In the PlanQA tutorial, “LocQA” is used for location-based question answering over 2D indoor layouts represented in JSON, with tasks spanning distance, visibility, path planning, affordance, clearance, balance, and usability (Rodionov et al., 10 Jul 2025). In LocalRQA, “LocQA” denotes fully local retrieval-augmented QA systems that are generated, trained, tested, and deployed on local infrastructure (Yu et al., 2024).

This terminological breadth makes disambiguation necessary. In multilingual bias research, “LocQA” refers specifically to the locale-ambiguous benchmark of (Mor-Lan et al., 21 Apr 2026); in adjacent work, it can denote localized QA more generally, location-based reasoning, or locally deployed RAG pipelines.

The benchmark’s practical recommendations are correspondingly direct. Downstream systems and evaluation suites should treat locale as first-class context, allowing users to specify or systems to infer locale through user profile, geolocation, or explicit prompting. The paper also argues that current instruction-tuning recipes universalize the US perspective, so future cultural alignment should penalize unwarranted US intrusions and incentivize correct local selection, potentially through locale-balanced instruction data, counter-prompting against inappropriate US analogies, or a calibrated regional fairness term in the loss function (Mor-Lan et al., 21 Apr 2026).

LocQA therefore functions both as a benchmark and as a formalization of a broader evaluation principle: the correctness of an answer may depend not only on the language of a query, but on the social, legal, and demographic context in which that language is used.