GeoFact-X: Multilingual Factual Benchmark

• GeoFact-X is a geography-based benchmark that assesses multilingual factual reasoning by evaluating answer correctness, explicit reasoning traces, and language consistency.
• It leverages culturally grounded contexts and parallel multilingual formulations to reveal hidden language biases in large language models.
• The benchmark supports both supervised fine-tuning and reinforcement learning methods, reducing language mismatch and improving overall factual accuracy.

GeoFact-X is a geography-based benchmark for multilingual factual reasoning with explicit reasoning traces, introduced in "Learn Globally, Speak Locally: Bridging the Gaps in Multilingual Reasoning" (Hwang et al., 7 Jul 2025). It is designed to test not only whether a LLM produces the correct factual answer, but whether it reasons in the same language as the question. This emphasis targets a specific multilingual failure mode: models may answer correctly while defaulting to reasoning in English or another high-resource language, especially for medium- and low-resource languages such as Hindi, Swahili, and Thai. GeoFact-X therefore treats answer correctness, reasoning quality, and language consistency as distinct but jointly relevant dimensions of multilingual factual reasoning.

1. Conceptual rationale

GeoFact-X is motivated by the claim that multilingual reasoning cannot be evaluated adequately by final answers alone (Hwang et al., 7 Jul 2025). Existing multilingual benchmarks often score only the endpoint, which can obscure a substantive failure mode: a model may know the answer but fail to reason in the target language. In the paper’s framing, this is not merely a stylistic defect. It bears on interpretability, user trust, inclusivity, and factual reliability, particularly when culturally grounded concepts are best accessed and explained in the local language.

The benchmark is built around an implicit English bias in multilingual LLMs. The central empirical claim is that multilingual models often exhibit a gap between answer production and language-aligned reasoning. GeoFact-X exists to make that gap measurable. This makes it a reasoning-oriented benchmark rather than a conventional multilingual QA set.

The choice of geography is deliberate. The paper argues that geography naturally ties factual knowledge to country-specific, culturally grounded contexts. Rather than using generic world facts, GeoFact-X centers topics such as history, politics, geography, art, culture, literature, religion, food/cuisine, science/technology, sports, and language. This design allows the benchmark to test not only multilinguality in the abstract, but multilingual reasoning over locally grounded knowledge.

A further conceptual distinction is between associated and non-associated language-country pairs. Thai questions about Thailand are “associated,” whereas Thai questions about the USA are “non-associated.” This distinction makes it possible to examine whether factual recall transfers across languages and cultural contexts, rather than only within language-country pairs that are naturally aligned.

The benchmark is explicitly contrasted with multiple-choice multilingual benchmarks such as XCOPA, XWinograd, and XStoryCloze. In the paper’s account, such benchmarks can be solved through surface cues or guessing and do not reveal whether a model actually reasons in the target language.

2. Benchmark structure and coverage

Each GeoFact-X example contains three components: a factual question, a correct answer, and a step-by-step chain-of-thought explanation annotated with <step> tokens (Hwang et al., 7 Jul 2025). The reasoning trace is part of both supervision and evaluation. The benchmark therefore asks whether a model can produce a coherent intermediate explanation, in the proper language, that leads to the answer.

GeoFact-X covers five languages and five geographically distinct countries:

Country	Predominant local language
USA	English
India	Hindi
Japan	Japanese
Kenya	Swahili
Thailand	Thai

The inclusion of Swahili and Thai, and to some extent Hindi, is important because the benchmark is explicitly designed to expose multilingual reasoning failures in medium- and low-resource languages. This suggests that the benchmark’s value lies partly in revealing when answer production is transferred from English-dominant training without genuine target-language reasoning.

The benchmark contains 3,000 unique factual questions, approximately 600 per country. Its appendix organizes content into ten high-level domains: History, Geography, Politics, Literature, Arts & Culture, Science & Technology, Sports, Food & Cuisine, Language, and Religion. Each topic is subdivided into finer subcategories such as “Person,” “Date,” and “Place,” and for each subcategory the authors automatically generated 20 questions per country. Generation was constrained to produce diverse, unambiguous factual questions with a single definite answer, while preserving consistency across the five languages.

A representative figure in the paper shows the same factual question and answer content translated across languages, including English, Hindi, and Thai, with corresponding reasoning traces generated by Gemini 2.0 Flash. The exact sample text is not reproduced in the provided excerpt, but the structure establishes that GeoFact-X is organized around parallel multilingual formulations with language-matched rationales.

3. Construction, validation, and partitioning

GeoFact-X was generated using Gemini 2.0 Flash (Hwang et al., 7 Jul 2025). The pipeline first produces multilingual factual QA pairs and then applies a two-stage validation process. In stage one, clearly incorrect or ambiguous question-answer pairs are removed through rule-based filtering and automated checks. One specific check is cross-language answer consistency: each answer is translated into English with the Google Translate API, and mismatches are identified.

In stage two, the authors manually validate a stratified 10% subset of the dataset using Google and Wikipedia searches, removing hallucinated or inaccurate content. The appendix states that the remaining dataset was still undergoing complete verification. This is a material caveat, because it qualifies the current status of the benchmark’s factual validation.

After validation, Gemini 2.0 Flash is used again to generate step-by-step reasoning traces for each valid question-answer pair. The prompt explicitly instructs the model that each reasoning step should begin with a <step> token. These traces serve two functions: they are reference explanations for evaluation, and they provide supervision for training.

The final dataset is split randomly by unique factual question into three parts:

• 70% for supervised fine-tuning (SFT)
• 15% for GRPO training
• 15% as the held-out benchmark/test set

The paper describes GeoFact-X both as a benchmark and as a training dataset. One sentence in the main text describes it as being “used exclusively for GRPO training,” but the appendix and conclusion state that the reasoning traces are also used for SFT and BRIDGE-based fine-tuning. A cautious reading is that GeoFact-X supports multiple training stages, while the held-out partition remains the benchmark proper.

4. Evaluation methodology

GeoFact-X uses a richer evaluation scheme than plain accuracy (Hwang et al., 7 Jul 2025). The three main test-set metrics are:

1. Reasoning score
2. Language mismatch
3. Answer correctness

All three are reported on a 0–100 scale.

Scoring is performed with an LLM-as-a-judge protocol using Qwen-2.5-72B-Instruct as the judge model. The judge compares a model-generated response against the reference reasoning trace and answer produced by Gemini 2.0 Flash. According to the paper and appendix, the judge prompt evaluates criteria such as logical structure, key insights, factual correctness, conclusion validity, language mismatch detection, and answer validation. The exact prompt text is not fully reproduced in the provided material.

The paper also formalizes language correctness through a language identifier $f$. Although the following equation is introduced in the MGSM analysis rather than GeoFact-X itself, it expresses the benchmark’s underlying criterion of language-aligned output:

$$A_\text{lang} = \frac{1}{N}\sum_n^N \delta [f(o_n) = l_n],$$

where $N$ is the number of samples, $o_n$ is the generated output, $l_n$ is the target question language, and $\delta[\cdot]$ is the indicator function.

A stricter joint metric, again defined for MGSM but conceptually aligned with GeoFact-X, requires both correctness and language match:

$$A_\text{joint} = \frac{1}{N} \sum_n^N \left( \delta[f(o_n) = l_n] \cdot \delta[\hat{a}_n = a_n] \right),$$

where $\hat{a}_n$ is the predicted answer and $a_n$ is the gold answer.

These formulas clarify the paper’s central claim: a multilingual reasoning output is fully successful only if it is both correct and expressed in the intended language. The excerpt does not provide inter-annotator agreement or a full human-versus-judge calibration study, so the reliability of the LLM-as-a-judge layer is not fully characterized there.

5. BRIDGE and the training role of GeoFact-X

GeoFact-X is not only an evaluation benchmark; it is also embedded in the paper’s broader training framework, BRIDGE, which targets multilingual reasoning more generally (Hwang et al., 7 Jul 2025). BRIDGE combines supervised fine-tuning (SFT) with reinforcement fine-tuning (RFT), specifically Group Relative Policy Optimization (GRPO).

The SFT objective is standard next-token cross-entropy. Given a question $q \sim P(\mathcal{Q})$, ground-truth sample $$A_\text{lang} = \frac{1}{N}\sum_n^N \delta [f(o_n) = l_n],$$0, and model output token distribution $$A_\text{lang} = \frac{1}{N}\sum_n^N \delta [f(o_n) = l_n],$$1, the paper defines

$$A_\text{lang} = \frac{1}{N}\sum_n^N \delta [f(o_n) = l_n],$$2

The multilingual component is supplied by GRPO. The paper presents a clipped GRPO/PPO-style objective with a KL anchor to the frozen base model; its printed formula appears partially corrupted in the manuscript, but the intended mechanism is standard policy optimization over grouped samples with clipping and KL regularization.

The distinctive component is the language consistency reward:

$$A_\text{lang} = \frac{1}{N}\sum_n^N \delta [f(o_n) = l_n],$$3

where $$A_\text{lang} = \frac{1}{N}\sum_n^N \delta [f(o_n) = l_n],$$4 is a language detector, $$A_\text{lang} = \frac{1}{N}\sum_n^N \delta [f(o_n) = l_n],$$5 is the model output, and $$A_\text{lang} = \frac{1}{N}\sum_n^N \delta [f(o_n) = l_n],$$6 is the target language. The paper specifies that this reward is applied to the model’s reasoning, or “thinking,” tokens. The point is therefore not only to make the final answer appear in the target language, but to align the reasoning process itself with that language.

To prevent GRPO from overwhelming SFT, BRIDGE samples whether to apply the GRPO term with Bernoulli probability $$A_\text{lang} = \frac{1}{N}\sum_n^N \delta [f(o_n) = l_n],$$7. The total objective is

$$A_\text{lang} = \frac{1}{N}\sum_n^N \delta [f(o_n) = l_n],$$8

where $$A_\text{lang} = \frac{1}{N}\sum_n^N \delta [f(o_n) = l_n],$$9 is sampled from a Bernoulli distribution with probability $N$0. The appendix specifies $N$1 in the reported experiments. This makes BRIDGE a lightweight hybrid: mostly supervised learning, with occasional language-consistency-guided policy updates.

The paper’s terminology is reinforcement fine-tuning (RFT) rather than test-time RL. GeoFact-X is a central benchmark for demonstrating BRIDGE’s factual reasoning effects, but BRIDGE is presented as a broader multilingual reasoning method spanning both factual and mathematical tasks.

6. Empirical results

GeoFact-X reveals a clear separation between factual competence and language-consistent reasoning (Hwang et al., 7 Jul 2025). On the full test set, the base model Qwen2.5-7B-Instruct obtains:

• Reasoning score: 45.6
• Language mismatch: 14.2
• Answer correctness: 28.6

The associated/non-associated split also matters. On associated pairs, the base model’s answer correctness is 36.7; on non-associated pairs, it is 26.6. This indicates that local language-country alignment remains easier than cross-context transfer.

The reported full-test results for the main methods are as follows:

Method	Reasoning score	Language mismatch	Answer correctness
Qwen2.5-7B-Instruct	45.6	14.2	28.6
SFT	not stated in provided excerpt	1.3 ± 0.1	29.4 ± 0.1
GRPO	not stated in provided excerpt	0.9 ± 0.1	28.6 ± 0.0
BRIDGE	45.7 ± 0.1	1.1 ± 0.2	30.0 ± 0.2

BRIDGE is the strongest overall method on GeoFact-X. Its most salient effect is the reduction of language mismatch from 14.2 to 1.1, while slightly improving answer correctness and preserving reasoning quality. GRPO alone reduces mismatch even further, to 0.9 ± 0.1, but does not improve answer correctness beyond the base model. SFT alone also nearly eliminates mismatch and provides a smaller accuracy gain. The paper therefore presents BRIDGE as the most balanced method rather than the universal best on every metric slice.

The per-language analyses are also informative. For Hindi, answer correctness is low overall, but BRIDGE improves the associated India-Hindi score from 13.6 to 19.7 ± 1.7, slightly above SFT at 19.3 ± 1.1. Language mismatch was already low in the base model, which suggests that Hindi is less a language-detection problem than a factual reasoning or recall problem.

For Japanese, BRIDGE improves the associated Japan-Japanese answer correctness from 35.6 to 42.5 ± 2.0, reasoning score from 51.6 to 52.1 ± 1.1, and reduces language mismatch from 2.3 to 0.4 ± 0.7. This is one of the clearest cases in which language fidelity and factual outcome improve together.

For Swahili, the base model shows substantial mismatch and weak accuracy. On the associated Kenya-Swahili pair, answer correctness rises from 18.6 to 28.3 ± 3.4 with BRIDGE and 29.1 ± 4.0 with SFT. The paper notes that BRIDGE’s mismatch is not always the very lowest on Swahili slices, which reinforces the broader pattern that its strength is balance rather than universal best-in-cell performance.

For Thai, baseline mismatch is especially high in non-associated settings. All training methods reduce mismatch sharply. On the associated Thailand-Thai pair, BRIDGE increases answer correctness from 23.5 to 24.3 ± 1.8. Here the larger effect is language consistency rather than answer accuracy.

The broader paper also reports that many models on MGSM have much lower joint accuracy than plain math accuracy because they answer correctly while reasoning in the wrong language. Although MGSM is not GeoFact-X, this result is conceptually central: it supports the claim that answer-only multilingual evaluation systematically understates language-alignment failures.

7. Limitations and significance

Several limitations are explicit in the paper (Hwang et al., 7 Jul 2025). The reasoning traces are synthetically generated by Gemini 2.0 Flash, not manually authored by native speakers. The authors perform filtering and manual validation on a 10% stratified subset, but the appendix states that complete verification was still in progress. Evaluation relies on LLM-as-a-judge, which is scalable and nuanced but not fully calibrated in the provided excerpt. The experiments are limited to 7B-parameter models due to resource constraints, and the benchmark itself covers only five languages and five countries.

These limitations do not negate the benchmark’s main contribution, but they bound its current scope. A plausible implication is that future extensions would need more exhaustive human validation, broader language coverage, and stronger calibration of automatic judging.

GeoFact-X nonetheless fills a specific gap in multilingual LLM evaluation. It operationalizes a stronger criterion than answer correctness alone: whether a model can answer correctly, explain coherently, and do so in the language of the question. By using geography-based factual reasoning, it makes multilingual evaluation culturally grounded rather than purely abstract. By attaching reference reasoning traces, it supports both reasoning-aware training and reasoning-aware evaluation. By measuring language mismatch explicitly, it surfaces an otherwise hidden English bias in multilingual models.

In the paper’s overall architecture, GeoFact-X and BRIDGE form a coupled contribution. GeoFact-X provides benchmark data and training supervision for multilingual factual reasoning with language-aligned explanations, while BRIDGE offers a practical hybrid of supervised learning and language-consistency-guided reinforcement fine-tuning. The benchmark’s central lesson is that multilingual reasoning should not be evaluated by final answers alone.