Improving Cross-Lingual Factual Recall via Consistency-Driven Reinforcement Learning

Abstract: LLMs trained predominantly on English data encode substantial world knowledge, yet often fail to express it reliably in other languages, a phenomenon known as cross-lingual factual inconsistency. To study and address this, we introduce PolyFact, a large-scale parallel multilingual factual QA dataset containing 100K Wikidata-grounded facts across 12 typologically diverse languages. Using PolyFact, we compare light continual pretraining (CPT), supervised fine-tuning (SFT), and reinforcement learning via Group Relative Policy Optimization (GRPO) for improving cross-lingual factual recall in Qwen-2.5-7B and OLMo-2-1124-7B. We find that GRPO consistently outperforms SFT, improving both cross-lingual consistency and generalization to unseen languages, while CPT on parallel data yields limited additional gains. Mechanistic analyses further show that GRPO reorganizes multilingual routing by reducing language specialization in MLP layers and attention heads, thereby promoting more shared cross-lingual representations. We release our code, models, and dataset.

• The paper proposes consistency-driven reinforcement learning (GRPO) to restructure language routing, directly improving cross-lingual factual recall in LLMs.
• The PolyFact dataset, spanning 12 languages and 100K Wikidata-grounded facts, enables rigorous evaluation and demonstrates significant accuracy gains over baseline methods.
• Mechanistic insights reveal that GRPO induces deeper, language-agnostic internal representations, enhancing performance in both high- and low-resource as well as zero-shot languages.

Improving Cross-Lingual Factual Recall via Consistency-Driven Reinforcement Learning

Motivation and Problem Formulation

LLMs trained on predominantly English corpora encode significant parametric factual knowledge, yet they frequently exhibit cross-lingual factual inconsistency: the ability to recall facts degrades or displays instability when queried in non-English languages. The literature converges on the hypothesis that this does not primarily stem from knowledge deficits but instead from failures during language transition phases—either in the mapping of prompts to internal conceptual spaces or, more commonly, during the decoding process into the target language. These facts motivate the study of how multilingual factual retrieval can be enhanced without computationally expensive and destructive continual pretraining (CPT).

The work introduces PolyFact, a rigorously constructed, parallel factual question-answering (QA) benchmark spanning 100K Wikidata-grounded facts in 12 typologically diverse languages, and investigates whether consistency-driven reinforcement learning (RL) via Group Relative Policy Optimization (GRPO) can directly target and remedy cross-lingual inconsistency by post-training already English-centric LLMs.

PolyFact Dataset Construction

A significant contribution is the PolyFact dataset: a large-scale, curated, parallel, multiple-choice QA corpus. Facts are grounded in Wikidata and span key relations with coverage over English, German, Spanish, French, Portuguese, Indonesian, Russian, Chinese, Japanese, Arabic, Swahili, and Bengali. Each fact is represented in all 12 languages and paired with three type- and length-matched distractors, supporting controlled, cross-lingual evaluation. Validation employs both LLM-based and human judges, with high agreement (91%), and a clean subset is provided to mitigate ambiguity.

Figure 1: Expansion from English-only to coverage of 12 globally dominant languages, raising addressable population fraction from 18.5% to 70%.

Methodology: Post-Training Approaches

Three adaptation methods are assessed:

• Continual Pretraining (CPT): Further training on parallel corpora (TED2025) for representation alignment. CPT improves translation fluency but has limited impact on factual recall.
• Supervised Fine-Tuning (SFT): Joint classification with a cross-lingual consistency term, incentivizing the model to agree on answers across languages.
• Group Relative Policy Optimization (GRPO): RL on PolyFact, conducting rollouts in parallel over all languages and directly rewarding cross-lingual consistency and correctness, with a structured penalty/bonus system to suppress hallucinations and induce shared reasoning pathways.

  Figure 2: GRPO intensifies shared internal representations for factual predictions, whereas SFT leads predominantly to language-specific memorization.

Empirical Results

Performance Gains

GRPO consistently outperforms SFT and CPT on both in-domain (PolyFact) and out-of-distribution (KLAR, Global-MMLU) factual recall tasks, for both Qwen-2.5-7B and OLMo-2-1124-7B. For example, in OLMo-2-1124-7B, GRPO leads to a relative gain from 57.93% to 64.21% in high-resource and 51.80% to 54.48% in low-resource PolyFact accuracy. On KLAR's free-form evaluation, SFT-trained models even underperform baselines, evidencing that RL induces generalization of factual retrieval, not mere candidate memorization.

Cross-Lingual Generalization

GRPO increases performance not only on seen languages but also on held-out (zero-shot) languages—a clear indicator that internal representations become more language-agnostic and are reused cross-lingually, whereas SFT's benefits remain confined to training languages.

Figure 3: GRPO achieves higher accuracy on both trained and held-out languages, in contrast to the limited transfer of SFT.

Mechanistic Insights: Language Routing and Specialization

Thorough mechanistic analysis, using recently introduced interpretability frameworks (LAHIS, LAPE), reveals profound architectural reorganization induced by GRPO:

• Delayed and redistributed language specialization: Non-Latin scripts (Arabic, Japanese, Chinese) allocate language-specific neurons to later layers post-GRPO, while Latin and Cyrillic scripts exhibit earlier specialization, indicating a bifurcation in representation strategies.

  Figure 4: Late-layers accumulate script-dependent specialization, whereas Latin scripts' neurons shift to intermediate layers under GRPO.

• English-alignment effect: GRPO increases the count of English-specific neurons (+38%) and reduces those for certain low-resource languages, underscoring RL's tendency to “squeeze” reasoning through the most stable (English-centric) pathways.
• Routed attention heads: Baseline models concentrate language routing in the first attention layer. GRPO and SFT redistribute this load, but GRPO disperses language-important attention heads more broadly, reducing early routing dependence and yielding deeper, more flexible language identification.

  Figure 5: Language-important heads are less concentrated at low layers post-GRPO, distributing language routing across the stack.

• Cross-lingual consistency in output layers: GRPO-modified models overcome the persistent failure mode where the correct answer is retrieved in English but not correctly expressed in the target language. Layer-wise rank analysis shows target language tokens overtaking English tokens at deeper layers.

Qualitative Error Analysis

Proper noun questions—subject to Wikidata's Anglophone bias—persist as a challenging regime. In these cases, the reward structure can inadvertently reinforce the model's inclination to output English forms or Anglicized answers, particularly in languages closely related by script. This subtle phenomenon suggests the need for further reward shaping or dataset curation.

Broader Implications and Future Directions

The results firmly demonstrate that cross-lingual factual inconsistency is primarily one of knowledge access and routing, not parametric deficiency. Consistency-driven RL (GRPO) efficiently restructures language interfaces towards shared, robust pathways, improving factual access in a scalable, post-hoc manner—circumventing the cost and instability of CPT. While LLMs' underlying English-centric structure persists and even amplifies under RL, routing through stable representations can be leveraged for robust multilingual deployment.

Practical implications include significantly improved performance in mid- and low-resource languages, with positive transfer to unseen scripts contingent on token overlap and representation proximity. The approach supports rapid cross-lingual enhancement of English-centric LLMs, particularly valuable in scenarios with limited parallel corpora availability and where large-scale retraining is infeasible.

Theoretically, the work provides evidence for the utility of consistency rewards in eliciting robust, generalizable latent alignment, opening avenues for fine-grained reward engineering and the design of intrinsic objectives targeting representational entanglement. However, fine reward shaping and benchmark extension are required to mitigate residual bias (especially in proper nouns) and fully generalize gains beyond surface recall to reasoning-intensive and open-domain tasks.

Conclusion

Consistency-driven RL via GRPO, applied to a parallel multilingual factual QA dataset, robustly improves cross-lingual factual recall in English-dominant LLMs. The improvements arise not from increased knowledge, but from restructured language routing and enhanced target-language mapping. Mechanistic analysis reveals deep changes in the allocation of neurons and attention resources, with a clear bifurcation by script and deeper, more language-agnostic inference pathways. The approach effectively enables multilingual robustness without the trade-offs associated with continual pretraining. These findings establish a solid methodological basis for post-hoc cross-lingual adaptation strategies and point to the critical importance of reward structure and dataset design for future advances in multilingual LLM policy shaping.