Language Bias under Conflicting Information in Multilingual LLMs

Abstract: LLMs have been shown to contain biases in the process of integrating conflicting information when answering questions. Here we ask whether such biases also exist with respect to which language is used for each conflicting piece of information. To answer this question, we extend the conflicting needles in a haystack paradigm to a multilingual setting and perform a comprehensive set of evaluations with naturalistic news domain data in five different languages, for a range of multilingual LLMs of different sizes. We find that all LLMs tested, including GPT-5.2, ignore the conflict and confidently assert only one of the possible answers in the large majority of cases. Furthermore, there is a consistent bias across models in which languages are preferred, with a general bias against Russian and, for the longest context lengths, in favor of Chinese. Both of these patterns are consistent between models trained inside and outside of mainland China, though somewhat stronger in the former category.

• The paper demonstrates that multilingual LLMs consistently fail to detect conflicting information, defaulting to a single confident answer.
• The study applies a multilingual NIAH paradigm and hierarchical Bayesian modeling to reveal significant language retrieval biases across five languages.
• The findings imply systemic limitations in LLM architectures, highlighting the need for improved contradiction detection and unbiased multilingual processing.

Biases in Handling Conflicting Information Across Languages in Multilingual LLMs

Methodological Overview

The paper employs a multilingual extension of the "conflicting needles in a haystack" (NIAH) paradigm to systematically probe language bias in information retrieval using contemporary multilingual LLMs. Conflicting pseudo-facts (“needles”) are embedded in authentic news domain texts (“haystacks”) translated into five languages (English, Chinese, German, Russian, Turkish). The experimental protocol generates contrastive haystack pairs differing only in the language in which conflicting facts are presented, facilitating robust assessment of language-driven retrieval preference. Thirteen LLMs, spanning major architectures and geographic training origins (West vs PRC), are evaluated with varying context lengths (up to 25,000 tokens) and prompt languages, ensuring coverage of context-dependent retrieval dynamics.

Failure to Detect Conflicts

Across model families and haystack sizes, all LLMs essentially fail to recognize or flag the presence of conflicting information in context. The models almost universally produce a single, confident answer, neglecting alternate conflicting facts even when both are equally accessible. This pattern persists irrespective of prompt language, model size, or geopolitical origin. The rate of explicit conflict detection is negligible (ranging from $<1\%$ to $5\%$), with rare exceptions in short contexts and under minimally restrictive prompts. Notably, this inability to signal uncertainty or enumerate alternatives persists even in commercial and instruct-tuned models, indicating systemic limitations well beyond isolated architectural or training differences.

Language Bias in Retrieval

When conflicting pseudo-facts are distributed across languages, the retrieval process exhibits nontrivial, statistically significant biases. Language pairing analysis reveals:

• Information presented in Russian is robustly disfavored compared to all other evaluated languages—even though monolingual Russian haystacks are retrieved accurately.
• In long contexts, Chinese exhibits a significant bias, being preferred over German, English, Russian, and Turkish.
• English and Turkish compete with variable preference, with Turkish sometimes being selected over English and other Latin-alphabet languages.

The bias is validated quantitatively via hierarchical Bayesian modeling of retrieval log-odds, with posterior estimates confirming the magnitude and consistency of biases for Russian (strongly negative) and Chinese (strongly positive in long contexts).

Consistency Across Model Origins

Bias patterns are observed both in LLMs trained in the West and in the PRC, with only minor differences in relative preference for English in PRC-trained models. Both model groups maintain the universal bias against Russian and tendency to prefer Chinese in longer contexts. The scale of bias is slightly elevated for PRC-origin models, but the qualitative structure remains the same. This suggests that such language retrieval biases are not merely a function of training corpus or political context, but emerge across diverse training pipelines and architectures.

Empirical Highlights and Bold Claims

• Every tested LLM—regardless of architecture or origin—fails to recognize explicit contradictions when presented with conflicting information, asserting a single answer in the majority of cases.
• The bias against Russian is nearly universal across all models and context lengths, and is statistically robust.
• Chinese becomes the favored language for retrieval in the largest contexts, regardless of whether the LLM was trained in or outside mainland China.
• Model origin has only a marginal effect: PRC models show a slight bias against English relative to their Western counterparts.

Theoretical and Practical Implications

These findings expose critical limitations in LLMs’ capacity for contradiction detection and resolution, undermining their reliability in document-level information retrieval and summarization tasks. The results indicate that the prevailing prompting and training methodologies fail to enable robust conflict reasoning, particularly in multilingual settings. The observed language biases complicate expectations of impartial cross-linguistic summarization, raising concerns for applications in knowledge aggregation, multi-source QA, RAG systems, and multilingual fact-checking. The universal biases, particularly against Russian (despite accurate monolingual retrieval), also hint at deeper systemic mechanisms, possibly rooted in tokenization, script handling, or contextual embedding interaction.

From a theoretical perspective, the invariance of bias across model families and training regions points to emergent properties in foundation models, possibly related to context compression, localization heuristics, or language modeling regularities. The failure of repetition- or layout-based mitigation strategies, alongside the persistence of biases regardless of source credibility (Schuster et al., 7 Jan 2026), further supports this hypothesis.

Directions for Future Research

Future developments should prioritize the investigation of bias mechanisms, including analysis of token-level attention, script and orthography effects, and the contribution of training data distributions. The expansion to a broader and more diverse set of languages—especially those with varying scripts and typological properties—would inform the generality of observed biases. There's also a need for research into training protocols or architectural changes (e.g., adversarial contrastive fine-tuning, explicit contradiction supervision) to enhance conflict detection and language neutrality.

Increased chain-of-thought effort or advanced prompting strategies may ameliorate the detection problem, but are associated with compute overhead. Development of lightweight yet robust contradiction detection submodules or explicit uncertainty signaling within generative LLMs is warranted, especially for deployment in high-stakes multi-source information retrieval scenarios.

Conclusion

The systematic evaluation reveals that current multilingual LLMs consistently fail to signal or handle conflicting information, confidently defaulting to a single answer. Information retrieval across conflicting linguistic contexts displays statistically robust, model-invariant biases, notably against Russian and, for long contexts, in favor of Chinese. These biases persist irrespective of model origin, highlighting systemic issues in model architecture and training. Addressing these limitations requires rethinking LLM training and evaluation paradigms for conflict detection, bias mitigation, and reliable multilingual information aggregation.