Linguistic Nepotism: Trading-off Quality for Language Preference in Multilingual RAG

Abstract: Multilingual Retrieval-Augmented Generation (mRAG) systems enable LLMs to answer knowledge-intensive queries with citation-supported responses across languages. While such systems have been proposed, an open questions is whether the mixture of different document languages impacts generation and citation in unintended ways. To investigate, we introduce a controlled methodology using model internals to measure language preference while holding other factors such as document relevance constant. Across eight languages and six open-weight models, we find that models preferentially cite English sources when queries are in English, with this bias amplified for lower-resource languages and for documents positioned mid-context. Crucially, we find that models sometimes trade-off document relevance for language preference, indicating that citation choices are not always driven by informativeness alone. Our findings shed light on how LLMs leverage multilingual context and influence citation behavior.

• The paper introduces a controlled methodology to quantify language bias in citation behavior across eight languages.
• It reveals that LLMs consistently prefer citing English documents even when non-English sources are more relevant.
• Layer-wise analysis indicates that citation decisions solidify early in the model, highlighting challenges for bias mitigation and scalability.

Linguistic Nepotism in Multilingual Retrieval-Augmented Generation: Quality Trade-offs and Language Preference

Introduction and Motivation

The paper addresses a critical issue in multilingual Retrieval-Augmented Generation (mRAG): the systematic language preference exhibited by LLMs when generating citation-supported responses. Specifically, it investigates whether LLMs trade off document relevance for language preference, with a focus on long-form mRAG tasks. The authors introduce a controlled methodology to measure language preference using model internals, isolating the effect of document language while holding relevance and content constant. This approach enables precise quantification of citation behavior across eight languages and six open-weight models.

Figure 1: Overview of the controlled methodology for measuring language preference, including synthetic data generation and next-token prediction analysis.

Controlled Experimental Design

The methodology consists of four key steps:

1. Synthetic Data Generation: English evidence documents are translated into multiple target languages using machine translation, ensuring parallel content across languages.
2. Reference Report Generation: Citation-supported reports are generated for each query using a strong LLM, segmented into sentence-level statements with single-document citations.
3. Statement Pool Construction: A two-stage filtering pipeline (LLM-as-Relevance-Judge and NLI entailment) ensures that only verifiably supported statements are retained, minimizing hallucinated or spurious citations.
4. Measurement via Next Token Prediction: Citation accuracy is measured by prompting models to predict the correct citation ID as the next token, varying only the language of the cited document while keeping all other factors fixed.

This design allows for direct comparison of citation accuracy across languages, controlling for confounding variables such as document relevance and position in context.

Empirical Findings: English Preference and Amplifying Factors

Across all tested models and languages, the study finds a consistent and statistically significant preference for citing English documents when the query is in English. This bias is amplified for lower-resource languages (e.g., Bengali, Swahili) and when the cited document is positioned mid-context, reflecting the "lost in the middle" phenomenon in long-context LLMs.

Figure 2: Accuracy difference between English and each target language binned by relative position, showing the largest drop for mid-context documents.

The preference persists even in models explicitly trained for multilingual tasks, and is more pronounced in smaller models (8B) compared to larger ones (70B, 27B), indicating that model scale improves citation accuracy but does not eliminate language bias.

Layer-wise Analysis of Citation Decisions

To probe the internal decision process, the authors employ logit lens analysis, tracking the evolution of token predictions across model layers. The results show that models make a decisive choice of which document to cite at a specific layer (e.g., layer 22 in LLaMA-3.1 8B), and this choice is largely preserved in subsequent layers. The gap between correct and incorrect predictions narrows for lower-resource languages, further confirming the amplified English preference in these cases.

Figure 3: Logit lens visualization for LLaMA-3.1 8B, showing the layer-wise evolution of citation predictions per language.

Query Language Effects

When the query is posed in a non-English language, models exhibit a query-language preference: citation accuracy is highest when the cited document matches the query language, especially when there is a language contrast between the cited and distractor documents. This mirrors human citation behavior in scientometrics, where "own-language preference" is well-documented.

Figure 4: Accuracy per model for queries in the target language, demonstrating query-language preference in citation behavior.

Relevance vs. Language Preference: Quality Trade-offs

The study further examines scenarios with both relevant and irrelevant evidence documents. It finds that models frequently cite irrelevant English documents over relevant non-English ones, indicating that language preference can outweigh document relevance. Conversely, irrelevant distractors in the target language are more easily ignored than English distractors, reinforcing the dominance of English preference in citation selection.

Figure 5: Accuracy per model with one relevant and one irrelevant evidence document in different languages, showing trade-offs between relevance and language preference.

Human Annotation and Attribution Analysis

Human annotation validates the automatic filtering process, with annotators reliably distinguishing supported from unsupported statements. Attribution analysis using ContextCite further reveals that models rely more on English sources during generation, not just in surface-level citation patterns but in contributive attribution as well.

Implications and Future Directions

The findings have significant implications for the design and deployment of mRAG systems:

• Bias Mitigation: The strong and persistent English preference necessitates explicit bias mitigation strategies in multilingual retrieval and generation pipelines, especially for low-resource languages.
• Evaluation Protocols: Controlled, model-internal metrics should be adopted for robust evaluation of citation correctness and language preference, moving beyond coarse frequency-based measures.
• Model Training: Scaling model size improves citation accuracy but does not eliminate language bias; targeted multilingual pretraining and fine-tuning may be required.
• Real-world Deployment: The trade-off between relevance and language preference poses risks for knowledge access equity, particularly in search and scientific information retrieval applications.

Theoretically, the work highlights the entanglement of language modeling and retrieval augmentation, suggesting that cross-lingual generalization remains a challenge even in state-of-the-art LLMs. Future research should explore dynamic context adaptation, improved multilingual document representation, and retrieval strategies that explicitly balance relevance and language diversity.

Conclusion

This paper provides a rigorous, controlled analysis of linguistic nepotism in mRAG, demonstrating that LLMs systematically trade off document relevance for language preference, with a pronounced bias toward English. The methodology and findings offer actionable insights for building more robust, inclusive multilingual retrieval-augmented systems, and underscore the need for continued research into mitigating language bias in AI-driven knowledge access.