BordIRlines: A Cross-Lingual RAG Benchmark

• BordIRlines is a benchmark designed to assess the cross-lingual robustness of retrieval-augmented generation in culturally sensitive territorial disputes.
• It systematically evaluates multilingual retrieval using ground-truthed dispute pairs and thousands of Wikipedia passages across 49 languages.
• It offers rigorous metrics—consistency, BiasScore, and citation variance—that quantify bias and performance differences in cross-lingual model outputs.

BordIRlines is a benchmark designed to assess the cross-lingual robustness of retrieval-augmented generation (RAG) in tasks that are both multilingual and culturally sensitive, with a primary focus on territorial disputes. By coupling ground-truthed dispute pairs—each formulated in multiple languages—with retrieved Wikipedia passages sampled from the full landscape of claimant and reference languages, BordIRlines provides a controlled testbed for evaluating not only the knowledge integration of LLMs but also their susceptibility to cultural or geopolitical bias and their response consistency across languages. The benchmark targets key open questions of knowledgeability, unbiasedness, and consistency within multilingual RAG architectures, filling a critical gap for researchers studying fairness and equity in information retrieval and generation (Li et al., 2024).

1. Motivation and Scope

LLMs are known to both hallucinate and to adapt their outputs to the distributional properties of pretraining data, inheriting biases in the process. While the RAG paradigm mitigates hallucinations by grounding LLM outputs in retrieved external content, it raises a central question: how do the language and provenance of retrieved documents shape the model's answers—specifically in scenarios where cultural and geopolitical disputes drive significant information asymmetries?

Territorial disputes are emblematic of such sensitive tasks, serving as natural adversarial cases: queries regarding disputed territories can elicit answers that differ drastically depending on query language, document language, and the sources retrieved (e.g., “Is Ceuta Spanish or Moroccan?”). BordIRlines was constructed as a rigorous stress-test for such multilingual RAG (abbreviated as xlRAG) settings, enabling the formal study of (i) the interplay between LLMs’ parametric knowledge and non-parametric retrieval, (ii) the dependence of model outputs on the language of the query and retrieved documents, and (iii) the consistency and potential biases in LLM responses as a function of multilingual context composition.

2. Dataset Construction and Statistics

BordIRlines comprises 720 queries derived from the BorderLines dataset, which enumerates 251 distinct territorial disputes. Each query is formulated in the official languages of all claimant countries, resulting in coverage across 49 languages (including English, Arabic, Russian, Ukrainian, Spanish, Traditional and Simplified Chinese, among others).

For each query–language pair:

• The benchmark identifies all relevant Wikipedia articles for the disputed territory and each claimant, splitting these into individual paragraphs (passages).
• Each passage is embedded and retrieval is performed using several IR systems (OpenAI-embedding + Chroma, M3-Embedding hybrid), storing the top-10 passages per query (7,200 contexts in total).
• Ground truth answers are binary (A vs. B), determined based on international recognition and BorderLines annotation.

Table 1. BordIRlines Dataset Statistics

Statistic	Value
Number of territories	251
Number of queries	720
Number of languages	49
Number of retrieved passages (top-10)	7,200
Avg. articles per territory	8.46
Avg. words per article (en / non-en)	5,263 / 3,492

This scale and annotation rigor enables systematic assessment of multilingual retrieval settings and ground-truthed, language-specific answer distributions (Li et al., 2024).

3. Retrieval Modes and Formal Evaluation Formalism

All passages and queries are embedded via a shared encoder $f(\cdot)$. Given a query $q$ in language $\ell_q$ and a candidate passage $d$ in language $\ell_d$:

• Similarity score: $\operatorname{similarity}(q, d) = \cos(f(q), f(d))$
• Monolingual Retrieval: $R_{\ell}(q)$ retrieves top-$k$ passages from $D_{\ell}$, the set of all passages in language $\ell$.
• Cross-lingual (Bilingual) Retrieval: $q$0 retrieves from $q$1 for query in $q$2.
• Multilingual Retrieval: Retrieve top-$q$3 from each language in $q$4, pool, and re-rank for overall top-$q$5.
• Context Aggregation: Prompts may use (a) only in-language passages ("in-language RAG"), (b) claimant-language cross-lingual retrieval, or (c) a balanced mixture (multilingual RAG).

This design enables systematic ablation between context settings to isolate the effects of language provenance on both retrieval and LLM answer behavior.

4. Evaluation Metrics

Each query $q$6 is evaluated in all relevant languages $q$7; candidate answers $q$8 are binary. Key metrics quantify:

• Consistency across languages:

$q$9

where $\ell_q$0 is empirical answer frequency and $\ell_q$1 is the majority answer.

• Geopolitical Bias Score:

$\ell_q$2

Aggregate: $\ell_q$3 (lower is better).

• Citation Variance: For cited supporting spans,

$\ell_q$4

with $\ell_q$5 the number of citations from language $\ell_q$6 out of total $\ell_q$7, higher variance indicates monolingual citation dominance.

These metrics rigorously quantify not only answer agreement and bias, but also the degree to which models rely on a single language in justifying their answers.

5. Experimental Pipeline

The main evaluation pipeline employs:

• LLMs: GPT-4 (gpt-4-1106-preview, temperature $\ell_q$8).
• IR Systems: OpenAI text-embedding-3-large + Chroma (cosine similarity); M3-Embedding hybrid; ablated with BM25, mDPR, and ColBERT.
• Retrieval depth: $\ell_q$9 per language; prompts use top $d$0 passages.
• Prompt Structure: "Given the passages below and the question in language $d$1, choose exactly one answer (A or B) in $d$2, then briefly justify citing spans."
• Pipeline: (1) Encode query and passages, (2) Retrieve context per selected mode, (3) Construct prompt, (4) Query LLM for answer and citations.

This systematic setup allows quantitative and qualitative comparison between retrieval compositions with minimal confounders (Li et al., 2024).

6. Results and Analysis

Quantitative Results:

• Monolingual RAG: Consistency $d$3, BiasScore $d$4 across 720 queries.
• Multilingual Retrieval: Consistency increases to $d$5 (+24 points), BiasScore drops to $d$6 (−60%).
• Cross-lingual retrieval: Performance sits between these extremes.

Qualitative Insights:

• Case studies (e.g. Crimea, Golan Heights) reveal that in-language context often induces answers favorable to the controlling power in that language. Balanced multilingual retrieval, conversely, regularly forces model answers to align with internationally recognized control (Ukraine for Crimea), yielding superior cross-lingual agreement.
• In low-resource languages, retrieved passages are generally shorter, resulting in higher citation variance and less robust citation coverage.

A plausible implication is that multilingual retrieval not only improves fairness across languages but also drives models toward more globally accepted, less partisan answers (Li et al., 2024).

7. Broader Impacts and Resource Availability

BordIRlines demonstrates that leveraging multilingual retrieval in RAG can substantially reduce both cultural and geopolitical bias, enhancing the equitability of information systems for culturally sensitive queries. The dataset, complete with queries, annotated Wikipedia passages, and IR scores, as well as all code required to reproduce experiments and metrics, is publicly released at https://huggingface.co/datasets/borderlines/bordirlines, supporting ongoing research into robust and fair cross-lingual retrieval-augmented generation pipelines (Li et al., 2024).

Researchers interested in extending, re-evaluating, or benchmarking their systems on BordIRlines have access to the full benchmark suite, fostering comparative evaluation and development in the study of responsible, bias-aware LLM deployment for global information access.