Retrieval Bias: Concepts & Implications

• Retrieval bias is the systematic divergence between true relevance and what is made accessible by retrieval systems, driven by factors like corpus inequality and sensitive attributes.
• It is measured through metrics such as retrievability, Gini coefficients, and group disparity that reveal uneven document access and differential representation across sources and languages.
• Mitigation strategies incorporate counterfactual filtering, adversarial refinement, and fairness-aware optimization to enhance equitable and robust information access.

Retrieval bias denotes systematic departures between relevance and what a retrieval system actually makes accessible, visible, or selected. In the literature, the term covers several related phenomena: corpus-wide inequality in accessibility measured by retrievability; disproportionate representation of social groups in ranked results; preferential treatment of particular sources, languages, or modalities; and position- or history-dependent preferences in retrieval-augmented generation and LLMs. A common thread is that retrieval outcomes change in structured ways under variations that should not, by themselves, determine relevance: document type, demographic markers, source provenance, temporal position, or the order of knowledge updates (Roy et al., 2022, Ji et al., 19 Mar 2025, Dai et al., 2023, Bajaj et al., 26 Oct 2025).

1. Conceptual scope

The retrievability tradition defines retrieval bias as inequality in “how easily the information within the collection can be accessed with a given retrieval model.” In this view, bias is a property of the system–collection interaction: ranking function, indexing, and rank cut-off determine which items repeatedly enter retrievable positions, and unequal distributions of retrievability across documents indicate that the system itself induces unequal access (Roy et al., 2022).

A second line of work frames retrieval bias around protected or sensitive attributes. In gender representation studies, retrieval bias is the systematic, disproportionate representation of certain social groups in returned content for neutral information needs, distinct from exposure bias, selection bias, and ranking bias. In medical RAG, retrieval bias arises when clinically equivalent questions that differ only in race or gender yield systematically different retrieved evidence sets. In multilingual retrieval, bias appears when certain languages receive better ranking outcomes than others despite semantically equivalent topics. In cross-modal retrieval, it appears as prevalence bias toward high-resource languages and association bias toward culturally proximate images (Krieg et al., 2022, Ji et al., 19 Mar 2025, Yang et al., 2023, Saengsukhiran et al., 30 Oct 2025).

Recent work also extends the concept beyond classical IR into LLM behavior. In dense and neural retrieval, source bias is the tendency to favor LLM-generated text over semantically equivalent human-written text. In long-context LLMs, retrieval bias can mean preferential recovery of earlier rather than current information under repeated updates, or preferential recovery of items near the beginning or end of context rather than the middle. This suggests that retrieval bias is no longer only a property of external search engines; it is also a property of internal retrieval dynamics in sequence models (Dai et al., 2023, Qiao et al., 18 Feb 2026, Bajaj et al., 26 Oct 2025).

2. Formalization and measurement

A canonical measure is document retrievability. With collection $C$, query set $\mathsf{Q}$, rank $k(d,q)$, cut-off $c$, and query weight $w_q$, retrievability is

$$r(d) = \sum_{q \in \mathsf{Q}} w_q \cdot f(k(d,q), c)$$

with

$$f(k(d,q), c) = \begin{cases} 1, & \text{if } k(d,q) \leq c \ 0, & \text{otherwise.} \end{cases}$$

Under uniform $w_q$, $r(d)$ is the count of queries for which $d$ appears within the top-$\mathsf{Q}$0 results. Inequality in the resulting $\mathsf{Q}$1 distribution is summarized with Lorenz curves and the discrete Gini coefficient

$\mathsf{Q}$2

where $\mathsf{Q}$3 is the sorted retrievability value. Higher $\mathsf{Q}$4 means stronger concentration of access among fewer items (Roy et al., 2022).

Group-conditional retrieval bias is often measured by comparing retrieved sets across controlled variants. In demographic RAG, if $\mathsf{Q}$5 denotes the retrieved document IDs for demographic variant $\mathsf{Q}$6, pairwise consistency is the intersection-over-union

$\mathsf{Q}$7

and group disparity is summarized by $\mathsf{Q}$8. The same studies report answer-level fairness with $\mathsf{Q}$9, demographic parity $k(d,q)$0, and equal opportunity $k(d,q)$1 (Ji et al., 19 Mar 2025).

Source bias is quantified in mixed human/LLM corpora by computing standard ranking metrics separately for each source on the same ranked list and then taking a relative difference:

$k(d,q)$2

Negative values indicate preference for LLM-generated documents (Dai et al., 2023).

For repeated knowledge updates in long contexts, retrieval bias is operationalized as the earliest–latest accuracy gap. If $k(d,q)$3 is accuracy on queries asking for the initial value and $k(d,q)$4 is accuracy on queries asking for the most recent value, then

$k(d,q)$5

As update length $k(d,q)$6 increases, larger $k(d,q)$7 indicates stronger competition between historical and current values (Qiao et al., 18 Feb 2026).

In multimodal RAG, position sensitivity is summarized by the Position Sensitivity Index

$k(d,q)$8

with $k(d,q)$9 the accuracy when the gold evidence is placed at position $c$0. For $c$1, $c$2. This isolates the amplitude of positional effects independent of content (Yao et al., 30 May 2025).

These measures are not interchangeable. Retrievability and Gini quantify corpus-wide access inequality; overlap and fairness metrics quantify disparities across controlled groups; relative $c$3 isolates source preference; ELAG measures interference across updates; and PSI measures sensitivity to evidence position. A plausible implication is that “retrieval bias” functions as a family resemblance term rather than a single metricized construct.

3. Retrieval bias in integrated search and dense retrieval

Integrated digital library studies make the access-inequality view concrete. In GESIS Search, datasets and publications are returned on type-specific SERPs under a fixed BM25 model. At $c$4, retrievability Gini is $c$5 and $c$6; at $c$7, $c$8 and $c$9. Top-$w_q$0 coverage is 95.02% for publications and 76.19% for datasets, while usefulness Gini is $w_q$1 for publications and $w_q$2 for datasets. The same system therefore makes datasets substantially harder to surface, more heterogeneously retrievable, and more concentrated in implicit relevance signals than publications (Roy et al., 2022).

The extended GESIS study adds variables and explicitly factors out query popularity by deduplicating queries. At $w_q$3, Gini is $w_q$4 for publications, $w_q$5 for datasets, and $w_q$6 for variables; at $w_q$7, it is $w_q$8, $w_q$9, and $$r(d) = \sum_{q \in \mathsf{Q}} w_q \cdot f(k(d,q), c)$$0, respectively. Including repeated queries materially changes which items are most retrievable: for the top 1,000 items, Jaccard overlap between rankings built from repeated versus unique queries is $$r(d) = \sum_{q \in \mathsf{Q}} w_q \cdot f(k(d,q), c)$$1 for publications and $$r(d) = \sum_{q \in \mathsf{Q}} w_q \cdot f(k(d,q), c)$$2 for datasets. The study explicitly concludes there is “significant popularity bias with certain items being retrieved more often than others” (Roy et al., 2023).

A methodological complication is that retrievability estimates depend heavily on the query set. When real query logs are unavailable, many studies generate artificial queries from corpora. Comparing artificial queries with AOL log queries, one study found “a minimal or negligible correlation between retrievability scores from artificial queries and those from query logs.” The proposed RSQ alternative achieved the highest correlations, with Pearson’s $$r(d) = \sum_{q \in \mathsf{Q}} w_q \cdot f(k(d,q), c)$$3 of 0.2076 on TREC Robust, 0.3135 on WT10g, and 0.3065 on Wikipedia, but these remained modest (Sinha et al., 2024).

Controlled experiments on dense retrievers show that access inequality is not only about document types. Dense models favor superficial heuristics over answer presence. Under combined brevity, early-position, repetition, and literal-match biases, the answer-containing document is selected in less than 3% of cases: Contriever 0.4%, RetroMAE 0.4%, Contriever-MSMARCO 0.8%, Dragon-RoBERTa 0.8%, Dragon+ 1.2%, and COCO-DR 2.4%. In downstream RAG, poisoned retrieval can be worse than no retrieval: for GPT-4o, accuracy is 30.8% with poisoned documents, 62.8% with foil documents, 64.8% with no document, and 93.6% with the true evidence document (Fayyaz et al., 6 Mar 2025).

Code retrieval exhibits an analogous textual bias. Function-name normalization drops HumanEval Recall@1 for GIST-large from 54.3% to 18.9%, and full normalization on MBPP drops Recall@1 from about 98.6% to 9.0%. Under asymmetric normalization, average ground-truth rank for GIST-large jumps from 87.18 to 288.27, showing bias toward well-documented distractors even when they are functionally irrelevant (Gupta et al., 25 Jun 2025).

4. Social, demographic, and multilingual manifestations

Gender representation bias has been operationalized through tightly controlled datasets. Grep-BiasIR contains 118 bias-sensitive, gender-neutral queries across seven categories, each paired with one relevant and one non-relevant document, each document instantiated in female, male, and neutral variants, for 708 documents total. Because the topical content, title, and body are held constant except for gender-indicating words, differences in ranking can be attributed to representation effects rather than content variation (Krieg et al., 2022).

Using a related setup, one study on BERT-based IR models found that adapter-based fine-tuning achieved a zero-shot “Average*” of 0.6935 versus 0.5144 for full fine-tuning, and that the baseline adapter-tuned model preferred male over female variants in six of seven categories, with an average absolute male–female fraction gap of 11.14 ± 5.39%. Adding a score-parity regularizer reduced the average gap to 5.7% while keeping accuracy loss within a 2–5% window (Sundararaman et al., 2022).

Human relevance assessment can itself encode retrieval bias. In graded judgments of bias-sensitive results, the strongest tendency appeared for Domestic Work: relevant female-content documents received a mean score of 2.10 ± 0.86 versus 1.76 ± 0.82 for male-content documents, $$r(d) = \sum_{q \in \mathsf{Q}} w_q \cdot f(k(d,q), c)$$4, $$r(d) = \sum_{q \in \mathsf{Q}} w_q \cdot f(k(d,q), c)$$5. For non-relevant Appearance documents, the significant effect ran in the opposite direction: female 0.70 ± 0.81 versus male 1.00 ± 0.83, $$r(d) = \sum_{q \in \mathsf{Q}} w_q \cdot f(k(d,q), c)$$6, $$r(d) = \sum_{q \in \mathsf{Q}} w_q \cdot f(k(d,q), c)$$7. The paper interprets this as evidence that judged relevance can shift with gendered content even when topical relation is controlled (Krieg et al., 2022).

Demographic retrieval bias is explicit in medical RAG. For DeepSeek-R1-8B on MedMCQA, the Plain system has $$r(d) = \sum_{q \in \mathsf{Q}} w_q \cdot f(k(d,q), c)$$8, $$r(d) = \sum_{q \in \mathsf{Q}} w_q \cdot f(k(d,q), c)$$9, and $$f(k(d,q), c) = \begin{cases} 1, & \text{if } k(d,q) \leq c \ 0, & \text{otherwise.} \end{cases}$$0; Counterfactual raises accuracy to 21.3% and lowers $$f(k(d,q), c) = \begin{cases} 1, & \text{if } k(d,q) \leq c \ 0, & \text{otherwise.} \end{cases}$$1 to 0.09/0.08; Majority Vote reaches 22.9% with $$f(k(d,q), c) = \begin{cases} 1, & \text{if } k(d,q) \leq c \ 0, & \text{otherwise.} \end{cases}$$2 and $$f(k(d,q), c) = \begin{cases} 1, & \text{if } k(d,q) \leq c \ 0, & \text{otherwise.} \end{cases}$$3. On EquityMedQA, ROUGE-L rises from 43.0% to 48.0% from Plain to Majority Vote, while $$f(k(d,q), c) = \begin{cases} 1, & \text{if } k(d,q) \leq c \ 0, & \text{otherwise.} \end{cases}$$4 drops from 0.15 to 0.08 and $$f(k(d,q), c) = \begin{cases} 1, & \text{if } k(d,q) \leq c \ 0, & \text{otherwise.} \end{cases}$$5 from 0.12 to 0.07 (Ji et al., 19 Mar 2025).

Language bias appears sharply in multilingual ranking. In Multi-EuP one-vs-many retrieval, English reaches MRR@100 = 62.79, while German is 28.27, French 24.22, Spanish 22.57, Polish 4.80, and Maltese 0.40. Replacing language-specific analyzers with whitespace tokenization reduces aggregate MRR from 15.02 to 14.18 but increases the explanatory power of collection size and is presented as reducing language bias (Yang et al., 2023).

Previously Fact-Checked Claim Detection exhibits a frequency-based retrieval bias. On MultiClaim, Multilingual E5 obtains Success@1 of 38.80%, Success@20 of 88.50%, and slightly higher MAP/MRR than T5. Yet exposure is highly concentrated: in the top-20 most frequently retrieved claims, the same JavaScript snippet appears in 159 different cases, and topic modeling shows concentration around COVID-19, Ukraine/Russia, viral videos, and region-specific themes such as Nigeria and Kenya (Vykopal et al., 29 Sep 2025).

5. Source, temporal, positional, and perspectival bias in neural retrieval and RAG

Neural retrievers can prefer particular sources independent of relevance. In mixed corpora of human-written and LLM-generated rewrites, lexical retrievers favor human text, but neural retrievers consistently favor LLM-generated content. On SciFact+AIGC with Llama2 rewrites, Relative $$f(k(d,q), c) = \begin{cases} 1, & \text{if } k(d,q) \leq c \ 0, & \text{otherwise.} \end{cases}$$6 on NDCG@1 is $$f(k(d,q), c) = \begin{cases} 1, & \text{if } k(d,q) \leq c \ 0, & \text{otherwise.} \end{cases}$$7 for ANCE, $$f(k(d,q), c) = \begin{cases} 1, & \text{if } k(d,q) \leq c \ 0, & \text{otherwise.} \end{cases}$$8 for BERM, $$f(k(d,q), c) = \begin{cases} 1, & \text{if } k(d,q) \leq c \ 0, & \text{otherwise.} \end{cases}$$9 for TAS-B, and $w_q$0 for Contriever. Re-rankers amplify the effect: MiniLM reaches $w_q$1 and monoT5 $w_q$2 (Dai et al., 2023).

Long-context LLMs show a different retrieval bias under multiple in-context updates. In the Dynamic Knowledge Instance framework, earliest-state accuracy remains high while latest-state accuracy drops as updates increase. On real-world DKIs, LLaMA-3.1-8B has $w_q$3, $w_q$4, and $w_q$5; Qwen-2.5-7B has 99.39%, 88.41%, and 10.98%. Attention, hidden-state similarity, and logits become flatter and weakly discriminative on wrong latest-state cases, providing little stable basis for selecting the newest update (Qiao et al., 18 Feb 2026).

Position-dependent retrieval bias also appears in autoregressive models even when semantics are randomized away. Across transformer and state-space models, next-token probability peaks at the tokens immediately following repeated triggers, but episodes near the beginning or end of the prompt are retrieved more reliably than those in the middle. Induction-head ablations support a causal role for this mechanism: across 16 transformer-model-by-ablation settings, the drop in average +1 probability after ablating high-induction heads exceeded the drop after ablating random heads in 13/16 settings (Bajaj et al., 26 Oct 2025).

In multimodal RAG, position bias becomes a property of evidence presentation. On ChartQA with GPT-4o, accuracy is 0.7333 ± 0.007 when gold evidence is at Top, 0.7059 ± 0.012 at Mid, and 0.8125 ± 0.008 at End, an 11% absolute improvement from Mid to End. Across models, $w_q$6 increases approximately as $w_q$7, with $w_q$8 in $w_q$9, indicating logarithmic amplification as retrieval range grows (Yao et al., 30 May 2025).

Cross-modal retrieval exhibits both prevalence and association bias. On Crossmodal-3600, CLIP-L/14 has DLBKL@10 = 5.684, whereas XLM-R-B/16plus reduces it to 0.125 while maintaining Acc@5 = 0.968 and NDCG@10 = 0.791. On 3XCM/XCM, CLIP-L/14 has $r(d)$0, $r(d)$1, and $r(d)$2, whereas XLM-R-B/16plus has $r(d)$3, $r(d)$4, and $r(d)$5. The paper concludes that explicit alignment is more effective for mitigating prevalence bias, whereas association bias remains more challenging (Saengsukhiran et al., 30 Oct 2025).

6. Mitigation strategies, design implications, and open problems

Mitigation depends on the mechanism producing bias. In integrated digital libraries, proposed responses include increasing effective depth for underexposed item types, metadata enrichment, diversification and de-duplication, cluster-based pseudo-relevance feedback, type-aware field weighting, and ongoing monitoring of retrievability distributions, Lorenz curves, and Gini coefficients as system-health indicators (Roy et al., 2022).

In demographic RAG, mitigation is organized around controlled variants and aggregation. Counterfactual Filtering generates answers across demographic variants and selects the most consistent response; Adversarial Prompt Refinement neutralizes sensitive tokens when they are not clinically necessary; Chain-of-Thought Filtering anchors reasoning in evidence; and Majority Vote aggregation delivered the strongest gains in the reported experiments. The same paper also proposes a fairness-aware retrieval objective that penalizes divergence across demographic variants through $r(d)$6 (Ji et al., 19 Mar 2025).

For source bias in neural retrieval, a plug-and-play debiasing constraint penalizes a higher score for $r(d)$7 than for its paired human-written $r(d)$8:

$r(d)$9

combined with the ranking loss as $d$0. On SciFact+AIGC, increasing $d$1 moves Relative $d$2 from strongly negative toward near-zero or positive while maintaining or slightly improving human-only effectiveness (Dai et al., 2023).

Several studies push mitigation into the scoring architecture. The Backpack-based reranker suppresses the two most gender-sensitive senses at inference time through a control weight $d$3. At cutoff 10, reducing the control weight from 1 to 0.5 lowers TF RaB/ARaB from 0.064/0.064 to 0.053/0.056, with NDCG@10 changing from 0.402 to 0.395. In text-based image retrieval, PBM performs post-hoc group-aware reranking; on Occupation 1, AbsBias@100 drops from .6231 for original CLIP to .0560 for PBM with zero-shot prompt, while maintaining Recall@100 of 46.1 (Afzali et al., 2 Nov 2025, Kong et al., 2023).

In code retrieval, SACL fuses code similarity with LLM-generated semantic descriptions using

$d$4

with $d$5 reported as best across most normalization settings. This raises Recall@1 by 12.8% on HumanEval, 9.4% on MBPP, and 7.0% on SWE-Bench-Lite file localization, suggesting that one way to combat textual bias is to add semantic text that reflects functionality rather than surface form (Gupta et al., 25 Jun 2025).

In RAG more broadly, fairness-aware retrieval optimization treats position-aware bias propagation as an optimization problem. With

$d$6

the objective can maximize relevance subject to $d$7, or use a soft penalty $d$8. The FARO method decomposes the resulting quadratic problem into per-question assignment problems through a dual approximation, enabling scalable exploration of the fairness–relevance frontier (Zhao et al., 15 May 2026).

The literature remains methodologically fragmented. Some studies use real query logs and system-level inequality; others rely on synthetic counterfactuals; others diagnose internal retrieval dynamics in LLMs through attention, logits, or forced-choice triplets. Many papers note limited significance testing, reliance on synthetic demographic or temporal manipulations, binary or coarse group definitions, and dependence on specific models, corpora, or cut-offs. This suggests that future work will need both better causal isolation and broader evaluation regimes: graded exposure models rather than binary top-$d$9, more realistic multilingual and multimodal datasets, and retrieval objectives that treat fairness, diversity, and robustness as first-class constraints rather than post-hoc diagnostics.