Do Neural Retrievers Prefer Certain Documents? Evidence of Learned Relevance Priors

Abstract: Neural retrievers are trained to estimate query-document relevance from annotated query-document pairs. Yet annotation protocols may not purely reflect relevance: they select only a subset of documents for labeling, and this selection can favor certain document types over others. We investigate whether supervised bi-encoder retrievers implicitly learn a document-level relevance prior: a query-independent signal encoded in their representation space as a side effect of training on annotated data. We estimate this prior by training simple classifiers on frozen document embeddings and evaluate three state-of-the-art retrievers across multiple IR benchmarks. We find that supervised neural retrievers encode relevance priors that generalize to unseen documents and are consistent across models. These priors create a findability gap: documents with lower prior are systematically harder to retrieve, even when genuinely relevant. This effect appears in supervised dense retrievers but is weaker and less consistent in BM25, and it persists under controlled matched-document comparisons. Using LLM-based explanations, we find that judged-relevant documents tend to be comprehensive, self-contained summaries of mainstream topics, while niche, fragmentary, or highly technical content is often left unjudged. Retrievers internalize this bias, ranking documents with these favored features higher than documents that lack them, independently of their actual relevance. Our findings expose a structural limitation of supervised retrieval: models trained on annotated data do not just learn relevance, but also the implicit document preferences in their training data.

• The paper demonstrates that neural retrievers implicitly learn document-level relevance priors from annotation protocols.
• Controlled experiments show that metrics like AUC > 0.85 confirm the robust, generalizable nature of these priors across datasets.
• The resulting findability gap indicates that niche or non-mainstream documents are systematically disadvantaged in retrieval performance.

Document-Level Relevance Priors in Supervised Neural Retrieval

Introduction

This paper rigorously interrogates the phenomenon of "relevance priors" in state-of-the-art supervised neural retrievers for information retrieval (IR), specifically within the bi-encoder (dense retriever) paradigm. It systematically demonstrates that beyond modeling query-document compatibility, neural retrievers implicitly encode document-level relevance priors—a query-independent preference for certain documents. These priors are robust, generalizable, and originate as side effects of annotation and data selection protocols, which steer neural models toward favoring content-rich, mainstream, or stylistically canonical documents. The work further articulates the tangible practical consequence of these learned priors: the emergence of a findability gap, where documents with low prior scores become systematically harder to retrieve—even when their semantic relevance is high.

Emergence and Generalization of Relevance Priors

The investigation begins with a series of controlled and real-world experiments that demonstrate the emergence of document-level priors in dense retrieval systems. By constructing a synthetic "spurious prior" via the injection of an irrelevant token ([X]) into subsets of training data, the authors show that bi-encoder retrievers can easily absorb such biases, distinguishing between marked and unmarked documents in their embedding spaces, even when no signal is present at test time. Crucially, BM25 and lexical models do not show this behavior: only neural models learn and internalize such priors.

Figure 1: UMAP projections of document embeddings show segregation by spurious token, demonstrating that retrievers robustly internalize the bias.

By extending this analysis to three state-of-the-art models—bge-en-icl, NV-Embed-v2, gte-Qwen2-7B-instruct—evaluated across major benchmarks (e.g., MIRACL, LoTTE, FEVER, MSMARCO, Natural Questions), the work demonstrates that prior models (logistic classifiers over frozen document representations) consistently output high AUC values (often $> 0.85$ in biased datasets) for distinguishing annotated relevant from unjudged documents. These effects generalize robustly to unseen corpora.

Figure 2: AUC of the bge-based prior model on held-out datasets confirms the generalizability of learned priors.

Moreover, prior scores extracted from independently trained retrievers exhibit moderate-to-high Pearson correlation coefficients—even when model architectures and training procedures differ—implicating the shared annotation and data sources as the true progenitor of these priors rather than architectural idiosyncrasy.

Figure 3: Cross-model agreement in relevance prior scores substantiates that priors are consistent features of the entire supervised dense retrieval paradigm.

Findability Gap: Measurable Effects on Retrieval

The document-level relevance prior directly impacts retrieval performance. The paper introduces the metric of query-conditioned "findability," measuring, for each genuinely relevant document, its expected reciprocal rank across all queries for which it is relevant. Empirical results demonstrate a strong positive monotonic relationship between relevance prior and findability: documents with low prior are substantially harder to retrieve under dense models, whereas BM25 does not show analogous trends and, in some cases, is anti-correlated with the learned prior.

Figure 4: Average findability rises with average relevance prior across datasets for neural retrievers, but not for BM25.

Matched control experiments, where high- and low-prior documents are carefully paired on confounders (e.g., length, named entity density, position in article, lexical complexity), confirm the causal effect: the findability gap persists after accounting for potential confounds and is present only in neural retrievers.

Figure 5: Findability differences between high- and low-prior documents remain even under tight controls on confounding variables.

Drivers of Priors: Annotation and Annotation-Induced Content Bias

To understand the underlying mechanisms driving the emergence of relevance priors, the paper employs LLM-based comparative explanations, generating and validating natural language hypotheses about systematic differences between annotated-relevant and unjudged content, as well as between documents at the extremes of the prior distribution.

Key findings include:

• Relevant documents are overwhelmingly comprehensive, self-contained, and mainstream—typically article introductions or summaries—while unjudged or low-prior documents are niche, fragmentary, often lacking context, or overly technical.
• Neural retrievers strongly favor the annotation protocol's implicit content preferences, not universally semantically relevant material.
• The LLM-based explanations are predictive: given such a classifier, out-of-sample classification accuracy of document type routinely surpasses 0.8 in the most biased datasets, confirming the presence of strong, generalizable regularities.

  Figure 6: Dataset-dependent distributions of relevance prior scores illustrate protocol-induced variability and define thresholds for analysis.

Implications

Theoretical Implications

The existence of a learned, document-level relevance prior in neural retrievers constitutes a structural shift in IR modeling: supervised dense retrievers function as hybrid systems that not only encode query-document compatibility but also a corpus-specific, query-independent prior dictated by annotation and collection design. This duality challenges the implicit i.i.d. assumption in retrieval evaluation and foregrounds annotation artifacts as major determinants of model behavior.

Practical Consequences

The prevalence of relevance priors has several ramifications:

• Accessibility and Equity: Niche, technical, or context-dependent content is inherently disadvantaged, even when it would be highly relevant for certain queries, leading to systematic findability gaps.
• Index Pruning and Efficiency Trade-offs: Approaches leveraging learned priors for index optimization risk codifying undesirable biases unless priors are carefully audited.
• Adversarial Risks: Awareness of the features associated with high prior could be exploited, intentionally or inadvertently, in content creation to game neural retrievers and ranking systems.

Future Directions

• Extending the analysis to cross-encoder and sparse neural retrieval paradigms.
• Investigating the interplay between document-level priors and retrieval-augmented generation (RAG) pipelines.
• Systematically studying transferability to multilingual or cross-lingual IR, where language or cultural priors may be amplified.
• Developing mitigation strategies: e.g., prior-aware debiasing losses, more inclusive annotation protocols, or explicit decorrelation during training.

Conclusion

This paper delivers a substantial, methodical analysis of the existence, origin, and consequences of document-level relevance priors in supervised neural retrieval, supported by strong empirical and explanatory evidence. The results challenge the prevailing assumption that neural retrievers learn only query-conditioned relevance and illuminate new modes of failure and bias, with significant practical and theoretical impacts on the design and evaluation of both retrievers and IR benchmarks. The findings underscore the importance of annotation protocol design and push the field towards a new understanding of fairness, robustness, and generalization in neural information retrieval.