Document-Level Relevance Prior

• Document-Level Relevance Prior is a document-oriented signal that aggregates historical, counterfactual, and contextual evidence to inform ranking decisions in retrieval systems.
• It employs various formulations—query-conditioned, context-marginalized, and query-independent—to capture nuanced document relevance across ad-hoc, generative, and multimodal retrieval tasks.
• Integrating these prior signals enhances IR performance by balancing direct query-document matching with broader evidence, mitigating biases and improving overall generalization.

A document-level relevance prior is a document-oriented relevance signal used before, alongside, or instead of direct query–document matching. In recent research, it has been instantiated as a counterfactual signal from previously relevant documents in recurring-query retrieval, a calibrated query-conditioned probability that a document is useful for answering a question, a context-marginalized relevance rate over reranking contexts, and a query-independent bias learned from annotated retrieval data (Keller et al., 6 Feb 2025, Ni et al., 2024, Huang et al., 3 Nov 2025, Valentini et al., 1 Jun 2026). The concept therefore spans ad-hoc retrieval, RAG, generative retrieval, multimodal long-document retrieval, and document-level extraction, but its common function is to inject document-level evidence or preference into downstream ranking or decision making.

1. Definitions and formal variants

Recent work does not use a single formalization of document-level relevance prior. Instead, several related constructions recur. In retrieval for recurring queries on evolving corpora, historical feedback is represented as

$H=\{(q,d,t)\},$

with graded relevance $rel(q,d,t)$, and the previously positive set for query $q$ is

$$D^{+}=\{d \mid (q,d,t)\in H \wedge rel(q,d,t) > 0\}.$$

The prior is explicitly counterfactual: the relevant evidence is taken from the document version at the timestamp when relevance was observed, even if the document has since been deleted, updated, or replaced (Keller et al., 6 Feb 2025).

In RAG-oriented annotation, the prior is query-conditioned and definition-aware. DIRAS treats relevance as whether a paragraph is “helpful for answering the question,” including cases that are only partially relevant, and uses a binary guess plus calibrated confidence as the score. Its appendix formalizes graded gain by assigning partially relevant documents score $0.5$ and relevant documents score $1$, so the confidence score functions as a document-level relevance prior estimate usable for ranking and thresholding (Ni et al., 2024).

In LLM reranking, the prior can be context-dependent rather than fixed. “Contextual relevance” is defined as

$$\theta_{i,q} = \mathbb{E}_{S \sim D_b(d_i)} \Bigl[ \Pr\bigl(d_i \text{ is judged relevant} \mid q, S\bigr) \Bigr],$$

where the expectation is over reranking batches that include document $d_i$. Here the prior is not an intrinsic property of the document alone, but a marginalized relevance probability over different candidate-set compositions and orderings (Huang et al., 3 Nov 2025).

A stricter notion appears in work on dense retrievers’ annotation bias, where the prior is explicitly query-independent:

$\operatorname{P}(R \mid d).$

In that setting, the prior is the probability that a document will be judged relevant based only on document content, not on any query (Valentini et al., 1 Jun 2026). By contrast, some work uses “prior” in a looser IR sense. In cross-lingual retrieval with multilingual BERT, the document-level relevance prior is simply the BM25 score $S_r$, interpolated with sentence-level neural relevance, and the paper is explicit that this is a document-level lexical relevance signal rather than a Bayesian prior (Shi et al., 2019).

2. Antecedents in ranking and semantic targeting

Earlier IR models already treated relevance as more than direct lexical match. A multi-criteria ranking model based on the Choquet integral defined per-criterion scores $rel(q,d,t)$0 and learned a fuzzy measure over subsets of criteria, thereby modeling not only criterion importance but also interaction among relevance dimensions such as topicality, recency, authority, and user relevance (Moulahi et al., 2014). This moved document scoring beyond additive independence and made relevance aggregation itself into a learned document-level prior.

The role-relevance model applied a similar intuition to personalized search. It combined three document-level signals—keyword evidence, geographic or entity relevance, and topical relevance—into a convex score

$rel(q,d,t)$1

with $rel(q,d,t)$2 and $rel(q,d,t)$3 in the reported experiments. On a pre-labeled Reuters corpus, it reported an average improvement in search precision of approximately $rel(q,d,t)$4 compared to keyword search alone (George et al., 2018). Here the prior is role-conditioned semantic fit: documents “about the right thing” are promoted above documents that merely contain the right words.

Cross-lingual document retrieval provides another antecedent. The multilingual BERT reranker separates fine-grained sentence evidence from a document-wide lexical prior, scoring documents by

$rel(q,d,t)$5

where $rel(q,d,t)$6 is BM25 and $rel(q,d,t)$7 are top sentence-level mBERT relevance scores. The model improved over BM25 in monolingual and cross-lingual retrieval across Chinese, Arabic, French, Hindi, Bengali, and English, indicating that a coarse document-level prior can stabilize local semantic evidence under both language shift and domain shift (Shi et al., 2019).

3. Priors derived from feedback, judgments, and synthetic evidence

The clearest recent retrieval use of document-level relevance prior is recurring-query retrieval on an evolving corpus. Three formulations were compared: direct boosting of previously judged documents, tf-idf query expansion from previously relevant documents, and keyqueries that force those documents to the top of the ranking. The direct boosting method is strong when the corpus is stable, but it cannot generalize to new documents. By contrast, the rewriting methods transfer relevance knowledge through content rather than fixed document identities. On CLEF LongEval, BM25$rel(q,d,t)$8 and especially BM25$rel(q,d,t)$9 improved retrieval effectiveness, outperformed five baselines including ColBERT, List-in-T5, and monoT5, and still improved on unseen documents when previously seen documents were removed, whereas BM25$q$0 gave no improvement in that ablation (Keller et al., 6 Feb 2025).

A related but distinct use of priors appears in generative relevance feedback. GRM observes that the quality of LLM-generated expansion documents varies drastically: on Robust04 titles, MAP ranged roughly from $q$1 for the worst generated documents to $q$2 for the oracle-best one, and Recall@1k ranged from $q$3 to $q$4. GRM therefore weights generated documents by an estimated relevance score $q$5 derived from semantically similar real documents, rather than treating all synthetic feedback items uniformly. The best practical model, GRM-T5, improved MAP by about $q$6–$q$7 and Recall@1k by $q$8–$q$9 over prior state of the art on Robust04 and CODEC (Mackie et al., 2023). The prior here is document-level because terms are weighted indirectly through the relevance of the generated document they came from.

Judgment extension for evaluation uses the same principle. Topic-specific monoT5 classifiers with LoRA are trained on the judgments of a single assessor for a single topic so that the classifier approximates that assessor’s notion of relevance for previously unseen documents in the same pool. As little as $$D^{+}=\{d \mid (q,d,t)\in H \wedge rel(q,d,t) > 0\}.$$0 initial human judgments per topic were reported to suffice to improve comparability, and the strongest settings achieved Spearman’s $$D^{+}=\{d \mid (q,d,t)\in H \wedge rel(q,d,t) > 0\}.$$1 between system rankings induced by predicted versus original judgments (Gienapp et al., 6 Oct 2025). In this case, the prior is not used to retrieve documents directly, but to extend qrels in a topic-bound manner.

4. Calibrated, contextual, and structural priors

In RAG, document-level relevance prior is frequently operationalized as a calibrated utility score rather than as a ranking heuristic. DIRAS uses a pointwise prompt with explicit relevance definitions, outputting [Guess] and [Confidence], and evaluates the resulting scores with ECE, Brier Score, AUROC, F1, nDCG, MAP, and AP for uncertainty prediction. Because the score is interpreted as a calibrated relevance probability, the paper recommends threshold-based retrieval over naive fixed top-$$D^{+}=\{d \mid (q,d,t)\in H \wedge rel(q,d,t) > 0\}.$$2, arguing that different questions require different amounts of relevant evidence. It further reports that fine-tuned open models, especially Llama3-Tok, achieve GPT-4-level performance on ranking and calibration, and that expert-informed definitions improve MAP and nDCG when ranking all $$D^{+}=\{d \mid (q,d,t)\in H \wedge rel(q,d,t) > 0\}.$$3k $$D^{+}=\{d \mid (q,d,t)\in H \wedge rel(q,d,t) > 0\}.$$4 pairs (Ni et al., 2024).

Context-sensitive LLM reranking pushes this further by making the prior explicitly batch-dependent. The TS-SetRank framework models each document’s contextual relevance with a Beta posterior, initialized by $$D^{+}=\{d \mid (q,d,t)\in H \wedge rel(q,d,t) > 0\}.$$5 and updated from binary relevance observations across sampled batches. The posterior mean

$$D^{+}=\{d \mid (q,d,t)\in H \wedge rel(q,d,t) > 0\}.$$6

is then used for ranking, while Thompson sampling balances exploration and exploitation under a fixed inference budget. On BRIGHT, contextual factors explained about $$D^{+}=\{d \mid (q,d,t)\in H \wedge rel(q,d,t) > 0\}.$$7 of total variability for batch size $$D^{+}=\{d \mid (q,d,t)\in H \wedge rel(q,d,t) > 0\}.$$8 and about $$D^{+}=\{d \mid (q,d,t)\in H \wedge rel(q,d,t) > 0\}.$$9 for batch size $0.5$0, with most of the added variability coming from order sensitivity. Empirically, TS-SetRank improved nDCG@10 over retrieval and reranking baselines by $0.5$1–$0.5$2 on BRIGHT and $0.5$3–$0.5$4 on BEIR (Huang et al., 3 Nov 2025).

In long multimodal document retrieval, the prior becomes structural. EviProp builds a heterogeneous Chunk–Page graph with hierarchical, sequential, and similarity edges, combines dense visual page priors with sparse chunk seeds, and diffuses relevance via Personalized PageRank:

$0.5$5

Final page ranking interpolates direct visual retrieval and diffused posterior relevance,

$0.5$6

On MMLongBench-Doc, Top-3 Recall improved from $0.5$7 to $0.5$8, Top-5 Recall from $0.5$9 to $1$0, and Top-5 NDCG from $1$1 to $1$2; gains were also reported on LongDocURL and in downstream QA (Zhang et al., 8 Jun 2026). The prior is query-conditioned, graph-diffused, and explicitly designed to recover evidence pages that page-only scoring under-ranks.

5. Granularity, generation, and transfer beyond ad-hoc retrieval

A recurrent issue in document relevance is granularity. HiNT argues that ad-hoc retrieval cannot assume either document-wide relevance or passage-only relevance, because a document may be relevant as a whole or because one passage is highly relevant. Its hierarchical architecture computes passage-level signals and then allows local and accumulated evidence to compete in a hybrid global decision layer. The paper is explicit that this is not a separate prior term in the Bayesian sense, but an implicit document-level relevance estimator. On MQ2007 and MQ2008, HiNT outperformed BM25, LambdaMart, MSP, PLM, PPM, and DeepRank, and the ablation showed a progression from Independent Decision to Accumulative Decision to Hybrid Decision, with the hybrid model performing best (Fan et al., 2018).

Generative information retrieval exposes a different misalignment. DDRO argues that token-level next-token training over docids does not directly optimize document-level relevance. It therefore replaces reward-model-plus-RL pipelines with direct pairwise optimization over $1$3 and defines an implicit reward

$1$4

On MS MARCO Document, DDRO improved MRR@10 by $1$5 over GenRRL (Sum); on Natural Questions, the improvement was $1$6 (Mekonnen et al., 7 Apr 2025). Here the prior is embedded in the model’s document-level generative preference relative to a reference policy.

Document-level extraction tasks use analogous constructions. DocuNet treats document-level relation extraction as semantic segmentation over an entity-pair matrix, using a U-shaped network to encode local and global interdependency among triples (Zhang et al., 2021). SCPRG for document-level event argument extraction injects two prior-like signals: non-argument contextual clue information via Span-Trigger-based Contextual Pooling and semantic relevance among roles via Role-based Latent Information Guidance; it reported improvements of $1$7 F1 on RAMS and $1$8 F1 on WikiEvents (Liu et al., 2023). RelPrior in LLM-based DocRE uses binary relation as a prior to filter unrelated entity pairs and predefined relations as priors for head–tail matching, improving over prior LLM-based methods on DocRED and RE-DocRED (Pi et al., 11 Nov 2025). FewDocAE clarifies why such priors matter in low-resource settings: in DocEE, arguments are spread across documents averaging $1$9 sentences, and more than $$\theta_{i,q} = \mathbb{E}_{S \sim D_b(d_i)} \Bigl[ \Pr\bigl(d_i \text{ is judged relevant} \mid q, S\bigr) \Bigr],$$0 of labels are O, making purely local or weakly supervised matching especially difficult (Yang et al., 2022).

6. Biases, limitations, and methodological disputes

Document-level relevance priors are not always beneficial. Supervised dense retrievers can internalize annotation artifacts as query-independent preferences. A study of bge, nv-embed, and gte showed that simple logistic regressions on frozen document embeddings recover a prior $$\theta_{i,q} = \mathbb{E}_{S \sim D_b(d_i)} \Bigl[ \Pr\bigl(d_i \text{ is judged relevant} \mid q, S\bigr) \Bigr],$$1 that generalizes to unseen documents with AUC above chance on several held-out datasets, correlates across retrievers, and produces a findability gap: documents with higher prior are systematically easier to retrieve even when genuinely relevant documents are matched on confounders (Valentini et al., 1 Jun 2026). The favored documents were described as comprehensive, self-contained, introductory, or mainstream, while niche, fragmentary, or highly technical content was often left unjudged and correspondingly harder to retrieve.

LLM-based judging raises related questions about what evidence a prior is allowed to compress. Summary-based relevance judgments using GPT-4o and Llama-3.1-8B showed that summaries can preserve system-ranking stability while shifting label distributions and introducing model- and dataset-dependent biases. For GPT-4o, weighted $$\theta_{i,q} = \mathbb{E}_{S \sim D_b(d_i)} \Bigl[ \Pr\bigl(d_i \text{ is judged relevant} \mid q, S\bigr) \Bigr],$$2 under summary-based judging was often comparable to full-document judging, ranking correlations were frequently above $$\theta_{i,q} = \mathbb{E}_{S \sim D_b(d_i)} \Bigl[ \Pr\bigl(d_i \text{ is judged relevant} \mid q, S\bigr) \Bigr],$$3, and token usage on RAG-24 dropped by about $$\theta_{i,q} = \mathbb{E}_{S \sim D_b(d_i)} \Bigl[ \Pr\bigl(d_i \text{ is judged relevant} \mid q, S\bigr) \Bigr],$$4 on the judged pool, but the paper also reported mild score inflation and lower RBO at the very top of the ranking (Mohtadi et al., 5 Dec 2025). Summarization therefore functions as a practical proxy, but not a neutral one.

Evaluation methodology is another point of dispute. Topic-specific classifiers were proposed partly because LLM-as-a-judge can be circular, prompt-sensitive, and less reliable than topic-specialized models trained from seed judgments (Gienapp et al., 6 Oct 2025). DIRAS likewise argues that fixed top-$$\theta_{i,q} = \mathbb{E}_{S \sim D_b(d_i)} \Bigl[ \Pr\bigl(d_i \text{ is judged relevant} \mid q, S\bigr) \Bigr],$$5 retrieval is often suboptimal and that relevance is definition-dependent rather than purely semantic, with partial relevance and uncertainty needing explicit treatment (Ni et al., 2024). A recurring misconception is to treat a relevance prior as a single static score. The literature instead includes counterfactual historical priors, calibrated query-conditioned priors, context-marginalized priors, graph-diffused priors, and learned annotation biases. This suggests that the central methodological question is not whether a prior exists, but which evidence source defines it, how it is calibrated, and whether it improves generalization without importing undesirable bias.