Retrieval Probability Score (RPS)

• Retrieval Probability Score (RPS) is an entity-level measure that quantifies the likelihood of a target entity appearing in a retriever’s top-k results across proxy query contexts.
• It uses cosine similarity in a controlled candidate pool to isolate geometric biases, thereby identifying systematic blind spots in dense retrieval models.
• RPS enables pre-index auditing and fuels remediation pipelines like ARGUS, which augment documents to improve retriever performance and mitigate miss probabilities.

Retrieval Probability Score (RPS) is an entity-level measure of retrievability risk in neural retrieval. It was introduced to move retrieval evaluation from the usual query-centric, post-hoc setting to a pre-index, entity-centric audit that can identify systematic blind spots: entities that are relevant in principle, but are hard for a dense retriever to surface because of unfavorable embedding geometry. In this formulation, RPS quantifies the expected probability that a target entity appears in a retriever’s top-$k$ list across the contexts in which that entity should be relevant. High RPS indicates consistent retrievability; low RPS indicates a high miss probability and therefore a retrieval blind spot (Taghavi et al., 10 Feb 2026).

1. Definition and problem setting

The central motivation for RPS is the observation that reliable retrieval-augmented generation systems depend fundamentally on the retriever’s ability to find relevant information, yet standard retrieval benchmarks do not reveal which entities are intrinsically hard to retrieve. The blind spots targeted by RPS are defined as the failure to retrieve entities that are relevant to the query, but have low similarity to the query embedding. The underlying claim is that these failures are not merely random retrieval noise; they are training-induced geometric biases in which some entities are mapped to inaccessible parts of the embedding space and therefore have low discoverability under a practical top-$k$ budget (Taghavi et al., 10 Feb 2026).

RPS is described as a “probability of discovery” or “retrieval consistency” score. For a target entity $x$, high RPS means that $x$ is consistently retrieved across its relevant query contexts, whereas low RPS means that $x$ is often missed. The source gives the concrete interpretation that an entity with $\mathrm{RPS}_k = 0.2$ is retrieved in only about 20% of its associated query contexts. In this sense, RPS is not a relevance label and not a benchmark aggregate; it is an entity-level accessibility statistic (Taghavi et al., 10 Feb 2026).

The measure was introduced to serve four explicit purposes: to quantify entity-level retrievability rather than query-level average performance, to decouple geometric accessibility from benchmark query distribution, to enable pre-index risk estimation, and to support remediation by identifying high-risk entities that need augmentation. This makes RPS particularly relevant for pre-deployment auditing of dense retrievers used in RAG pipelines (Taghavi et al., 10 Feb 2026).

2. Formal construction of RPS

For each target entity $x$, the construction begins by collecting a set of related entities $T_x$ from Wikidata 1-hop relations. Each related entity $t_i \in T_x$ is treated as a proxy query. Retrieval is then evaluated in a controlled candidate pool in which the target entity competes against query-specific neutral entities that are strictly disjoint from the query in Wikidata:

$$C_i = \{x\} \cup \{z_{i,1}, \ldots, z_{i,N-1}\}, \quad z_{i,j} \sim \text{Uniform}(Z_{\text{neut}(t_i)}).$$

Candidates are ranked by cosine similarity between the query embedding $k$0 and the candidate embedding $k$1. A top-$k$2 hit is defined as

$k$3

RPS is then the expectation of this hit indicator over the proxy-query distribution:

$k$4

In practice, the expectation is approximated by the empirical average over sampled related entities:

$k$5

A major methodological component is the use of the neutral pool. The neutral entities are query-specific and have no direct 1-hop KG edge to the proxy query. This is intended to make retrieval failures attributable more to embedding geometry than to obvious semantic or graph relations. The source further states that at small pool sizes $k$6, the chance hit rate $k$7 is too high to be diagnostic; measured behavior stabilizes beyond about $k$8, and the conservative default is $k$9. The stability analysis uses $x$0 because it is a difficult setting where randomness is especially problematic (Taghavi et al., 10 Feb 2026).

3. Embedding geometry, representation, and RPS prediction

RPS is defined over the geometry of entity embeddings produced by the retriever. Let $x$1 be the retriever encoder and $x$2 a retriever-specific pooling operator. An entity $x$3 is represented by

$x$4

where $x$5 is the entity’s Wikipedia first paragraph and $x$6 is the token or span corresponding to the mention of the entity surface form in that paragraph. The full Wikipedia paragraph is always encoded, while the mention span is used only for pooling. For token-level models, pooling is span averaging:

$x$7

Proxy queries are embedded analogously from the first paragraph of the related entity:

$x$8

The empirical argument for RPS is geometric. Low-RPS and high-RPS entities occupy different regions in embedding space; the study visualizes this by splitting entities into low, mid, and high RPS terciles and projecting embeddings into two dimensions with LDA. The reported pattern is that low-RPS entities cluster in distinctive regions, suggesting that blind spots are structured geometric zones rather than random retrieval errors. Standard retrievers show increasing dominance of low-RPS regions as neutral competition grows, whereas more robust retrievers preserve more high-RPS mass (Taghavi et al., 10 Feb 2026).

The paper then turns RPS into a supervised prediction problem. A probe $x$9 maps an entity embedding directly to predicted RPS. Three probe families are tested: Ridge regression, MLP, and XGBoost. Training is performed per retriever, with empirical RPS labels computed under the stable audit setting $x$0. The best model is selected by validation RMSE. For risk flagging, the practical threshold is

$x$1

with default global threshold $x$2, which is stated to be slightly below the low-tercile boundary of approximately $x$3. Probe performance is strong enough to support pre-index auditing: Pearson and Spearman correlations are reported roughly in the range $x$4–$x$5, low/mid/high classification accuracy around $x$6–$x$7, and RMSE around $x$8–$x$9. Concrete examples include QWEN3-EMBEDDING + XGBoost with Pearson $x$0 and accuracy $x$1, REASONIR-8B + Ridge with Pearson $x$2 and accuracy $x$3, REASON-EMBED + XGBoost with Pearson $x$4 and accuracy $x$5, and CONTRIEVER + XGBoost with Pearson $x$6 and accuracy $x$7 (Taghavi et al., 10 Feb 2026).

4. ARGUS as an RPS-driven remediation pipeline

RPS is not only a diagnostic statistic; it is also the control signal for ARGUS, short for “Assessing Retrieval Gaps via Uncertainty Scoring.” ARGUS is a two-stage offline pipeline that uses predicted RPS to identify and remedy retrievability blind spots. In the diagnosis stage, for each corpus document $x$8, the system runs named entity recognition using dslim/bert-base-NER, embeds each entity mention using the target retriever, applies the retriever-specific RPS probe, and flags any mention or entity with predicted score below threshold. If an entity appears multiple times, the aggregation is conservative: the document-level score is the minimum over mention occurrences, so any risky mention causes the entity to be treated as risky (Taghavi et al., 10 Feb 2026).

In the remedy stage, ARGUS retrieves defining context for each flagged entity from a reference knowledge base, specifically Wikipedia first paragraphs, using BM25S. Two augmentation strategies are used. In Document Expansion, each retrieved KB passage is appended to the original document, and if a document has $x$9 flagged entities with $\mathrm{RPS}_k = 0.2$0 passages per entity, the procedure yields $\mathrm{RPS}_k = 0.2$1 indexed views including the original. In KB-guided LLM Synthesis, the retrieved passages are aggregated and used by an LLM to generate a single unified augmented document, producing one additional view per document. ARGUS never replaces the original document; it indexes augmented views alongside it (Taghavi et al., 10 Feb 2026).

The stated rationale is geometric rather than purely lexical. A low-RPS entity is a blind spot because the retriever is unlikely to find it from natural queries. By injecting short, entity-focused defining context near the mention, ARGUS aims to move that entity into a more retrievable region of the embedding space. This suggests that RPS functions as both a measurement of retrievability uncertainty and a targeting mechanism for selective augmentation (Taghavi et al., 10 Feb 2026).

5. Empirical behavior and downstream retrieval effects

The empirical evaluation of ARGUS spans three benchmarks—BRIGHT, IMPLIRET, and RAR-B—and eight dense retrievers: BGE-M3, CONTRIEVER, QWEN3-EMBEDDING, NV-EMBED-V2, REASON-EMBED, GRITLM-7B, JINA-V3, and REASONIR-8B. The main evaluation metrics are nDCG@5 and nDCG@10, with appendices also reporting nDCG@20 and nDCG@50. The headline result is that ARGUS improves retrieval quality consistently across benchmarks and retrievers, with average absolute gains of $\mathrm{RPS}_k = 0.2$2 nDCG@5 and $\mathrm{RPS}_k = 0.2$3 nDCG@10, and with substantially larger gains in challenging subsets (Taghavi et al., 10 Feb 2026).

Averaged over nDCG@5 and nDCG@10, the reported gains for Document Expansion are $\mathrm{RPS}_k = 0.2$4 on BRIGHT, $\mathrm{RPS}_k = 0.2$5 on IMPLIRET, and $\mathrm{RPS}_k = 0.2$6 on RAR-B. For LLM Synthesis, the corresponding averages are $\mathrm{RPS}_k = 0.2$7, $\mathrm{RPS}_k = 0.2$8, and $\mathrm{RPS}_k = 0.2$9. The paper states that both remedies generally outperform the original indexing baseline, with Document Expansion usually the most stable and strongest overall intervention, while LLM Synthesis is more index-efficient but more variable. It also notes retriever-specific interactions: in some settings synthesis can outperform expansion, while in others—explicitly including certain retrievers such as JINA-V3 on IMPLIRET—synthesis can underperform expansion (Taghavi et al., 10 Feb 2026).

The relation between RPS and downstream retrieval is examined directly. An analysis comparing the maximum predicted RPS in retrieved versus unretrieved gold documents shows that, in several subsets, retrieved documents tend to contain higher-RPS entities than unretrieved ones. The source adds that this association is not perfect, which is expected because retrieval depends on more than entity geometry alone. The broader interpretation offered is that preemptively remedying blind spots is critical for robust and trustworthy RAG systems (Taghavi et al., 10 Feb 2026).

6. Terminological boundaries and adjacent formulations

A recurring source of confusion is that “RPS” is not a uniform term across the literature. In the retrieval setting just described, RPS explicitly denotes an entity-level discovery probability over proxy query contexts. By contrast, several related retrieval papers employ probability-like or expectation-based formulations without defining an explicit metric called Retrieval Probability Score (Chang et al., 29 Aug 2025).

Construct	Status relative to RPS	Source
Retrieval Probability Score	Explicit entity-level top-$x$0 discovery probability for blind-spot auditing	(Taghavi et al., 10 Feb 2026)
Retrievability / T-Retrievability	Related exposure proxy; the paper states there is no explicit metric called RPS	(Chang et al., 29 Aug 2025)
Probabilistic CDF cutoff in pEBR	Related probability-based retrieval criterion; not named RPS in the main text	(Zhang et al., 2024)
Ranked Probability Score	Different scoring rule for ordered football-match forecasts, not retrieval	(Wheatcroft, 2019)

The clearest adjacent construct within information retrieval is retrievability. “T-Retrievability: A Topic-Focused Approach to Measure Fair Document Exposure in Information Retrieval” defines retrievability as a collection-based statistic that measures a document’s expected rank of being retrieved and argues that standard collection-level retrievability can conflate exposure bias with topical relevance priors. Its proposed T-Retrievability computes retrievability locally within topical query groups and aggregates Gini coefficients across groups. The paper explicitly states that it does not define something called RPS, but it also states that retrievability functions as a document-level retrieval probability or exposure score proxy. In that sense, T-Retrievability is conceptually adjacent to RPS while addressing a different object—document exposure fairness rather than entity blind-spot risk (Chang et al., 29 Aug 2025).

A second adjacent line is probabilistic embedding retrieval. The pEBR framework models query-conditioned item distributions and derives a dynamic cosine threshold from a learned cumulative distribution function, using a default CDF value of $x$1 unless otherwise specified. The paper states that it does not introduce a single named formula called RPS in the main text, but the operational retrieval criterion is explicitly probability-based: items are kept when their similarity exceeds a query-specific threshold selected from the learned CDF. This is closely related to probabilistic retrieval decisions, yet distinct from the pre-index blind-spot auditing role of RPS (Zhang et al., 2024).

Finally, the acronym collision with the Ranked Probability Score is purely terminological. In the football-forecasting literature, RPS is a non-local scoring rule for ordered outcomes and is defined over cumulative forecast probabilities rather than retrieval events. That score belongs to probabilistic forecast evaluation, not information retrieval. The shared acronym therefore does not imply methodological continuity (Wheatcroft, 2019).