Entities as Retrieval Signals: A Systematic Study of Coverage, Supervision, and Evaluation in Entity-Oriented Ranking

Abstract: Entity-oriented retrieval assumes that relevant documents exhibit query-relevant entities, yet evaluations report conflicting results. We show this inconsistency stems not from model failure, but from evaluation. On TREC Robust04, we evaluate six neural rerankers and 437 unsupervised configurations against BM25. Across 443 systems, none improves MAP by more than 0.05 under open-world evaluation over the full candidate set, despite strong gains under entity-restricted settings. The best configuration matches the official Robust04 best system and outperforms most neural rerankers, indicating that the architecture is not the limiting factor. Instead, the bottleneck is the entity channel: even under idealized selection, entity signals cover only 19.7\% of relevant documents, and no method achieves both high coverage and discrimination. We explain this via a distinction between Conceptual Entity Relevance (CER) -- semantic relatedness -- and Observable Entity Relevance (OER) -- corpus-grounded discriminativeness under a given linker. All supervision strategies operate at the CER level and ignore the linking environment, leading to signals that are semantically valid but not discriminative. Improving supervision therefore does not recover open-world performance: stronger signals reduce coverage without improving effectiveness. Conditional and open-world evaluation answer different questions: exploiting entity evidence versus improving retrieval under realistic linking, but are often conflated. Progress requires datasets with entity-level discriminativeness and evaluation that reports both coverage and effectiveness. Until then, conditional gains do not imply open-world effectiveness, and open-world failures do not invalidate entity-based models.

• The paper demonstrates that entity signals are hindered by a coverage–discrimination tradeoff, limiting gains over BM25.
• It evaluates six entity-aware reranking models and 437 configurations using both conditional and open-world evaluation methods.
• The study reveals that improving CER supervision does not ensure observable (OER) benefits, underscoring the need for refined diagnostic benchmarks.

Entities as Retrieval Signals: Diagnosing Coverage, Supervision, and Evaluation in Entity-Oriented Ranking

Problem Statement and Motivation

Entity-oriented retrieval models posit that the presence of query-relevant entities in documents provides discriminative signals for ad hoc IR beyond surface-level term matching. However, empirical outcomes in the literature vary, with reports of strong entity-based improvements in some settings and marginal or negligible gains in others. This paper systematically interrogates the causes of these inconsistencies, focusing on the evaluation confound between the coverage of entity signals and their discriminative utility, across a broad experimental landscape including six neural architectures, 437 unsupervised entity configurations, and multiple supervision regimes on TREC Robust04.

The critique is data- and evaluation-centric: rather than attributing failures to architectural limitations or model tuning, the analysis traces the bottleneck to two sources—the entity channel’s inherent coverage ceiling under practical entity linking, and the lack of direct entity-level discriminative supervision.

Evaluation Regimes and the Nature of the Inconsistency

The paper distinguishes two evaluation settings:

1. Conditional evaluation: Performance is computed only on the subset of documents containing at least one selected entity for the query. This highlights model capacity to exploit entity signals, assuming those signals are present (i.e., idealized ascertainment).
2. Open-world evaluation: The system ranks over the full candidate set, reflecting realistic retrieval constraints, where many relevant documents lack any entity-link overlap due to limitations in entity linking and selection.

Conditional settings yield superficially impressive MAP (up to 0.60–0.70 for QDER/DREQ), suggesting large gains from entity signals. However, when assessed in open-world form, all architectures—including recent neural and unsupervised pipelines—collapse to BM25-level MAP (maximum observed ≈0.34; e.g., QDER+LGBM selector), exhibiting a nearly complete loss of the conditional gain.

Dissecting the Bottleneck: Coverage–Discrimination Dilemma

The central finding is that the entity channel fundamentally fails to achieve both high coverage (fraction of relevant documents intersected by selected entities) and high discrimination (selectivity of entities for relevant vs. non-relevant documents). The coverage ceiling imposed by the entity linker/environment places an upper bound on recall attainable through entity filtering, regardless of model. For example, even with an "oracle" binary-derived supervision, only 19.7% of relevant documents in Robust04’s BM25 candidate pool are covered by the top-20 entities; attempts to push coverage higher necessarily dilute discriminative power, as high-frequency entities function as "bait" (generic, non-discriminative signals).

Figure 1: Coverage–discrimination space for entity selection at $k=20$; oracle achieves near-perfect discrimination but covers only 19.7% of relevant documents, while practical runs cluster at high coverage but poor discrimination.

Examination of 194 selection configurations demonstrates a strong, linear coupling ($r=0.954$); any attempt to increase relevant-coverage brings an almost proportional increase in non-relevant-coverage, blocking the ability to select entities that simultaneously achieve high recall and discrimination.

Figure 2: RelCov@20 versus NonRelCov@20 for all entity selection configurations, showing structural coupling—no run populates the high-coverage/low-contamination quadrant.

Supervision: Conceptual vs. Observable Entity Relevance

The paper advances a critical distinction between Conceptual Entity Relevance (CER)—semantic adjacency as judged from query–entity pairing—and Observable Entity Relevance (OER)—the empirical discriminative power of an entity as actually produced by linking in the particular collection. All existing supervision strategies, including LLM-generated or binary supervision, operate at the CER level and fail to guide models toward entities with strong OER.

The empirical alignment between CER-based supervision and OER signals is weak (Spearman correlation for LLM CER: +0.13 with OER log-odds; for "oracle" supervision: –0.21). Even when using LLM-derived CER, the method selects up to 29.2% bait entities. Post-hoc IDF adjustment or more selective supervision can reduce bait, but only by heavily sacrificing coverage.

Figure 3: OER log-odds distribution for the common entity partition discarded by binary derivation; 83.6% of discarded entities are in fact discriminative under OER, illustrating that exclusivity criteria discard useful signals.

Figure 4: Distribution of OER log-odds among top-20 selected entities; the LGBM selector (best compromise) shifts the signal rightward but still retains a substantial bait component, confirming the coverage-discrimination constraint.

The supervision-coverage dilemma thus arises because the only entities guaranteed to be both conceptually and observably relevant are rare, leading to structurally sparse coverage.

Empirical and Frontier Analyses

A sweep of 193 entity selection methods (6 supervised, 187 unsupervised) confirms:

• Not a single configuration escapes the coverage–discrimination frontier; all practical configurations form a Pareto curve bounded far from the ideal high-coverage/high-discrimination region.
• Explicit OER filtering, though improving the discrimination ratio, devastates recall to the point that MAP falls below BM25, demonstrating that the dilemma is inescapable without new annotation standards or linking environments.

  Figure 5: Coverage–discrimination space across all configurations; no method occupies the high coverage–high discrimination region, with arrows showing the trade-off from oracle through LLM CER to LGBM.

  

  Figure 6: Coverage–discrimination Pareto frontier for all entity selection runs; the shaded region (high RelCov, low NonRelCov) remains empty across all method families and thresholds.

Evaluation Implications

• Conditional and open-world evaluations interrogate different scientific questions. Conditional regimes probe model capacity under ideal entity-channel conditions, while open-world regimes are confounded by coverage and cannot reflect either model or supervision quality meaningfully.
• Strong conditional gains cannot be interpreted as evidence of open-world effectiveness. Likewise, open-world collapse does not imply a failed architecture but rather a structural failure of the collection–linker–supervision pipeline.
• No conceivable improvement to current state-of-the-art entity supervision (e.g., improved LLMs, crowdsourced CER annotation) will overcome the absence of OER-level annotation and the coverage ceiling inherent to current environments.

Guidelines for Future Progress

The study argues that meaningful progress in entity-oriented retrieval requires:

• Collections with entity-level discriminativeness annotations: Only direct OER training can enable selection of high-coverage, high-discrimination entity sets necessary for robust, open-world improvements.
• Standardized evaluation reporting: Future work should present conditional and open-world results, as well as entity-level coverage, discrimination, and CER/OER diagnostics.
• Diagnostic frameworks: Explicitly track the coverage and discrimination properties of the entity selection, independently of downstream model structure.
• Potential future directions include designing new benchmarks that jointly annotate document and entity-level discriminativeness, or developing collection-grounded, OER-aligned LLM annotation pipelines.

Conclusion

Entity-oriented retrieval on conventional ad hoc collections is evaluation-bound: no matter the model or supervision, the combination of incomplete entity linking and the lack of OER-level supervision produces a maximum open-world MAP increase of 0.05 over BM25, with gains under conditional evaluation reflecting coverage-restricted idealization. The central finding is not a refutation of the entity-signal hypothesis, but a diagnosis: evaluation infrastructure, not model capacity, is the current limiting step. Progress in entity-based retrieval now requires a realignment of both annotation standards and evaluation paradigms to decouple model diagnosis from the persistent coverage–discrimination trap.

---