Coverage, Not Averages: Semantic Stratification for Trustworthy Retrieval Evaluation

Abstract: Retrieval quality is the primary bottleneck for accuracy and robustness in retrieval-augmented generation (RAG). Current evaluation relies on heuristically constructed query sets, which introduce a hidden intrinsic bias. We formalize retrieval evaluation as a statistical estimation problem, showing that metric reliability is fundamentally limited by the evaluation-set construction. We further introduce \emph{semantic stratification}, which grounds evaluation in corpus structure by organizing documents into an interpretable global space of entity-based clusters and systematically generating queries for missing strata. This yields (1) formal semantic coverage guarantees across retrieval regimes and (2) interpretable visibility into retrieval failure modes. Experiments across multiple benchmarks and retrieval methods validate our framework. The results expose systematic coverage gaps, identify structural signals that explain variance in retrieval performance, and show that stratified evaluation yields more stable and transparent assessments while supporting more trustworthy decision-making than aggregate metrics.

• The paper proposes a semantic stratification framework that uncovers hidden retrieval performance gaps, revealing that 40-50% of corpus clusters remain untested with standard queries.
• It employs LLM-based entity extraction and unsupervised clustering to partition the corpus into semantic regimes, enabling precise diagnostic evaluation of retrieval metrics.
• The stratified evaluation exposes systematic biases in aggregate metrics, providing actionable insights for robust system selection and reliable RAG deployment.

Coverage, Not Averages: A Semantic Stratification Framework for Trustworthy Retrieval Evaluation

Motivation and Problem Formulation

Reliable evaluation of retrieval systems, especially in the context of retrieval-augmented generation (RAG), is crucial for robust downstream application performance. Standard protocols, as commonly instantiated in BEIR and related benchmarks, rely on heuristic, ad hoc query sampling that fails to guarantee proportional semantic coverage of the corpus. The implicit assumption of query homogeneity is fundamentally invalid in real-world settings where the distribution of information-seeking queries is both heterogeneous and ill-defined.

The paper demonstrates—using the NFCorpus benchmark—that large regions of the semantic space with high document density can have little to no query coverage, which results in two pathologies: (1) aggregate metrics such as nDCG@10 systematically overestimate retrieval performance, and (2) high-prevalence but low-query clusters show weaker retrieval quality. The mismatch between corpus semantics and evaluation queries introduces irreducible estimand bias and spurious confidence in aggregate system comparison.

Figure 1: Semantic coverage in NFCorpus quantified via cluster-document versus cluster-query count, with color encoding mean nDCG@10; gray denotes clusters with no evaluation queries.

Statistical Formalization of Retrieval Evaluation

The paper recasts the retrieval evaluation protocol as a statistical estimation problem under structured population heterogeneity. The corpus is partitioned into $K$ semantic or structural regimes (strata), each inducing distinct conditional retrieval performance. The aggregate metric (e.g., mean nDCG@10) is then shown to be a mixture over these regime-level means weighted by unknown mixture proportions.

When standard evaluation queries are non-representative or omit entire regimes, the empirical mean is a biased estimator. The bias is a function of the discrepancy between the empirical regime weights and their corpus-level prevalence, and is non-vanishing in the presence of semantic coverage gaps. Moreover, the law of total variance implies that regime omission produces underestimated metric variance, fostering unwarranted metric stability. This theoretical analysis motivates regime-level summary statistics as the primary diagnostic, relegating aggregate means to a secondary, often misleading role.

Semantic Stratification Methodology

The authors introduce a corpus-driven, method-agnostic protocol for stratified retrieval evaluation composed of three steps:

1. Semantic Structure Construction: Documents are annotated with atomic, normalized semantic entities using LLM-based extraction. These entities populate a semantic graph, with edges denoting cosine affinity in embedding space.
2. Unsupervised Entity Clustering: The graph undergoes Leiden community detection to identify coherent, human-interpretable semantic clusters, balancing granularity with interpretability.
3. Query Stratification: Queries are mapped to clusters according to their relevant entities. Additional axes of stratification are defined via two structural signals: relevance dispersion ($\Delta$) and query-document semantic alignment ($J$), casting each query into a $(\Delta,J)$ regime that modulates intrinsic retrieval difficulty.
4. Coverage-Aware Dataset Curation: New evaluation queries are iteratively synthesized to fill regime/cluster coverage gaps, using LLM-based controlled generation and filtering. The resultant evaluation set is coverage-guaranteed across both semantic and structural axes, ensuring all semantic regions and difficulty regimes are exercised.

   Figure 2: Schematic of corpus entity clustering, query stratification, and regime vectorization for coverage accounting.

Empirical Results and Diagnostics

Experiments are conducted on BEIR's NFCorpus, FiQA, and SciDocs, comparing the proposed stratified protocol with standard aggregate evaluation across dense, sparse (BM25), and hybrid retrieval models.

Stratum Homogeneity and Differential Performance

Semantic clusters display high internal metric homogeneity and expose up to 3× variation in mean nDCG@10 across regimes. Stratified metric standard deviations are consistently lower than dataset-level variance, confirming cluster coherence.

Figure 3: Distribution of per-query nDCG@10 within semantic clusters for various retrievers, revealing systematic performance variance.

Exposure of Systematic Coverage Gaps

Coverage analysis reveals that standard BEIR-like evaluation queries leave 40-50% of corpus clusters untested (including large, methodologically central regions), while stratified evaluation achieves >90% cluster-level coverage. Zero-query clusters are identified, often corresponding to critical information domains (e.g., clinical methods, dietary assessment), on which benchmark-oblivious retrieval failures are entirely undetectable by aggregate metrics.

Differential Regime Difficulty

The cross ($\Delta,J$) grid exposes strong performance stratification: high-dispersion/low-alignment queries are demonstrably harder, with nDCG@10 varying from 0.25 in the most challenging regime to 0.62 in the best. These axes explain up to ~23% of within-benchmark performance variance.

Fine-Grained System Diagnostics

Stratified evaluation enables precise failure mode diagnostics. For example, semantic-lexical disconnects (where queries and relevant documents use non-overlapping vocabulary due to annotation artifacts) and register mismatches (colloquial queries with only technical document matches) are isolated as corpus-intrinsic failures, not retriever deficiencies. Conversely, well-aligned clusters (e.g., "endocrine disruptors") demonstrate strong, robust retrieval for all methods.

Figure 4: Semantic cluster graph with nodes sized by document count and colored by average in-cluster nDCG@10, demonstrating variable retrieval quality across corpus topology.

Aggregation Protocols and Decision Implications

The choice of metric aggregation (query-volume weighted vs. macro-average over clusters) is formally shown to dictate system ranking, with win rates for competing retrievers shifting by >30% depending on the weighting protocol. Practitioners thus must articulate their priorities—overall throughput or worst-case coverage—explicitly, as stratified statistics expose the latent trade-offs masked by global averages.

Figure 5: Head-to-head semantic cluster performance comparison between OpenAI and Cohere retrievers, showing localized dominance.

Figure 6: Bootstrap win rates across aggregation protocols, demonstrating the sensitivity of system preference to stratification.

Implications and Future Directions

The practical implication is that aggregate retrieval metrics derived from under-stratified evaluation sets engender unreliable system selection and risk undetected deployment of models with pathological failure modes in high-impact regions (e.g., medical methods, domain-specific entities). Stratified, coverage-aware evaluation provides actionable, interpretable, and decision-stable diagnostics for retrieval selection and debugging, particularly in RAG pipelines where inference reliability is constrained by semantic retrieval bottlenecks.

Theoretically, the stratification protocol generalizes to arbitrary corpus structures and is compatible with dynamic query distributions, efficient test collection design, and synthetic data generation regimes. Extension to multilingual and multimodal retrieval evaluation is suggested as a promising direction. The framework also enables more principled configuration of downstream LLM calibration and RAG prompt optimization pipelines, as empirical coverage can be tied to failure probabilities on deployment-specific semantic strata.

Conclusion

The paper presents a robust, theoretically motivated framework for corpus-grounded, coverage-aware, stratified evaluation of retrieval and RAG systems. It rigorously demonstrates that aggregate metrics, as typically deployed, are unreliable proxies for system trustworthiness and that semantic stratification yields actionable system diagnostics and actionable, trustworthy comparison. Public release of stratified datasets is planned to enable ongoing methodological advances.

For full bibliographic details, see (2604.20763).