Pool-Restricted Oracle Ceiling (PROC)

• Pool-Restricted Oracle Ceiling (PROC) is a diagnostic metric that disentangles retrieval headroom from ordering efficiency in RAG pipelines, clearly defining evidence capture performance.
• It calculates the oracle gain within a restricted candidate pool by optimally reordering retrieved passages, isolating system bottlenecks from suboptimal reranking.
• Empirical analyses demonstrate how varying pool sizes and reranking strategies impact PROC, offering actionable insights for production optimization and cost–latency trade-offs.

The Pool-Restricted Oracle Ceiling (PROC) is an operationally focused diagnostic metric designed to disentangle retrieval and ordering headroom in Retrieval-Augmented Generation (RAG) pipelines, specifically under a fixed prompt budget $K$. PROC quantifies, for any retrieval configuration, the fraction of ideal oracle gain attainable solely by optimally reordering the set of passages produced by the retriever. This isolates the impact of the retrieval stage from that of reranking or downstream ordering, enabling precise attribution of system bottlenecks and guiding principled, auditable optimization in large-scale production RAG deployments (Dallaire, 12 Nov 2025).

1. Formalism and Mathematical Definition

Let $q$ denote a query with a full graded pool of $N$ passages, where each passage $d$ is assigned a grade $g(d)\in\{1,\ldots,5\}$ and an associated rarity-aware weight $w_{g(d)}$. The full-pool oracle gain at cutoff $K$ is defined as: $$G_{\mathrm{ideal}}^{\mathrm{full}}(q,K) = \sum_{i=1}^K w_{g_i^{\star}},$$ where $$g_1^\star \geq g_2^\star \geq \cdots \geq g_K^\star$$ are the top-$K$ weights from the entire pool. For any pool $q$0 (the set of candidates surfaced by a specific retrieval configuration, of size $q$1), the oracle gain restricted to $q$2 is

$q$3

where $q$4 are the top-$q$5 weights in $q$6. The Pool-Restricted Oracle Ceiling is then

$q$7

If the retrieval pool captures all top evidence present in the full pool, $q$8; otherwise, any deficit reflects irrecoverable retrieval miss.

The observed metric, RA-nWG@$q$9, is defined as

$N$0

with $N$1 the sum over the actually surfaced top-$N$2 passages after reranking. The percentage of PROC, denoted $N$3, measures the fraction of the restricted ceiling actually realized: $N$4

2. Conceptual Distinction: Retrieval vs. Ordering Headroom

PROC explicitly disentangles retrieval headroom ("Is the needed evidence present?") from ordering headroom ("Can the evidence be surfaced to the LLM?"). If $N$5 is low, the retrieval stage has not delivered the decisive evidence; subsequent reranking or post-retrieval operations cannot recover this loss. Conversely, high PROC but low $N$6 identifies suboptimal reranking or ordering as the constraining factor. Such explicit decoupling is unavailable with traditional rank-centric IR metrics (e.g., nDCG, MAP, MRR), which lack pool restriction and fail to account for prompt-injected set consumption typical in RAG scenarios.

The relationship is summarized:

Scenario	PROC	%PROC	Diagnostic implication
Low	—	—	Retrieval pool misses key evidence—improve retrieval coverage
High but low %PROC	High	Low	Reranker underutilizes available evidence—improve ranking, deduplication, chunking
High	High	High	Near-optimal—either further gains saturate or are cost/latency dominated

3. Practical Computation

Empirical computation of PROC follows these steps:

1. Pool Construction: Run the retriever (dense, hybrid, or hybrid+ANN) to generate candidate pool $N$7 of size $N$8.
2. Grading: For all $N$9 candidate documents, obtain grades $d$0 and corresponding rarity-aware weights $d$1 (e.g., via rag-gs pipeline).
3. Oracle Calculation: Compute $d$2, the reference denominator for all normalized metrics.
4. Pool-Restricted Oracle Ceiling: Within $d$3, select and sum the top $d$4 weights to yield $d$5, then compute $d$6.
5. Observed Performance: Sum the weights of the actual surfaced top-$d$7 to find $d$8 and thus $d$9.
6. %PROC Calculation: Divide observed by ceiling to obtain $g(d)\in\{1,\ldots,5\}$0.

Key parameters include $g(d)\in\{1,\ldots,5\}$1 (typ. 50–200) and cutoff $g(d)\in\{1,\ldots,5\}$2 (e.g., injection points $g(d)\in\{1,\ldots,5\}$3).

4. Empirical Results and Diagnostic Illustrations

On a scientific-papers corpus, PROC exposes retrieval and ordering efficiencies across configurations:

• Hybrid+Rerank (RRF-100 → Cross-Encoder Rerank-2.5 → Top-50):
  • At $g(d)\in\{1,\ldots,5\}$4: $g(d)\in\{1,\ldots,5\}$5, actual $g(d)\in\{1,\ldots,5\}$6, $g(d)\in\{1,\ldots,5\}$7. Retrieval headroom closes; ordering captures ≈85%.
  • At $g(d)\in\{1,\ldots,5\}$8: $g(d)\in\{1,\ldots,5\}$9, actual $w_{g(d)}$0, $w_{g(d)}$1.
• Dense-only + Rerank (voyage-3.5 (1024d) on dense pool $w_{g(d)}$2):
  • At $w_{g(d)}$3: $w_{g(d)}$4, RA-nWG = 0.805, $w_{g(d)}$5. Dense pool misses ≈9.4% retrieval headroom at cutoff 10.
  • At $w_{g(d)}$6: $w_{g(d)}$7, RA-nWG = 0.819, $w_{g(d)}$8.
• Scaling $w_{g(d)}$9 (Appendix A.9): For voyage-3.5 1024d, at $K$0,
  • $K$1 increases from ≈0.837 (pool 50) to ≈0.936 (pool 200), but gains flatten above 100, indicating diminishing returns for larger pools—a favorable trade-off analysis for large-scale RAG.

5. Deployment Guidelines and Operational Implications

For production RAG, PROC provides actionable guidance:

• Metric Reporting: For each $K$2 and configuration, report (i) RA-nWG@$K$3, (ii) N-Recall$K$4@$K$5, (iii) $K$6, and (iv) $K$7. This clarifies whether observed improvements are rooted in retrieval expansion or in ordering enhancements.
• Diagnostic Routing: Low PROC mandates focus on retriever enhancement (hybridization, ANN recall tuning, query rewriting), while high PROC and low $K$8 direct attention to reranker upgrades (stronger models, deduplication, metadata cleaning, chunk length adjustment).
• Dynamic Parameter Routing: Route simple queries via low pool size ($K$9); use diagnostic signals (e.g., cosine margin, entropy, ablations) to trigger higher $$G_{\mathrm{ideal}}^{\mathrm{full}}(q,K) = \sum_{i=1}^K w_{g_i^{\star}},$$0 (e.g., 100), balancing efficiency and recall.
• Latency and Budget Control: Further increases in $$G_{\mathrm{ideal}}^{\mathrm{full}}(q,K) = \sum_{i=1}^K w_{g_i^{\star}},$$1 and $$G_{\mathrm{ideal}}^{\mathrm{full}}(q,K) = \sum_{i=1}^K w_{g_i^{\star}},$$2 are often dominated by cost and latency when $$G_{\mathrm{ideal}}^{\mathrm{full}}(q,K) = \sum_{i=1}^K w_{g_i^{\star}},$$3 is already high ($$G_{\mathrm{ideal}}^{\mathrm{full}}(q,K) = \sum_{i=1}^K w_{g_i^{\star}},$$4), signifying minimal benefit.
• ANN and Quantization Effects: Default to HNSW-F32 to preserve PROC ceiling. Int8 quantization improves memory but incurs 8–18% PROC ceiling loss; only use Int8 under hard memory constraints and always re-assess PROC to confirm retention of retrieval headroom.

6. Relation to Existing Metrics and Broader Impact

PROC addresses inadequacies of classical IR metrics—position discounts and rank-list bias are ill-suited to RAG, where the LLM consumes a set of passages at cutoff $$G_{\mathrm{ideal}}^{\mathrm{full}}(q,K) = \sum_{i=1}^K w_{g_i^{\star}},$$5. PROC enables direct auditability and reproducibility in benchmarking, allowing practitioners to make budget- and SLA-aware decisions anchored in a transparent decomposition of pipeline weaknesses. When integrated with golden-set pipelines and rarity-aware evaluation (e.g., RA-nWG), PROC forms part of a coherent diagnostic suite supporting optimization, interpretability, and guardrail assessment for RAG deployments in complex, cost-sensitive environments.

7. Limitations and Interpretation

PROC is bounded above by the quality of retriever-generated pools relative to the full graded set and depends on accurate document grading and rarity-weight assignments. The metric inherently reflects the granularity and coverage of the candidate pool and does not alone address content diversity or redundancy; these must be monitored via supplemental diagnostics. Its utility is maximized when incorporated alongside set-based and coverage-driven metrics within an end-to-end RAG evaluation and tuning process, as demonstrated in experimental benchmarks and operational practices on scientific corpora (Dallaire, 12 Nov 2025).