ScoreGate: Adaptive Chunk Selection for Retrieval-Augmented Generation via Dual-Score Statistical Fusion

Abstract: Fixed-cardinality retrieval injects a constant top-K chunks into the generator regardless of query complexity, causing over-retrieval for narrow queries and under-retrieval for compositional ones. We describe ScoreGate, a lightweight score-space decision mechanism that controls retrieval cardinality at inference time using two scores already produced by the standard pipeline: bi-encoder similarity s_i and cross-encoder reranker score r_i, with no additional model inference calls required. Its core insight is that cross-encoder affirmation can rescue semantically relevant chunks that bi-encoder retrieval ranks poorly due to vocabulary mismatch -- a failure mode unaddressed by fixed-K or single-score thresholding. On MS MARCO (200 dev queries), ScoreGate achieves MRR@10 = 0.401 with 35% fewer retained chunks than Standard Top-K. On an internal benchmark (n=300, Fleiss' kappa=0.87), ScoreGate observed zero false positives (95% CI [96.4%, 100%]) at 97.77-99.34% recall, with 34.8% fewer tokens per query and only 31ms added latency. Results on both MS MARCO and real-world production traffic suggest that adaptive retrieval cardinality can improve retrieval efficiency without degrading retrieval quality.

• The paper introduces ScoreGate, a dual-score statistical fusion method that adaptively selects retrieval chunks based on partitioned score thresholds.
• It leverages bi-encoder similarity and cross-encoder reranker scores to dynamically adjust chunk selection, reducing irrelevant context and hallucinations.
• Empirical results show enhanced precision, recall, and efficiency across benchmarks with a significant drop in token usage and minimal added latency.

ScoreGate: Adaptive Chunk Selection for Retrieval-Augmented Generation via Dual-Score Statistical Fusion

Motivation and Problem Statement

Retrieval-Augmented Generation (RAG) systems typically inject a fixed number of retrieved chunks (top-$K$) into a generator, regardless of query complexity. This fixed-cardinality approach leads to significant efficiency and quality challenges: simple (navigational) queries are over-served, increasing irrelevant context and risk of hallucination, while complex (compositional) queries are under-served, limiting coverage due to an insufficient context window. Prior efforts to filter or rerank retrieved passages—such as cross-encoder reranking and LLM-based filtering—either do not adapt cardinality or require added inference costs per query. Moreover, lexical or semantic mismatches (paraphrased or indirect references) routinely cause relevant chunks to be excluded when using single-score thresholding, as bi-encoder similarity and cross-encoder relevance often capture complementary signals.

ScoreGate addresses these limitations by introducing an adaptive, score-based mechanism that leverages both bi-encoder similarity and cross-encoder reranker scores without requiring extra model computation. Its decision rules utilize joint score statistics to dynamically adjust the number of chunks selected per query, partitioning candidates into four regions and applying asymmetrical fusion thresholds to resolve areas of score disagreement.

ScoreGate Pipeline and Score-Space Partitioning

Upon receiving a user query, the standard bi-encoder retrieves top-$N$ candidates based on vector similarity, and a cross-encoder reranker produces normalized relevance scores for these candidates. ScoreGate then partitions the $(s_i, r_i)$ score space into four buckets using empirically derived thresholds ($\tau_s$, $\tau_r$):

• B1 (both scores high): Keep chunk unconditionally.
• B2 (high similarity, low reranker): Retain only if a high fusion score $f_i \geq 0.255$.
• B3 (low similarity, high reranker—indicative of vocabulary mismatch): Retain if $f_i \geq 0.15$.
• B4 (both scores low): Discard unconditionally.

  Figure 1: The ScoreGate pipeline, including score derivation, partitioning, bucket-specific rules, and adaptive cardinality selection.

This four-region scheme is motivated by score distributions observed on annotated benchmarks, specifically highlighting the prevalence of semantically relevant (but lexically mismatched) B3 candidates that standard Top-K heuristics systematically miss.

Figure 2: Empirical score distributions by relevance category reveal that the B3 region (low $s_i$, high $r_i$) captures semantically relevant chunks; right panel confirms ScoreGate's adaptive $|C'|$ per query.

Figure 3: Visualization of the four-region $N$0 partitioning and operational retention rules.

Statistical Fusion and Threshold Determination

For B2 and B3 candidates—where evidence from the two scores conflicts—a weighted linear fusion is applied:

$N$1

Cross-encoder reranking dominates ($N$2 weight) due to its direct supervision and calibration to human relevance. Fusion thresholds are derived empirically: in B2 a high bar ($N$3) guards against false positives; in B3, a lower threshold ($N$4) enables recovery of genuinely relevant, vocabulary-divergent content affirmatively scored by the cross-encoder.

Figure 4: $N$5 sensitivity to $N$6 establishes reranker-dominant fusion as optimal; performance degrades gracefully away from $N$7.

Cardinality is further capped by a MAX-$N$8 at inference time to prevent context overflow. Within this cap, the highest-fusion-score chunks are retained when needed.

Experimental Evaluation and Results

Evaluation spans MS MARCO (public, large-scale), an internal Annotated Relevance Benchmark (ARB), and real-world deployment data. Key findings are:

• MS MARCO (200 dev queries): ScoreGate achieves MRR@10=0.401, outperforming Standard Top-K, with a $N$9 reduction in mean retained chunks per query. Zero-shot transfer of thresholds still yields comparable performance, and re-derivation adapts seamlessly to domain-specific score distributions.
• ARB ($(s_i, r_i)$0): On internal queries with balanced relevance labels, ScoreGate demonstrates observed precision within $(s_i, r_i)$1 CI [96.4\%, 100\%] and recall of $(s_i, r_i)$2 (Relevant) and $(s_i, r_i)$3 (Semantically Relevant), strictly dominating an LLM filter baseline by $(s_i, r_i)$4pp on semantically relevant recall at $(s_i, r_i)$5 lower token cost and only $(s_i, r_i)$6 added latency.
• Bucket Analysis: B3 region (low $(s_i, r_i)$7, high $(s_i, r_i)$8) accounts for $(s_i, r_i)$9 of candidates; $\tau_s$0 of these are retained by ScoreGate versus $\tau_s$1 by the LLM Filter, explaining most of the recall gap.
• False Positives/Negatives: Zero observed false positives under all evaluated configurations; three false negatives (all borderline B2/B3, attributed to deep semantic abstraction).
• Real-World Traffic ($\tau_s$2): ScoreGate outputs are correct in $\tau_s$3 of abstention/warranted-retrieval scenarios, confirming robustness under diverse query types.

  Figure 5: Recall by relevance category on the ARB: ScoreGate (blue) significantly outperforms LLM Filter (orange) in both Relevant and Semantically Relevant regions.

• Efficiency: Average per-query token usage drops from $\tau_s$4 to $\tau_s$5, and latency remains competitive. The overhead is minor ($\tau_s$6) compared to model inference calls required in LLM-based filtering ($\tau_s$7 s).
• Robustness: Threshold variation analysis shows that ScoreGate is insensitive to moderate parameter perturbations; recall varies by at most $\tau_s$8pp with zero false positives across all settings.

  Figure 6: $\tau_s$9 surface across $\tau_r$0, with peak at the selected threshold. Sensitivity of $\tau_r$1 and recall versus $\tau_r$2 is unimodal and stable.

Practical and Theoretical Implications

ScoreGate advances adaptive chunk selection by providing a deterministic, score-driven procedure that integrates seamlessly with existing RAG architectures. By exploiting score disagreements, it recovers relevant information that would be lost under fixed or single-threshold policies, especially for compositional or paraphrased queries—a notable failure mode for both classic retrieval and LLM-augmented filtering.

On the practical side, ScoreGate’s minimal computational overhead and lack of dependence on extra model calls enable large-scale, real-time deployment. The approach reduces hallucination incidence (from $\tau_r$3 with Top-K to $\tau_r$4 with ScoreGate) by filtering out irrelevant context before generation.

Theoretically, ScoreGate illustrates how discriminative signal fusion—even from non-independent sources—can be effectively operationalized under simple bucketed decision policies. The dual-score statistical fusion paradigm may extend to other IR and RAG settings, such as multi-hop retrieval, entity-oriented queries, or hybrid document–knowledge base pipelines.

Limitations and Future Directions

While ScoreGate robustly addresses vocabulary-mismatch and adaptive cardinality in two-stage RAG, a few constraints are acknowledged:

• Performance is bounded by bi-encoder recall; if the candidate pool fails to contain relevant chunks, subsequent fusion is ineffective.
• Domain-specific score distribution shifts require empirical threshold re-derivation.
• Rare, deeply abstract or highly indirect answers with low $\tau_r$5 and $\tau_r$6 remain challenging (typically lost at B2/B3 boundaries).
• Multi-hop and cross-document reasoning are not explicitly supported; extension to benchmark datasets such as HotpotQA and MuSiQue remains open.
• Application to multilingual or highly technical corpora may need enhanced calibration or even bucket design revision.

Future research may explore principled, cost-sensitive threshold optimization, explicit integration with multi-hop or retrieval triggering signals, and further ablations on even larger and more heterogeneous knowledge corpora.

Conclusion

ScoreGate provides an empirically validated and computationally efficient mechanism for adaptive retrieval in RAG pipelines using existing scoring signals. Its four-bucket, dual-threshold architecture demonstrably increases coverage of semantically relevant content, reduces irrelevant context and hallucinations, and maintains high precision without additional inference costs. The stability of its design across domains and score distribution regimes positions it as a robust engineering approach for dynamic context sizing in RAG, with potential extensibility to other retrieval under uncertainty scenarios.