Closing the Calibration Gap in Semantic Caching

Abstract: Semantic caching cuts LLM inference costs by serving a cached response to semantically similar queries. Standard practice evaluates these systems using PR-AUC, a metric that only measures how well scores rank and ignores whether they are usable at a fixed threshold. We show this mismatch leads to systematically poor deployment choices, as models with the highest PR-AUC are often the worst in operation. We introduce Precision-Cache Hit Ratio (P-CHR) AUC, a cache-aware metric that measures precision across cache utilization levels, and Calibration Retention Rate (CRR), which captures how much offline ranking quality survives at deployment. We decompose the operational gap between offline and deployed quality into a recoverable calibration component and an irreducible structural component fixed by the dataset's positive rate. Our experiments show that the calibration gap is governed by the training objective rather than data scale, and post-hoc calibration only partially closes it. Ultimately, model selection for semantic caching is a calibration problem, not a ranking one, and measuring it is the first step to closing the gap.

• The paper introduces novel cache-aware metrics, such as P-CHR AUC and CRR, to directly measure operational precision in semantic caching.
• It decomposes the operational gap into structural and calibration components, showing that training objectives largely determine calibration success.
• Empirical results demonstrate that high offline PR-AUC scores can mask precision collapse, highlighting the need for calibration-aware evaluation.

Calibration-Aware Evaluation and Deployment in Semantic Caching

Overview

"Closing the Calibration Gap in Semantic Caching" (2606.19719) fundamentally repositions semantic cache system design as a calibration problem rather than a ranking problem. Semantic caching substitutes LLM inference with cached responses for semantically similar queries, leveraging embedding-based similarity for efficiency. The work demonstrates that standard evaluation via metrics such as PR-AUC (Precision--Recall Area Under Curve) systematically misaligns with operational requirements, as offline ranking does not translate into deployable precision at practical thresholds. The paper introduces new cache-aware metrics—most notably, Precision-Cache Hit Ratio AUC (P-CHR AUC) and Calibration Retention Rate (CRR)—decomposes the operational gap, and empirically establishes that calibration is governed by training objective rather than dataset scale. The experimental deployment of novel metrics and large-scale data/model releases enable precise cache-aware evaluation and model selection.

Semantic Caching Deployment Mechanics

Semantic caching operates by thresholding embedding-based similarity to decide whether to return a cached response or invoke an LLM. In contrast to exact-match key-value stores, cached queries are matched on semantic equivalence. Dense retrieval (bi-encoder) and cross-encoder reranking pipelines are standard, but system deployment is determined not solely by the rank ordering of scores but by their calibration relative to a threshold $\tau$. High offline PR-AUC does not guarantee high operational precision at a practical cache utilization level. Miscalibrated models, despite excellent ranking, collapse precision the moment a useful threshold is set.

Dataset Curation and Model Training

The LangCache SentencePairs collection is constructed to address limitations in linguistic diversity and scale seen in previous benchmarks. Three versions, scaling from 1M to 40M pairs, comprise paraphrase detection, adversarial examples, and synthetic LLM/paraphrase corpora. All models are evaluated on a 74k-pair test split with 45% positives.

LangCache-Embed-v3 is a bi-encoder retriever trained with an ArcFace additive angular margin loss to maximize in-batch contrastive separation. LangCache Rerankers are cross-encoder models trained using either Binary Cross-Entropy (BCE, for probabilistic calibration) or Multiple Negatives Ranking Loss (MNRL, for margin ranking). ColBERT-family models are considered as multi-vector rerankers.

Cache-Aware Metrics and Operational Gap Decomposition

Traditional metrics (PR-AUC, recall, nDCG) are threshold-independent and agnostic to the calibration of score magnitudes, offering no leverage over binary cache deployment. The introduction of P-CHR AUC enables measuring the tradeoff between cache utilization and precision. As $\tau$ is swept, P-CHR traces deployment-relevant precision across utilization levels, fully sensitive to calibration failures. CRR quantifies the share of offline ranking quality retained operationally.

The operational gap $$\Delta_{\text{op}} = \text{PR-AUC} - \text{P-CHR AUC}$$ separates offline promise from deployable reality. It is decomposed into:

• Structural gap: irreducible, determined by dataset positive rate ($p$), with theoretical ceiling $p(1 - \ln p)$ for P-CHR AUC.
• Calibration gap: recoverable, resulting from miscalibrated score placement. Post-hoc calibration (temperature/Platt scaling) only partially closes this gap.

Numerical Results and Model Selection Contradictions

Experimental results over nine retrievers and ten rerankers confirm that domain-specific retrievers hold marginal deployment advantages (LangCache-Embed-v3 P-CHR AUC = 0.437), but calibration gaps exist in all families, including general-purpose models.

Figure 1: Retriever score distributions for LangCache-Embed-v3, positives concentrated at 1.0 and negatives centered at 0.5, maximizing usable threshold range with best P-CHR AUC ($0.437$).

Contradictory to standard practice, cross-encoder rerankers with highest PR-AUC (e.g., BCE-trained LangCache-Rerankers, PR-AUC = 0.816/0.748) retain <25% of offline quality in deployment (CRR = 0.244/0.231), with severe calibration gaps (>0.4). MNRL training mitigates but does not eliminate the gap (CRR ≈ 0.41–0.43). ColBERT-family models are the only rerankers achieving the lowest calibration gaps and highest CRR (up to 0.781) despite the lowest PR-AUC. Scaling the dataset 38$\times$ provides negligible deployment benefit—the calibration gap is fixed by training objective.

Figure 2: Precision--Recall curves are uniformly healthy, ranking BCE rerankers near the top; the operational P-CHR curves reveal BCE precision collapse and ColBERT-family ascent, inverting offline rankings.

Score distribution analysis confirms that operational deployment quality is controlled by where mass sits relative to threshold, not aggregate separation metrics (e.g., ROC-AUC, KS statistic, distribution overlap). BCE models cluster negatives at decision boundaries, creating precision collapse at every threshold; ColBERT-family models, with softmax normalization, maintain precision-favorable thresholds despite high distributional overlap.

Practical and Theoretical Implications

This work establishes that the central deployment challenge in semantic caching is calibration. Model selection by PR-AUC, F1, or ROC-AUC is fundamentally misleading for threshold-dependent systems. Practitioners should adopt cache-aware metrics, prefer contrastive or multi-vector objectives, and apply post-hoc calibration judiciously. Empirically, reranking often offers no deployment benefit and incurs latency overhead. For strong domain retrievers, thresholding the retriever is frequently optimal.

The structural floor for operational gap, fixed by the positive rate, enforces normalization by CRR for cross-deployment comparisons. Calibration must be maintained continuously, as query distributions drift in production.

Theoretically, the results indicate that evaluation methodology in retrieval-centric LLM systems must be realigned: threshold-sensitive metrics are essential for cache and retrieval decisions. The separation between structural and calibration gaps could inform design for adaptive or per-prompt thresholding policies, possibly integrating uncertainty estimation or Bayesian calibration.

Speculation on Future Directions

Future work should address semantic cache calibration in multilingual, multi-turn, or large-candidate-pool settings—with applicability expected to broader retrieval tasks (IR, QA, multimodal search). Adaptive (per-query) thresholding, online calibration, and feedback-driven score adjustment will become prominent as semantic caches grow in complexity and scale. As LLMs support dynamic user-driven and agentic workloads, the calibration challenge will generalize to retrieval-augmented generation, verification-centric caching, and cross-modal retrieval.

A new line of research may focus on neural score calibration methods sensitive to precision--utilization tradeoffs, exploiting the decomposed operational gap structure. Techniques for unsupervised calibration maintenance and real-time monitoring in production environments remain open.

Conclusion

Semantic cache deployment is fundamentally constrained by score calibration, not rank order. PR-AUC, while standard, systematically misleads operational selection, penalizing accurate models and promoting miscalibrated ones. This work clarifies that only cache-aware metrics that integrate deployment precision and calibration retention align offline evaluation with practical utility. Model selection, loss function design, and production validation must adapt. Release of large-scale datasets, metrics, and models in this work establishes a foundation for calibration-aware semantic caching, and calls for a broad re-thinking of evaluation in retrieval- and threshold-centric AI systems.