Calibration Retention Rate in Semantic Caching

• Calibration Retention Rate (CRR) is a scalar metric that quantifies the fraction of offline ranking performance preserved when deploying semantic caching with a global threshold.
• It is computed as the ratio of P-CHR AUC to PR-AUC, linking deployment precision with offline ranking quality for informed cache decision-making.
• CRR aids in model selection by highlighting recoverable calibration gaps, ensuring that operational performance reflects the underlying ranking accuracy.

Searching arXiv for papers defining or discussing Calibration Retention Rate and closely related calibration-retention notions. Calibration Retention Rate (CRR) is a scalar metric introduced for semantic caching to quantify how much offline ranking quality survives when a model is deployed with a global threshold to drive cache decisions. In its formal definition, $$\text{CRR} = \frac{\text{P-CHR AUC}}{\text{PR-AUC}}$$, where the numerator measures deployment quality through the Precision–Cache Hit Ratio tradeoff and the denominator measures offline ranking quality through precision–recall analysis (Baral et al., 18 Jun 2026). In adjacent literatures, closely related retention notions appear without the same name: trimming-based conformal prediction analyzes what fraction of calibration points survive trimming and how that retained law governs coverage (Wang, 7 May 2026); continual fine-tuning of LLMs studies how much conformal coverage is preserved over task sequences (Shihab et al., 27 Apr 2026); and language-model fine-tuning studies before/after preservation of probabilistic calibration capabilities without defining CRR explicitly (Baldelli et al., 12 May 2026). The term is therefore most precise when used in the semantic-caching sense, but it also functions as a broader label for calibration preservation under operational constraints.

1. Formal definition and semantic-caching setting

In semantic caching, each query $q$ is matched against candidate cached responses, and a model assigns a score $s(q,c)$ to each query–candidate pair. Deployment uses the top-1 candidate $\hat c(q)$ with score $\hat s(q)$ and fires the cache when $\hat s(q)\ge \tau$ for some global threshold $\tau$ (Baral et al., 18 Jun 2026). The key distinction is between offline ranking evaluation and deployed thresholded behavior.

The paper "Closing the Calibration Gap in Semantic Caching" defines CRR as

$$\text{CRR} = \frac{\text{P-CHR AUC}}{\text{PR-AUC}}, \qquad \text{PR-AUC} \in (0,1]$$

and interprets it as the fraction of offline ranking performance that survives in deployment (Baral et al., 18 Jun 2026). High CRR means that a model’s scores are not only good at ranking positives above negatives, but are placed in a way that allows a threshold to trade precision against cache utilization efficiently. Low CRR means that much of the ranking quality is lost operationally because the scores are miscalibrated, for example through compression or poor placement near the decision boundary.

The semantic-caching framework uses the following deployment quantities:

$$\text{CHR}(\tau) = \frac{|\{q : \hat{s}(q)\ge \tau\}|}{N},$$

$$\text{VCHR}(\tau) = \frac{1}{N}\Big|\{q : \hat{s}(q)\ge \tau,\ \hat{c}(q)=c^*,\ y_q=1\}\Big|,$$

$q$0

Here CHR is the Cache Hit Ratio, VCHR is the Valid Cache Hit Ratio, and deployment precision is the fraction of cache fires that are correct. CRR summarizes how well these deployed decisions preserve the quality implied by offline ranking scores.

A central claim of the paper is that model selection for semantic caching is fundamentally a calibration problem, not a ranking problem (Baral et al., 18 Jun 2026). This is the context in which CRR was introduced.

2. Relationship to PR-AUC, P-CHR AUC, and the operational gap

CRR is built from two areas under curves with different operational meanings. PR-AUC measures how well $q$1 ranks positives above negatives and is invariant to monotone transformations of scores. P-CHR AUC instead measures precision across cache utilization levels and depends on the numerical placement and spread of deployed scores $q$2 (Baral et al., 18 Jun 2026).

The Precision–CHR curve plots

$q$3

as the threshold varies, and the associated area is

$q$4

The paper defines the operational gap as

$q$5

This gap measures the loss between how good a model looks in offline ranking terms and how good it is when used with a threshold in the actual cache.

The paper further decomposes this operational gap into an irreducible structural component and a recoverable calibration component. With dataset positive rate

$q$6

the structural gap is

$q$7

and the calibration gap is

$q$8

The interpretation is that $q$9 is fixed by the label distribution, whereas $s(q,c)$0 is the extra loss due to score miscalibration and is therefore recoverable in principle (Baral et al., 18 Jun 2026).

For a perfect ranker, the theoretical maximum deployment quality is not $s(q,c)$1 but

$s(q,c)$2

Accordingly, for a model with $s(q,c)$3, the maximum possible CRR is also $s(q,c)$4. On the paper’s test set with $s(q,c)$5, this ceiling is approximately $s(q,c)$6 (Baral et al., 18 Jun 2026). This establishes that CRR is structurally bounded away from $s(q,c)$7 in semantic caching, even under ideal ranking.

3. Why CRR differs from ordinary ranking quality

The motivation for CRR is that PR-AUC can be operationally misleading. Because PR-AUC is threshold-independent and invariant to monotone transformations, it can rate a model highly even when no global threshold yields a good precision–utilization tradeoff in deployment (Baral et al., 18 Jun 2026). CRR corrects for this mismatch by asking how much of the ranking quality can actually be realized after top-1 selection and thresholding.

The paper presents several concrete examples. BCE-trained rerankers achieve strong PR-AUC, around $s(q,c)$8–$s(q,c)$9, but extremely low P-CHR AUC, $\hat c(q)$0–$\hat c(q)$1, which yields CRR around $\hat c(q)$2–$\hat c(q)$3. By contrast, ColBERTv2.0 has PR-AUC near $\hat c(q)$4, close to the positive rate $\hat c(q)$5, but P-CHR AUC of $\hat c(q)$6 and CRR of $\hat c(q)$7, close to the theoretical ceiling $\hat c(q)$8 (Baral et al., 18 Jun 2026). In deployment terms, the latter scores are far more usable.

This inversion of ranking shows why CRR is not merely a rescaled PR-AUC. PR-AUC measures ranking separability. P-CHR AUC measures precision retention across cache utilization. CRR measures the efficiency with which a model converts ranking quality into deployed cache quality (Baral et al., 18 Jun 2026).

A related implication appears in other calibration literatures. In continual LLM fine-tuning, conformal coverage can deteriorate earlier and more sharply than top-1 accuracy, with coverage loss exceeding accuracy loss by roughly $\hat c(q)$9 on average across seeds (Shihab et al., 27 Apr 2026). This suggests that retention-style calibration metrics are valuable precisely because operational reliability and headline accuracy can diverge substantially. In conformal trimming, similarly, the relevant object is not contamination alone but the retained law induced by trimming, which is governed by which calibration points survive the trimming rule (Wang, 7 May 2026).

4. Computation and interpretation

To compute CRR on a labeled test set, the semantic-caching paper requires three ingredients: offline scores $\hat s(q)$0 for PR-AUC, deployed top-1 scores $\hat s(q)$1 and predictions $\hat s(q)$2 for P-CHR AUC, and labels $\hat s(q)$3 (Baral et al., 18 Jun 2026). The computation proceeds by threshold sweeps over the score range.

First, PR-AUC is obtained from thresholding $\hat s(q)$4 and integrating precision as a function of recall:

$\hat s(q)$5

Second, P-CHR AUC is obtained from the deployed pipeline by thresholding $\hat s(q)$6, computing CHR and deployment precision, and integrating precision as a function of CHR:

$\hat s(q)$7

Finally,

$\hat s(q)$8

The same evaluation can be accompanied by

$\hat s(q)$9

$\hat s(q)\ge \tau$0

$\hat s(q)\ge \tau$1

In this diagnostic view, CRR is the relative retention summary, while the gap decomposition provides an absolute explanation of why retention is limited (Baral et al., 18 Jun 2026).

This suggests a useful interpretive rule. If $\hat s(q)\ge \tau$2, the model is already as operationally efficient as possible for its ranking quality and the dataset’s positive rate; further gains must come from better ranking or a different data regime. If $\hat s(q)\ge \tau$3, then low CRR reflects recoverable calibration inefficiency rather than purely structural limitations.

5. Empirical behavior and practical model selection

The semantic-caching experiments report that across nine retrievers, PR-AUC lies in $\hat s(q)\ge \tau$4, P-CHR AUC in $\hat s(q)\ge \tau$5, CRR in $\hat s(q)\ge \tau$6, and $\hat s(q)\ge \tau$7 in $\hat s(q)\ge \tau$8 (Baral et al., 18 Jun 2026). This indicates that all tested retrievers have positive calibration gaps, even when their raw ranking quality differs substantially.

Across rerankers, the contrast is sharper. BCE rerankers have the largest calibration gaps and the lowest CRR; MNRL rerankers improve P-CHR AUC and roughly double CRR relative to BCE; general cross-encoders occupy an intermediate position; and ColBERT-family models achieve the highest CRR, with $\hat s(q)\ge \tau$9 for all ColBERT-family models reported (Baral et al., 18 Jun 2026). In that sense, ColBERT scores are nearly optimally calibrated for threshold-based cache operation, even though their PR-AUC is relatively low.

The experiments also show that the calibration gap is governed by the training objective rather than data scale. Scaling reranker training data from roughly $\tau$0M to roughly $\tau$1M pairs does not materially improve CRR (Baral et al., 18 Jun 2026). Post-hoc temperature scaling modestly improves BCE models’ P-CHR AUC and therefore their CRR, but it does not close the gap; Platt scaling can even degrade deployment quality for some models. The paper therefore recommends P-CHR AUC and CRR, rather than PR-AUC alone, as primary model-selection metrics for semantic caching.

A broader implication is that retention metrics are especially informative when deployment imposes a hard operating threshold. In language-model calibration fine-tuning, the before/after comparisons of normalized Wasserstein-$\tau$2, trie-target KL, valid rate, TinyBenchmarks gp-IRT, PALOMA perplexity, and stochastic-generation metrics provide the ingredients for CRR-like ratios, even though the paper does not define a CRR formula explicitly (Baldelli et al., 12 May 2026). This suggests that CRR-style thinking generalizes whenever one wants to quantify how much calibration or capability survives an intervention.

6. Broader uses, adjacent notions, and ambiguity of the term

Although CRR is formally defined in semantic caching, nearby arXiv papers use closely related retention concepts under different mathematical objects.

In trimming under contaminated conformal calibration, the key retained quantity is the probability that a calibration point survives trimming. The paper distinguishes clean retention probability

$\tau$3

and dirty retention probability

$\tau$4

with overall expected retention

$\tau$5

These are described as the natural “Calibration Retention Rates” for clean vs dirty points, and they determine the retained mixture coefficient

$\tau$6

The paper’s central message is that trimming helps when the anomaly score separates retention probabilities while remaining score-neutral on the clean population; otherwise it cannot substantially reduce contamination through the retained mixture coefficient (Wang, 7 May 2026). In this literature, CRR is not a formal named metric, but the retention probability of calibration points is the mathematically relevant analogue.

In continual LLM fine-tuning, the paper on continual calibration does not define CRR explicitly, but it provides the components for a calibration-retention measure based on conformal coverage over task sequences. Coverage loss can exceed accuracy loss by a factor of roughly $\tau$7, and calibration replay with task-specific held-out buffers typically restores coverage to within two points of nominal at buffer size $\tau$8 (Shihab et al., 27 Apr 2026). This suggests a task-indexed retention notion based on current coverage relative to baseline coverage, especially when stale thresholds no longer satisfy exchangeability after model updates.

In probabilistic calibration for LLMs, the paper explicitly states that probabilistic calibration is a trainable capability, measured through trie-target KL, normalized Wasserstein-$\tau$9, and valid rate. It does not define CRR, but it notes that before/after comparisons are precisely the inputs one would use to define a CRR-like quantity, including structured-sampling fidelity, downstream capability retention, and language-model fit retention (Baldelli et al., 12 May 2026).

The term is also ambiguous outside calibration-retention usage. In "The Curvature Rate $$\text{CRR} = \frac{\text{P-CHR AUC}}{\text{PR-AUC}}, \qquad \text{PR-AUC} \in (0,1]$$0: A Scalar Measure of Input-Space Sharpness in Neural Networks," CRR means Curvature Rate Regularization, not Calibration Retention Rate (Poschl, 3 Nov 2025). That paper is relevant to calibration because it reports improved confidence calibration under derivative-based regularization, but its acronym refers to a different concept entirely.

7. Limitations and correct usage

CRR in the semantic-caching sense depends on labeled test data, because both PR-AUC and P-CHR AUC require ground-truth labels for representative queries (Baral et al., 18 Jun 2026). Continuous monitoring can therefore be difficult in production environments where labels are sparse. CRR is also evaluated on a particular test distribution: the structural ceiling depends on the positive rate $$\text{CRR} = \frac{\text{P-CHR AUC}}{\text{PR-AUC}}, \qquad \text{PR-AUC} \in (0,1]$$1, and changes in query distribution or duplicate rate alter the achievable P-CHR AUC.

The semantic-caching results are reported for English sentence-pair paraphrase detection with fixed top-$$\text{CRR} = \frac{\text{P-CHR AUC}}{\text{PR-AUC}}, \qquad \text{PR-AUC} \in (0,1]$$2 retrieval pools, so calibration behavior may differ in multilingual settings or at other retrieval depths (Baral et al., 18 Jun 2026). The paper also notes that CRR behavior is less straightforward when PR-AUC is very low, near the positive rate.

In broader usage, retention-style calibration metrics inherit the limitations of their base calibration notion. The regression-calibration benchmark emphasizes that many calibration metrics disagree, sometimes producing contradictory conclusions about the same recalibration result, and identifies ENCE and CWC as the most dependable metrics in its tests (Wibbeke et al., 25 Aug 2025). This suggests that any generalized CRR outside semantic caching should be defined explicitly relative to a chosen base metric, rather than treated as a universal standalone quantity.

Taken together, these results establish a precise encyclopedic meaning for Calibration Retention Rate in semantic caching—$$\text{CRR} = \frac{\text{P-CHR AUC}}{\text{PR-AUC}}, \qquad \text{PR-AUC} \in (0,1]$$3—while also showing that the broader idea of calibration retention recurs across conformal prediction, continual fine-tuning, and probabilistic calibration of LLMs. The common principle is that calibration must be evaluated in the operational regime in which decisions are actually made, not inferred from ranking quality, contamination level, or top-1 accuracy alone (Baral et al., 18 Jun 2026).