Risk-Constrained Freshness-Aware Semantic Caching for Open-Web Retrieval-Augmented LLMs

Abstract: Semantic caching reduces the latency and cost of retrieval-augmented generation (RAG) by serving cached answers to semantically similar queries, but most existing methods do not model the time-varying freshness of open-web evidence. We present FreshCache, a three-tier semantic cache that treats cache reuse as a risk-constrained temporal inference problem: before approving a cache hit, FreshCache estimates the probability that the cached result is stale using a fitted exponential decay model enhanced by a learned MLP, and approves reuse only when that probability falls below a per-tier error budget across answers (epsilon = 0.10), URL lists (epsilon = 0.20), and page content (epsilon = 0.35). This allows the system to degrade gracefully as entries age rather than forcing a binary choice between a stale hit and a full pipeline execution. We introduce FreshCache-Bench, a benchmark of 8,072 base queries across five freshness classes with ground truth staleness labels drawn from real web snapshots at 1, 12, 24 hours, and 7 days after a baseline crawl, expanded to 31,201 queries via paraphrase generation. At the 24-hour evaluation window, FreshCache_MLP achieves 97% search API savings at 0.1% hash-based stale error, and an LLM-judge evaluation on 396 confirmed change pairs shows that only 34.3% of detected content changes actually affect answer correctness, placing true answer-affecting stale error at approximately 0.034%. The rule-based FreshCache achieves 98% search savings at 3.3% stale error under a temporal holdout calibration, outperforming SemanticTTL (14.9% stale, 72% saved), vCache (7.2% stale, 47% saved), and SCALM (5.2% stale, 96% saved). Ablations show the temporal risk gate accounts for an 11.6 point reduction in stale error over similarity-only reuse, and the learned MLP reduces stale error a further 3.2 points over the rule-based model.

• The paper introduces FreshCache, a three-tier caching system that integrates semantic similarity screening with temporal staleness risk modeling.
• It employs a risk-gated inference mechanism across L1, L2, and L3 cache tiers with configurable error budgets and learned MLP predictors.
• Experimental results show FreshCache_MLP achieves 97% API savings with as low as 0.034% answer-level stale error, outperforming strong baselines.

Risk-Constrained Freshness-Aware Semantic Caching for Open-Web Retrieval-Augmented LLMs

Motivation and Context

Retrieval-augmented generation (RAG) with LLMs—integrating up-to-date external web evidence to augment model outputs—faces substantial challenges in serving low-latency, cost-efficient, and trustworthy answers. Semantic caching traditionally reduces redundant pipeline invocations by reusing cached responses to semantically similar queries. However, pre-existing approaches typically neglect the temporal volatility of web content, risking “stale” evidence even when semantic similarity remains high. This trade-off is problematic: overly aggressive cache reuse leads to out-of-date information, while conservative cache policies forfeit efficiency gains. The paper “Risk-Constrained Freshness-Aware Semantic Caching for Open-Web Retrieval-Augmented LLMs” (2607.04281) addresses this deficit by reframing caching as a temporal inference problem, introducing FreshCache—a multi-tier risk-gated cache that explicitly estimates staleness risk at query time.

FreshCache System Architecture

FreshCache is a three-tier architecture, aligning cache tiers with the stages of the RAG pipeline:

• L1 (Answer Cache): Stores final answers and associated retrieval context.
• L2 (URL-List Cache): Stores ordered URL results from previous search API calls.
• L3 (Content Cache): Stores raw fetched web page content keyed by URL.

Each tier maintains an independent, configurable error budget for acceptable stale responses ($\varepsilon_{L1} = 0.10$, $\varepsilon_{L2} = 0.20$, $\varepsilon_{L3} = 0.35$), reflecting the varying severity of staleness across artifact types. FreshCache combines semantic similarity screening with a per-tier, conditional staleness risk model. If a query is semantically close to a cache entry, FreshCache estimates the probability that the cache’s evidence is obsolete—leveraging both rule-based and learned models—and only approves reuse at the highest tier that satisfies the risk constraint. If staleness risk is excessive, the system degrades gracefully to safer but less efficient lower cache tiers or a full pipeline execution.

Figure 1: FreshCache system architecture showing semantic lookup, risk-gated cache reuse, and fallback execution across the L1, L2, and L3 cache tiers.

Staleness Modeling and Risk-Gated Inference

FreshCache introduces a principled formulation of temporal staleness risk. Workloads are partitioned into five empirically-driven freshness classes—TIMELESS, SLOW, MEDIUM, FAST, and REAL_TIME—governing the prior for web page volatility. Rule-based models fit an exponential decay function to observed page change rates, capturing class- and tier-specific half-lives ($h_\mathrm{TIMELESS}=22\text{d}$, $h_\mathrm{SLOW}=16\text{d}$, $h_\mathrm{MEDIUM}=15\text{h}$, $h_\mathrm{FAST}=3\text{h}$, $h_\mathrm{REAL-TIME}=30\text{s}$). A cache entry is only reused if the estimated staleness probability—accounting for class, artifact tier, cache entry age, and domain-specific features—is below the threshold for that tier.

Further, FreshCache extends temporal modeling using a lightweight learned MLP over features including query- and domain-level volatility, answer type, entity indicators, similarity, and keyword enrichments. This learned risk model outperforms purely rule-based approaches by discriminating between stable and unstable entries within the same freshness class.

Workload and Benchmark Construction

To support rigorous evaluation, the authors construct FreshCache-Bench—a temporally-structured benchmark containing 8,072 base open-web queries partitioned across the five freshness classes, extended to 31,201 paraphrase variants. Each query’s URLs are periodically re-fetched at 1, 12, 24 hours, and 7 days after baseline, and ground-truth staleness is labeled via page snapshot hashing.

Experimental Evaluation

At the 24-hour evaluation window, FreshCache with the learned MLP risk model (FreshCache_MLP) demonstrates 97% search API savings at 0.1% stale error, establishing a new Pareto frontier in freshness-efficiency tradeoff for open-web RAG caches. The true answer-affecting stale error, validated via an LLM-judge across 396 content-changing cases, is as low as 0.034%. Rule-based FreshCache, using holdout-calibrated half-lives, attains 3.3% stale error at 98% saved. In contrast, strong baselines such as SemanticTTL, vCache, and SCALM yield notably higher staleness or lower efficiency—e.g., SemanticTTL: 14.9% stale, 72% saved; vCache: 7.2% stale, 47% saved; SCALM: 5.2% stale, 96% saved.

Figure 2: Search API savings versus stale error rate at $t = 24$h using BGE-M3. FreshCache_MLP achieves the best deployable freshness–efficiency trade-off.

The authors conduct extensive ablation, confirming that only the combination of tiered temporal risk gating and advanced semantic encoding can deliver both high efficiency and low stale error. Incisively, stronger embedding architectures degrade staleness in TTL-only caches by increasing false positives for similarity, whereas temporal gating neutralizes this effect and enables safer cache expansions.

Figure 3: TTL Pareto frontier sweep at $t = 24$h using BGE-M3. TTL-based baselines do not surpass FreshCache’s results under any TTL configuration.

Analysis of Cache Tiers and Workload Semantics

Per-class and per-tier analyses illustrate not only that staleness risk and potential for safe reuse vary widely across freshness classes, but also that semantic similarity is often a poor proxy for safe reuse in the presence of volatility.

Figure 4: Cosine similarity between query pairs vs. Jaccard overlap of their top-returned URL sets, colored by freshness class; demonstrates selective L2 reuse is justified by workload dynamics.

TIMELESS and SLOW queries with high semantic overlap show reliable URL set overlap, making tiered L2 cache hits highly effective and safe. MEDIUM and FAST queries, despite high semantic similarity, demonstrate substantially lower URL overlap and higher volatility, requiring stricter gating to prevent drift-induced errors. Notably, only FreshCache’s fine-grained temporal estimation and stratified tier structure accommodate this non-uniformity in cacheable evidence.

Limitations and Future Directions

FreshCache demonstrates robust performance gains but also exposes several open challenges:

• Short-Term Calibration Lag: Rule-based half-life fitting displays limited advantage over TTL at very short ($\varepsilon_{L2} = 0.20$0h) horizons, a gap the learned risk model partially closes.
• Conditional GET Inefficacy: Lightweight page validation via HTTP headers proves unreliable and slow in practice, reinforcing the necessity of higher-tier risk gating for performance.
• Distribution Shift: The learned risk model demonstrates decrease in hit rate and increased cache-induced errors when evaluated on out-of-domain benchmarks (e.g., Wikipedia-style or highly event-driven queries), emphasizing an unmet need for adaptive risk calibration or richer entity- and domain-centric features to generalize beyond training distributions.

Practical implications point towards adaptive, self-calibrating error budgets and joint integration with authenticated data streams for FAST/REAL_TIME classes, while theoretical implications motivate further investigation into joint modeling of semantic and temporal similarity in open-domain retrieval.

Conclusion

FreshCache constitutes a substantial advance in semantic caching for open-web RAG by treating cache reuse as a risk-constrained, temporally-sensitive inference task. By combining three-tier caching, per-tier staleness risk modeling—supported by both calibrated parametric decay and learned ML predictors—and stratified error budgets, FreshCache achieves near-optimal trade-offs in efficiency and answer freshness. Its benchmark, evaluation methodology, and empirical findings set a new standard for both practical RAG deployments and foundational research on time-variance in open-domain LLM pipelines.

Key results: FreshCache_MLP achieves 0.1% stale error and 97% search savings, with answer-level stale error as low as 0.034%. Tiered and risk-gated design delivers efficiency without sacrificing correctness, outperforming all tested baselines (2607.04281).

For full technical details, see "Risk-Constrained Freshness-Aware Semantic Caching for Open-Web Retrieval-Augmented LLMs" (2607.04281).