RAC: Relation-Aware Cache Replacement for Large Language Models

Abstract: The scaling of LLM services faces significant cost and latency challenges, making effective caching under tight capacity crucial. Existing cache replacement policies, from heuristics to learning-based methods, predominantly rely on limited-window statistics such as recency and frequency. We show these signals are not robust for real-world LLM workloads, which exhibit long reuse distances and sparse local recurrence. To address these limitations, we propose Relation-Aware Cache (RAC), an online eviction strategy that leverages semantic relations among requests to guide eviction decisions. RAC synthesizes two relation-aware signals: (1) Topical Prevalence, which aggregates access evidence at the topic level to capture long-horizon reuse; and (2) Structural Importance, which leverages local intra-topic dependency structure to discriminate entries by their future reuse value. Extensive evaluations show that RAC maintains high effectiveness across diverse workloads, consistently surpassing state-of-the-art baselines by 20%--30% in cache hit ratio.

• The paper introduces RAC, a cache replacement policy that multiplies long-term topical prevalence by each query’s structural importance to preserve reusable context under sparse recurrence and tight capacity limits.
• RAC maintains topic and dependency statistics online with constant-time updates, using embedding-based topic routing and lightweight parent-link detection for semantic and KV-cache settings.
• Experiments show 15–20% gains on synthetic workloads and 5–12% gains over the strongest baselines on OASST1 traces, while highlighting sensitivity to topic-routing accuracy and dependency coverage.

Motivation and problem setting

LLM serving increasingly relies on caching to amortize inference cost, either at the semantic level (reusing generated responses via embedding similarity, as in GPTCache) or at the intermediate-state level (reusing KV states for prefill). Under tight cache budgets, the eviction policy determines how much of this reuse is realized. The paper's central observation, drawn from recent workload characterizations, is that real LLM request streams exhibit long reuse distances and sparse local recurrence: most entries are touched once or reappear only after long gaps. Under such workloads, short-window statistics—recency, frequency, and their combinations—cease to correlate with future reuse, and learning-based policies constrained by finite prediction horizons fail to capture reuse events beyond their observable window.

The paper formalizes the setting as an online admission-and-eviction problem over a query stream $Q=\{q_t\}$ with a capacity-$C$ cache, maximizing total hits under a system-defined hit criterion (semantic equivalence via embedding similarity threshold $\tau$, or content/prefix equivalence for KV caches). The workload model is topic-aware: each query carries a topic label $Z_t$, topics recur across episodes, and the episode-level sequence is modeled as a semi-Markov process. Within an episode, queries are connected by time-respecting discourse dependency links $\mathcal{E}_s$ forming a DAG, capturing prerequisite context flow. The formulation is deliberately agnostic to cache type, so the policy can be instantiated for both semantic and KV caching.

Design of RAC

RAC's eviction value is motivated by a mixture decomposition: the marginal probability of a query factors as $p(q_t)=\sum_s \pi_s\, p(q_t\mid s)$, where $\pi_s$ is the topic's long-run occupancy and $p(q\mid s)$ is the in-topic query distribution. Because of topic locality, the in-topic term $\pi_{Z_t}p(q_t\mid Z_t)$ dominates. RAC approximates this product with two online signals and defines the unified heuristic value:

$\text{Value}(q) = \text{TP}(Z)\cdot \text{TSI}(q).$

Topical Prevalence (TP) is a per-topic temporal score aggregating exponentially decayed hit evidence, $C$0. It is maintained with two per-topic scalars (last hit time and last stored value), giving $C$1 updates and lazy closed-form evaluation during eviction. Topic routing uses a cache-side index of representative embeddings with an ANN shortlist and a similarity gate at $C$2; representatives are anchor-based (the embedding of the highest-TSI resident member), with lazy refresh on anchor eviction to keep maintenance amortized.

Topic Structural Importance (TSI) proxies in-topic strength at the item level: $C$3, where $C$4 aggregates the request mass of downstream dependents. A theorem establishes the key justification: under prerequisite semantics, evicting an anchor $C$5 incurs a long-run miss increase lower-bounded by the number of dependent-query requests, hence monotone in $C$6. The proof relies on a conservative "unavoidable miss" accounting that charges only misses forced by the missing anchor—an assumption that makes the bound valid but partial, since it ignores interactions with other resident entries.

Dependencies are detected online by a lightweight one-parent scheme: each query attaches to at most one parent chosen among cached candidates within a look-back window $C$7 and similarity threshold $C$8, scored by $C$9. Cached parent pointers make each access a constant-time update cascade over $\tau$0, $\tau$1, and $\tau$2. An appendix extends the one-hop $\tau$3 to a PageRank/TextRank-style ranking on the reversed dependency DAG, proving existence and uniqueness of the stationary score and its computability by power iteration; this refinement is presented as optional and is not evaluated in the main experiments.

Evaluation

The evaluation addresses four questions using timestamp-continuous OASST1 sub-traces (10 non-overlapping traces of 10,000 requests each) and synthetic traces from a topic-level semi-Markov generator (120 topics, ~40 sessions per topic, capacity fixed at 10% for stress tests). Performance is reported as normalized hit ratio against an infinite-cache upper bound. Baselines span classic heuristics (LRU, FIFO, CLOCK, TTL), scan-resistant policies (TinyLFU, ARC, S3-FIFO, SIEVE, 2Q), and adaptive/learning methods (LHD, LeCaR), all under identical hit semantics.

Setting	RAC gain over strongest baseline
Synthetic, long-reuse-distance sweep (50–90%)	double-digit relative, growing to tens of percent
Synthetic, Zipf skew sweep ($\tau$4)	~15–20% consistently
Real OASST1 traces, 2.5–20% capacity	~5–12% (larger vs. baseline average)

The abstract's headline claim of a consistent 20–30% hit-ratio improvement is thus an aggregate across regimes: gains are largest on synthetic stress workloads and moderate (5–12% over the strongest baseline) on real traces. This distinction matters—on real OASST1 dialogue, the advantage over the best baseline is real but narrower than the abstract's framing suggests, and the larger "baseline average" margins partly reflect weak baselines under sparse recurrence.

The ablation shows complementary roles: removing TSI causes the sharpest degradation in the cache-cliff regime (tight budgets), since preserving within-topic context anchors matters most when capacity is scarce; TP contributes persistently across all capacities, supporting long-horizon revisits where local statistics are sparse. Parameter sensitivity results indicate a broad stable operating region for the routing threshold $\tau$5, decay coefficient $\tau$6, and structural weight $\tau$7, supporting the claim that RAC does not require delicate tuning.

Limitations and open questions

Several limitations are conceded or implicit. First, the workload model assumes topics are identifiable online via embedding routing with a fixed threshold ($\tau$8 calibrated against ChatGPT-judged semantic equivalence); routing errors propagate directly into TP, and the paper does not quantify sensitivity to topic-fragmentation or routing misassignment beyond the $\tau$9 sweep. Second, the dependency detector is restricted to a one-parent, within-episode, look-back-window design; dependencies spanning episodes or involving multiple parents are not captured, and the more expressive DAG-based structural ranking remains unevaluated. Third, the theorem's miss accounting is a conservative lower bound under prerequisite semantics, so TSI's theoretical justification is partial rather than a tight characterization of eviction cost. Fourth, the real-trace evaluation is limited to OASST1 dialogue; generalization to KV-cache workloads with compositional hit semantics, multi-tenant production traffic, and the overhead of semantic similarity computation at scale (which the authors note is itself costly) is not demonstrated. Finally, the synthetic generator's session structure was produced with ChatGPT, so the stress workloads may inherit structural regularities that favor relation-aware signals.

Conclusion

RAC replaces entry-local recency/frequency signals with two relation-aware statistics—topic-level prevalence and intra-topic structural importance—combined multiplicatively under a hard capacity constraint, with constant-time online maintenance throughout. The empirical results support robustness precisely in the regimes where short-window policies fail, though the magnitude of gains on real traces is more modest than on the synthetic stress tests, and the approach's dependence on accurate online topic routing and lightweight dependency detection remains the principal open question for deployment beyond dialogue workloads.