LLMCache: Efficient LLM Inference

• LLMCache is a set of algorithms and architectures designed to optimize caching for large language models, reducing inference costs and latency.
• It employs strategies from token-level key–value reuse to semantic and predictive caching to balance speed and memory constraints.
• LLMCache integrates methods from systems, databases, machine learning, and hardware design to address challenges like long sequence lengths and prompt variability.

LLM caching (LLMCache) encompasses a diverse set of algorithms, architectures, and system-level designs aimed at reducing inference cost, latency, and energy in large-scale LLM deployment. Techniques span from explicit key–value caching for token-level reuse to semantic caching of input–output pairs and predictive approaches using model-driven instruction pre-population. The field integrates methods from systems, databases, machine learning, and hardware design to address unique characteristics of LLM serving—including long sequence lengths, prompt variability, memory bottlenecks, and user-specific interaction dynamics.

1. Caching Taxonomy and Motivation

LLMCache mechanisms fall along several axes:

• Token-Level Key–Value (KV) Caches: Store hidden state key and value tensors for each generated token, enabling O(1) autoregressive decoding and avoiding recomputation. Widely implemented in production LLM inference engines, but impose linear or superlinear memory growth with sequence length, creating scaling and deployment bottlenecks (Cheng et al., 8 Oct 2025).
• Semantic and Instruction-Level Caches: Cache inference results based on input semantic equivalence or similarity (rather than syntactic exact match), generalized to clusters or centroids of queries to maximize cache coverage under memory constraints (Kim et al., 26 Aug 2025, Li et al., 2024).
• Predictive Instruction Caches: Use an LLM (fine-tuned for prompt prediction) to proactively pre-populate likely instructions and answers offline, enabling high hit rates with bounded memory via efficient hash lookup (Zou et al., 2024).
• Layer-wise Activation Reuse: Cache intermediate activations at the granularity of transformer layers, keyed by semantic fingerprints of the input, and adaptively reuse or recompute based on similarities (Bansal, 18 Dec 2025).
• Statistical, Bandit, and Learning-Based Caching: Frame cache replacement as online knapsack or bandit optimization—explicitly handling query heterogeneity, unknown workloads, or stochastic costs (Yang et al., 19 Sep 2025, Liu et al., 11 Aug 2025).
• Tool-Call and Workflow-Aware Caching: Integrate system and semantic features to cache LLM tool calls (API interactions), balancing recency, frequency, and estimated caching value under constraints of request freshness and semantic heterogeneity (Zhai et al., 20 Jan 2026).
• Statistical Independence-Preserving Caching: Special handling is required for workflows that draw multiple independent samples per query (e.g., Pass@k analysis, multi-round repair). These systems provide type-level abstractions to ensure probabilistic correctness when reusing cached responses (Dai et al., 27 Nov 2025).

LLMCache exists to address the growing computational and memory costs of serving ever-larger transformer models, especially in multi-turn, retrieval-augmented, or tool-calling settings with strict service-level objectives and high user concurrency.

2. Key Algorithms and Data Structures

Token- and KV-Level Strategies

• Naive KV Cache: Stores $K, V$ tensors per token and layer, with memory $O(b\,s\,d\,L)$. Essential for efficient decode-time incremental forwarding, but unsustainable for long contexts (He et al., 2024, Poudel, 23 Oct 2025, Cheng et al., 8 Oct 2025).
• Sparse/Fast Variants: KCache and related approaches offload most value tensors to CPU or host after prefill and fetch only the top-N attention-relevant values per step to drastically reduce GPU memory while retaining accuracy and throughput for long contexts ($N\ll s$, e.g., $N=128$) (He et al., 2024).

Semantic and Centroid-Based Caching

• Embedding-based Matching: Queries are embedded (e.g., via SBERT, MPNet, or custom LLM encoders), then matched to nearest neighbors in the cache by cosine similarity (Kim et al., 26 Aug 2025, Gill et al., 2024, Li et al., 2024).
• Centroid Clustering: To minimize redundancy, cache entries represent dense centroids of semantically grouped queries; SISO and SCALM algorithms assign each new query to the closest centroid above a similarity threshold, trading minor output fidelity loss for a 10x reduction in memory (Kim et al., 26 Aug 2025, Li et al., 2024).
• Adaptive Thresholding and Locality-Aware Eviction: Retrieval similarity thresholds and cache admission/eviction are dynamically tuned based on empirical hit ratio, service-level constraints, and long-term popularity (semantic locality score) (Kim et al., 26 Aug 2025).

Predictive and Instruction-Level Caching

• NLL-Driven Pre-population: InstCache scores all short candidate user instructions by their negative log-likelihood (under a prompt-aligned LLM), pre-computes high-probability responses, and stores them in a flat hash table; hit rate and cache memory are controlled via the NLL threshold $\sigma$ (Zou et al., 2024).
• Breadth-First Tree Search: Instruction candidates are generated in breadth-first order, branches are pruned as soon as cumulative NLL exceeds $\sigma$ or depth limit $d$.

Learning-based and Bandit Formulations

• Reverse-Greedy Supermodular Optimization: Semantic cache selection can be formulated as a knapsack problem minimizing expected semantic-mismatch loss; a reverse greedy removal algorithm achieves provable approximation guarantees (Liu et al., 11 Aug 2025, Yang et al., 19 Sep 2025).
• Accumulative Oracle: Cache recomputation/invalidation is triggered only after sufficient new evidence is collected per query or globally, reducing the computational cost of expensive knapsack updates (Yang et al., 19 Sep 2025).
• Value-Aware Admission and Eviction (VAAC): ToolCaching applies multi-factor scoring (recency, frequency, estimated latency/value) in both admission and eviction, often managed by bandit algorithms for adaptive, feature-group-aware cache decisions (Zhai et al., 20 Jan 2026).

Layer-Wise and Activation Caching

• Semantic Fingerprinting: Inputs are summarized via embedding aggregation, attention head statistics, and dimensionality reduction (e.g., SimHash) to generate fingerprints for cache look-up at every transformer layer. Cache hit at layer $l$ allows immediate reuse of cached $h_l$, skipping computation of $f_l(h_{l-1})$ (Bansal, 18 Dec 2025).
• Eviction Based on Freshness and Divergence: LRU-style freshness scoring, staleness-aware decay, and divergence monitors are combined to manage per-layer cache entries (Bansal, 18 Dec 2025).

Statistical Independence

• Type-Structured Caching: Systems such as Mnimi decorate LLM API interfaces with type wrappers (Repeatable, Independent) that guarantee repeatability or independence of cached output streams at the level of user workflow requirements (Dai et al., 27 Nov 2025).

3. Performance Trade-Offs and Evaluations

Quantitative gains and design trade-offs observed across LLMCache variants include:

System / Method	Hit Ratio / Coverage	Latency/Throughput	Memory Cost	Accuracy Impact
InstCache (Zou et al., 2024)	51.34% (LMSys, $O(b\,s\,d\,L)$0100 tokens)	2x speedup over vLLM	$O(b\,s\,d\,L)$14.5 GB (4.25M inst.)	Answer quality on par with LLM
SISO (Kim et al., 26 Aug 2025)	40–60% w/ 6% data cached	2–3x lower avg. latency	6% of queries as centroids	Negligible quality drop ($O(b\,s\,d\,L)$27% F1)
SCALM (Li et al., 2024)	+63% hit, +77% token savings	N/A	N/A	Contextual gains and lower redundancy
LMCache (Cheng et al., 8 Oct 2025)	Up to $O(b\,s\,d\,L)$3 throughput gain	2–14x lower latency	Leverages multi-tier storage	Transparent to answer quality
KCache (He et al., 2024)	N/A	40–55% higher for long ctx	only $O(b\,s\,d\,L)$4 layers kept on-GPU	$O(b\,s\,d\,L)$5 few-shot accuracy loss
LLMCache layerwise (Bansal, 18 Dec 2025)	Hit rate layers 1–4: 75–92%	3.1x total speedup	1.2 GB for BERT-base	$O(b\,s\,d\,L)$60.5% accuracy drop (SQuAD, WikiText)
IndexMem (Yang et al., 25 May 2026)	Max accuracy at each cache budget	Consistent wins on RULER, LB	Cache: $O(b\,s\,d\,L)$74–7.7 GB, latent add $O(b\,s\,d\,L)$80.5M params/layer	Up to 25 points gain vs. heuristics under high eviction
SwiftCache (Hu et al., 15 Jun 2026)	N/A	–69% P99 TTFT, 3.98x longer contexts	HBM/NVLink cooperation	No effect on answer correctness

Semantic caches (SISO, SCALM) achieve significant hit gains via clustering and locality-aware replacement. Predictive instruction caches (InstCache) dominate for short, repetitive workloads. Token-level KV caches excel in decoder-only, high-latency LLMs but are replaced by layerwise fingerprints or KCache in memory-constrained long-sequence settings. Learning-based bandit/knapsack approaches provide regret-based efficiency guarantees in non-stationary, heterogeneous workloads (Yang et al., 19 Sep 2025, Liu et al., 11 Aug 2025).

4. Deployment, Scalability, and Workflow Integration

Deployment choices reflect LLMCache’s design spectrum:

• Hash Tables and Dicts: Used in predictive and instruction-level caches for O(1) expected lookup and minimal memory overhead (Zou et al., 2024).
• Approximate Nearest-Neighbor Indexes: Employed (e.g., HNSW, IVF) for semantic cache matching, with sublinear or logarithmic lookup time (Kim et al., 26 Aug 2025, Gill et al., 2024).
• Chunked I/O, NVLink, and Multi-Tier Memory: Systems such as LMCache and SwiftCache exploit GPU memory, CPU RAM, local/remote NVMe, and high-bandwidth NVLink to enable cache sharing, chunked offload/reload, and cross-query reuse (Cheng et al., 8 Oct 2025, Hu et al., 15 Jun 2026).
• Federated Learning and Personalization: For user-centric caches, local FL-based updates yield privacy-preserving, adaptive similarity metrics and context-aware matching (Gill et al., 2024).
• Statistical Abstractions and Workflow Enforcement: Type-driven cache wrappers (Mnimi) enforce independence and reproducibility in metric-driven, research, or program synthesis pipelines (Dai et al., 27 Nov 2025).

Cross-model cache sharing and elastic memory donation (SwiftCache) increase context limits and reduce PCIe bottlenecks, but require high-bandwidth server interconnect (NVLink) and co-location of heterogeneous LLMs (Hu et al., 15 Jun 2026).

5. Limitations, Open Problems, and Research Frontiers

LLMCache design is shaped by foundational and emergent constraints:

• Architectural Limits: Token-level caches must honor LLM context window and positional encoding integrity (e.g., RoPE). Non-contiguous eviction or over-retention induces prompt repetition and degenerative output; contiguous “gist” block eviction is optimal for maintaining fidelity in long stateful conversations (Poudel, 23 Oct 2025).
• OOV Drift and Concept Shift: Semantic and fingerprint caches degrade under input distribution shift or frequent LLM fine-tuning unless divergence monitoring is aggressive and caches are adaptively recomputed (Bansal, 18 Dec 2025).
• Memory and Search Overhead: High-dimensional embedding caches must balance memory usage with lookup speed. Techniques include dimensionality reduction, centroid/cluster condensation, and, where possible, user-specific personalization (Kim et al., 26 Aug 2025, Gill et al., 2024).
• Statistical Integrity: In research/coding workflows, naive caching can bias metrics (e.g., Pass@k, program repair loop success), mandating cache designs that encode independence in their interfaces (Dai et al., 27 Nov 2025).
• Latency vs. Fidelity: Aggressive semantic or predictive caching may trade output fidelity for SLO and latency gains, typically below 0.5–7% F1 loss (Kim et al., 26 Aug 2025, Bansal, 18 Dec 2025).
• Dynamic Workloads and Adaptive Scheduling: Dynamic thresholding, multi-factor eviction, and bandit-based admission ensure resilience to fluctuating burstiness, hot-spot shift, and long-tail query distribution (Zhai et al., 20 Jan 2026, Kim et al., 26 Aug 2025, Yang et al., 19 Sep 2025).

Emerging directions include learned, trainable cache fingerprints, distributed cache state propagation, hybridizing semantic and activation reuse, and tighter hardware/software co-design for domain-specific inference accelerators (Bansal, 18 Dec 2025, Zhou et al., 26 Nov 2025, Cheng et al., 8 Oct 2025).

6. Notable LLMCache Systems and Their Impact

Predictive and Instruction Caches

• InstCache (Zou et al., 2024): Achieves high hit rates for short, repetitive user queries; scales to millions of instructions with low memory and O(1) lookup via hash-tables. Provides formal mapping from cache size to hit-rate via NLL-thresholding.

Semantic/Clustered Caches

• SISO (Kim et al., 26 Aug 2025): Employs centroid clustering for semantic locality, dramatically improving hit ratio and SLO compliance, with locality-aware replacement superior to LRU/LFU.
• SCALM (Li et al., 2024): Hierarchically clusters queries to patterns, ranks by token-saving and popularity, and realizes LFU-based eviction.

System-level Caches

• LMCache (Cheng et al., 8 Oct 2025): Open-source enterprise-grade KV cache layer supports prefix reuse, batched I/O, and disaggregation; enables up to $O(b\,s\,d\,L)$9 throughput gain in production.
• SwiftCache (Hu et al., 15 Jun 2026): Coordinates GPU-to-GPU memory over NVLink for elastic cache sharing, achieving substantial context-length extension and tail-latency drop.

Learning/Bandit-Driven Approaches

• LLM Cache Bandit (Yang et al., 19 Sep 2025): Models cache selection as an online knapsack problem, introducing accumulation-based knapsack updating and achieving O($N\ll s$0) regret.
• Semantic Caching with Learning (Liu et al., 11 Aug 2025): Reverse-greedy and bandit algorithms provide optimal submodular minimization and sublinear regret under uncertain query/cost environments.

Statistical/Workflow-Safe Caches

• Mnimi (Dai et al., 27 Nov 2025): Abstraction layer enforcing statistical invariants for workflows, critical for Pass@k, code repair, and experimentation.

Layer-Wise and Activation Caches

• LLMCache (Bansal, 18 Dec 2025): Semantic fingerprinted activation reuse at arbitrary transformer layers (both encoder and decoder), with empirically validated up to $N\ll s$1 speedup and $N\ll s$20.5% accuracy drop.

7. Conclusion

LLMCache research spans fundamental algorithmic, system, and practical engineering domains, offering multi-layered solutions for efficient, low-latency, cost-effective LLM serving at scale. Through advancements in predictive caching, semantic centroids, hardware-aware memory architectures, and adaptive learning-based scheduling, LLMCache is now able to match the diverse demands of modern LLM applications while maintaining strict quality, statistical integrity, and performance guarantees (Zou et al., 2024, Kim et al., 26 Aug 2025, Cheng et al., 8 Oct 2025, Bansal, 18 Dec 2025, Yang et al., 19 Sep 2025, Dai et al., 27 Nov 2025).