Content-Based Prefix Caching

• Content-based prefix caching is a method for reusing precomputed key/value states based on repeated content, enabling efficient, position-independent token processing.
• It employs techniques like hashing, prefix-tries, and content-defined chunking to optimize computation in multi-turn, retrieval-augmented, and agentic workflows.
• Adaptive eviction policies, robust security measures, and distributed architectures are integrated to enhance performance and mitigate multi-tenant side-channel risks.

Content-based prefix caching in the context of LLM serving systems refers to the practice of storing and reusing precomputed internal model states (typically per-layer key/value, or KV, tensors) associated with repeated content segments of input tokens, rather than strictly matching their prefix position in the prompt. This approach enables substantial efficiency gains by eliminating redundant “prefill” computation for recurrent or templated content, even when it appears at variable positions or in dynamic assembly patterns. Content-based caching thus supplants strict sequence-position indexing and captures semantic duplication invisible to traditional prefix caches. The field now recognizes multiple engineering, algorithmic, and security dimensions—from cache lookup, hashing, and state serialization, through adaptive eviction and semantic scoring, to robust defenses against side-channel leakage and attacks.

1. Core Mechanisms of Content-Based Prefix Caching

The canonical content-based prefix caching mechanism for transformer-based LLMs is Automatic Prefix Caching (APC). For any prompt $x = (x_1, ..., x_T)$, the model first performs a “prefill” pass computing per-layer key/value matrices $K_{1:T}, V_{1:T}$, with $K_{1:T} \in \mathbb{R}^{L \times d_k}$ and $V_{1:T} \in \mathbb{R}^{L \times d_v}$, where $L$ is the number of transformer layers and $d_k, d_v$ the per-head dimensions. APC stores the mapping from a prefix $p = (x_1, ..., x_k)$ to its cached state $s_p := (K_{1:k}, V_{1:k})$ in a global table, using a hash $h(p) = H(t_1 \Vert ... \Vert t_k)$ over the token IDs and a fast string-hash (e.g., FNV, Murmur3) (Pennas et al., 11 Mar 2026). On a new request, if its prefix matches a stored $h(p)$, the system resumes computation from $K_{1:T}, V_{1:T}$0 and only processes the prompt’s suffix.

Beyond hash tables, prefix-trie or radix-tree data structures efficiently map variable-length token sequences to cached states. Each node encodes the state up to its corresponding prefix. These representations allow both fast longest-prefix lookups and scalable storage.

In advanced deployments (e.g., distributed serving), KV cache entries are content-addressed and may be sharded to remote storage, enabling cache re-use at the scale of large, multi-node LLM clusters (Xiang et al., 21 Sep 2025). Compression and chunking allow large KV blocks to be fetched as compressed segments, further optimizing network- and memory-bound scenarios.

2. From Prefix to Content-Based and Position-Independent Caching

While traditional prefix-based caching requires exact prefix and token-position matches, content-based caching generalizes to support position-independent (and sometimes span-based) reuse. This is especially valuable in RAG, multi-turn chat, and agentic workloads where content may appear at arbitrary positions or be re-inserted (e.g., retrieval outputs, system prompts, tool results).

Systems like Irminsul and MiniPIC explicitly decouple cache keys from sequence position. In MLA (Multi-Head Latent Attention) models, the key–value row is factored into a “position-free” latent $K_{1:T}, V_{1:T}$1 and a small position-sensitive $K_{1:T}, V_{1:T}$2, so a cached segment at reference position $K_{1:T}, V_{1:T}$3 can be reused anywhere via a closed-form $K_{1:T}, V_{1:T}$4-rotation on $K_{1:T}, V_{1:T}$5: $K_{1:T}, V_{1:T}$6 This operation is $K_{1:T}, V_{1:T}$7 per token for typical MLA architectures and allows deterministic content-addressed block reuse regardless of offset (Ma et al., 7 May 2026).

For standard RoPE-based transformers, MiniPIC achieves position-independence by storing unrotated keys $K_{1:T}, V_{1:T}$8 in the cache and applying position-appropriate rotation $K_{1:T}, V_{1:T}$9 at attention time: $K_{1:T} \in \mathbb{R}^{L \times d_k}$0 All position bookkeeping is thus delayed to the attention kernel, so a single shared cache entry can be referenced from arbitrarily many logical prompt positions (Ordonez et al., 11 Jun 2026).

Explicit content-defined chunking (CDC), rolling hash boundaries, or block-aligned markers (e.g., SSep, PDep in MiniPIC) ensure repeatable segmentation for dynamic traffic. This enables robust matching of repeated document spans regardless of prompt assembly.

3. Eviction, Admission, and Adaptive Policies

Efficiency of content-based caching depends on admission and eviction strategies under strict GPU memory budgets. Standard policies (e.g., LRU, LFU) are agnostic to the semantic or structural value of tokens, treating every cached block as equally valuable. Empirical analyses show up to $K_{1:T} \in \mathbb{R}^{L \times d_k}$1 variation in reuse rates between token types—e.g., system prompts can have 92.3% reuse, compared to 2.2% for other regions (Fang et al., 12 May 2026).

SAECache introduces a semantic-adaptive eviction policy using a multi-queue architecture, routing blocks by session structure and token semantics. Each queue employs its own priority function:

• Structural templates use block-position decay,
• Multi-turn chat and agentic sequences use log-normal survival models,
• Token-type weights $K_{1:T} \in \mathbb{R}^{L \times d_k}$2 are updated online from eviction/hit feedback.

Eviction combines global queue weights, learned type weights, and local priorities: $K_{1:T} \in \mathbb{R}^{L \times d_k}$3 Meta-parameters and weights are continuously adapted online, ensuring robustness to workload drift and removing manual tuning.

Marconi targets hybrid attention + SSM (state-space model) architectures, where partial sequence overlaps cannot be efficiently rolled back. It employs a trie with per-node KV and SSM states and only admits checkpoints with high forecasted reuse—specifically, at the last decoded token and new speculative “input” branch points. Eviction priorities combine recency with FLOP efficiency (compute saved per memory footprint) (Pan et al., 2024).

4. Security and Multi-Tenancy: Side-Channel Leakage and Defenses

Content-based APC in multi-tenant LLM serving can introduce timing side-channels: cache hits incur lower prefill latency than misses, potentially leaking information about shared prompt prefixes across tenants (Pennas et al., 11 Mar 2026). Attackers can mount probing campaigns, reconstructing victim requests by observing time-to-first-token (TTFT) patterns.

CacheSolidarity addresses this by augmenting KV caches with metadata (OwnerID, AttackFlag) and deploying a lightweight detector-activator pipeline. The detector tracks cross-tenant cache hits, flags reused entries, and, upon detecting suspicious cross-user accesses, selectively isolates affected prefixes (i.e., disables cache sharing from that point for the non-owner). The activator continuously monitors TTFT distributions and disables cache sharing dynamically when hit/miss timing is statistically distinguishable (KDE overlap $K_{1:T} \in \mathbb{R}^{L \times d_k}$4 below threshold $K_{1:T} \in \mathbb{R}^{L \times d_k}$5). This design recovers $K_{1:T} \in \mathbb{R}^{L \times d_k}$670% more cache hits and reduces TTFT by 30% relative to full user isolation, while closing the side-channel for attackers. Overhead is minimal: $K_{1:T} \in \mathbb{R}^{L \times d_k}$7 ms per request, $K_{1:T} \in \mathbb{R}^{L \times d_k}$8 B metadata per entry, and up to 95% of APC performance is retained (Pennas et al., 11 Mar 2026).

5. Distributed and Agentic Caching: System Architecture and Advanced Use Cases

At cluster scale, content-based prefix caching may be distributed across multiple nodes, with entries fetched from remote stores as needed. ShadowServe implements a SmartNIC-accelerated chunked pipeline, where control-plane logic runs on the host and KV fetch, decompression, dequantization, and direct GPU DMA operate entirely on SmartNIC hardware. This removes host GPU/CPU interference (previously $K_{1:T} \in \mathbb{R}^{L \times d_k}$9 slowdown) and achieves up to $V_{1:T} \in \mathbb{R}^{L \times d_v}$0 lower time-per-output-token (TPOT) and $V_{1:T} \in \mathbb{R}^{L \times d_v}$1 higher throughput (under network constraints) versus GPU-side decompressing baselines. Efficient minimal-copy memory management across pipeline stages (preallocated, chunk-partitioned buffers) is key for performance (Xiang et al., 21 Sep 2025).

Agentic LLM serving, which dynamically assembles prompts from operator-controlled or environmental sources, introduces non-deterministic variation in prefix boundaries. Systems such as Irminsul leverage CDC chunking and content fingerprinting for variable-length segments, along with the native key/value factorization of MLA, to match repeated spans independent of their prompt offset. This results in up to 83% of “above-exact-prefix” token recovery in agentic workloads, with 63% GPU energy reduction per cache hit (Ma et al., 7 May 2026).

Position-Independent Caching (PIC) can also be “native” to the model design. COMB attaches a lightweight encoder and cross-attention adapter to a frozen decoder-only LLM, is trained to produce content-keyed KV for arbitrary chunk permutations, and integrates into both standard HuggingFace and vLLM stacks. This system delivers 51–94% TTFT reductions, $V_{1:T} \in \mathbb{R}^{L \times d_v}$2 higher throughput, and maintains accuracy parity under arbitrary span orderings (Zhao et al., 2 Feb 2026).

6. Results, Quantitative Metrics, and System Evaluation

Empirical evaluations of content-based and position-independent prefix caching systems demonstrate substantial efficiency gains:

System	Token Hit Rate	TTFT Reduction	Notable Details
CacheSolidarity	80%	30% (vs. isolated)	95% of vanilla APC performance (Pennas et al., 11 Mar 2026)
SAECache	+4.8–5.9 pp	1.4–2.7$V_{1:T} \in \mathbb{R}^{L \times d_v}$3	Robust to workload drift (Fang et al., 12 May 2026)
Marconi (Hybrid)	4.5–34.4$V_{1:T} \in \mathbb{R}^{L \times d_v}$4	71% (P95 $V_{1:T} \in \mathbb{R}^{L \times d_v}$5TTFT)	For hybrid models w/ SSM (Pan et al., 2024)
ShadowServe	---	Up to 1.38$V_{1:T} \in \mathbb{R}^{L \times d_v}$6	1.35$V_{1:T} \in \mathbb{R}^{L \times d_v}$7 throughput, SmartNIC (Xiang et al., 21 Sep 2025)
Irminsul	83% (above prefix)	63% energy savings	Agentic/MLA, 80% total coverage (Ma et al., 7 May 2026)
MiniPIC	Up to 49% throughput	10–100$V_{1:T} \in \mathbb{R}^{L \times d_v}$8 TTFT	<100 LOC core changes (Ordonez et al., 11 Jun 2026)
COMB	---	94% (TTFT hit)	Plug-in, encoder-based PIC (Zhao et al., 2 Feb 2026)

Systems are evaluated on metrics including TTFT, token-hit ratio, throughput, prefill energy, memory footprint, and accuracy (F1/Rouge). Adaptive eviction, position-independent reuse, and semantic differentiation are critical to robust, high-utilization cache operation.

7. Theoretical and Practical Significance

Content-based prefix caching has become a first-class efficiency and scalability primitive in LLM serving. Architectural insights—such as separating position from content in KV design, semantically weighted policies, and dynamic metadata extensions—have brought both workload robustness and new threat surfaces. Agentic and retrieval-augmented pipelines particularly benefit by minimizing redundant encoding of repetitive, shared, or dynamically assembled content.

The paradigm shift is toward decoupling what is computed from where it appears, supported by content hashes, position-adaptive compute, and compositional cache assembly. Native model architectures (MLA, encoder-decoder hybrids for PIC) and minimal-intrusive server integration (e.g., MiniPIC < 100 LOC) enable rapid adoption and flexible deployment (Ma et al., 7 May 2026, Ordonez et al., 11 Jun 2026, Zhao et al., 2 Feb 2026). Finally, security implications such as timing side-channels demand cache-aware multi-tenancy controls, as in CacheSolidarity, to align practical efficiency with user and tenant isolation requirements (Pennas et al., 11 Mar 2026).

Collectively, the literature establishes content-based prefix caching as a multi-layered competence: its efficacy now hinges on robust semantic adaptation, architectural compatibility, and security-aware design.