---
title: Radix Prefix Cache in LLM Serving
url: https://www.emergentmind.com/topics/radix-prefix-cache
type: topic
---

# Radix Prefix Cache in LLM Serving

A radix prefix cache is a prefix-indexed reuse mechanism that exploits shared leading segments of structured inputs to avoid redundant computation or state materialization. In the LLM-serving literature represented here, the central form is a radix trie keyed by BPE token sequences whose nodes point to KV-cache state for the corresponding prefix, enabling longest-prefix-match lookup and reuse of previously prefilling work; related formulations use prefix tries over \((\text{token\_id}, \text{position\_id})\) pairs for intra-batch deduplication, helper arrays of cached suffix prefixes in full-text indexing, and a distinct “prefixCache” array for inter-block prefix sums in GPU radix sort [2605.26289][2601.15013][1607.08176][2206.01784]. This breadth of usage suggests that the term denotes a family of prefix-oriented caching and compaction techniques rather than a single canonical data structure.

## 1. Core definition and data-structural forms

In multi-agent LLM serving, a radix prefix cache \(\mathcal R\) is defined as a radix trie keyed by BPE token sequences. A single unified KV-cache memory on the GPU is partitioned into fixed slots called “sequences” numbered \(\sigma_1 \dots \sigma_N\). Trie nodes record a token segment, a donor sequence ID \(\sigma_{\text{donor}}\) whose KV cells hold the model state for that prefix, and a last-access timestamp for LRU eviction at leaves. The central invariant is that, for every path from the root to a node \(u_m\) labeled by the token prefix \(T[0 \dots m-1]\), \(u_m.\text{donor}=\sigma_{\text{donor}}\) points to GPU KV-cache cells that contain exactly the prefill state after processing \(T[0 \dots m-1]\) [2605.26289].

Given an incoming prompt \(T=(T_1,T_2,\dots,T_{n_t})\), the lookup problem is to find
\[
m = \max \{ \ell \in [0..n_t] \mid (T_1 \dots T_\ell) \text{ exists as a path in } \mathcal R \}.
\]
The trie returns \((m,\sigma_{\text{donor}})\), and the system aliases the donor’s first \(m\) tokens’ KV pages into a new sequence slot \(\sigma\) in \(O(1)\) time. This is an exact-prefix semantics: the cache reuses only a contiguous leading subsequence already represented in the trie [2605.26289].

A related but not identical formulation appears in RadixMLP. There, the radix structure is a prefix trie over a batch \(\mathcal S=\{S_1,\dots,S_B\}\), with each node corresponding to a unique path of \((\text{token\_id},\text{position\_id})\) pairs from the root. The trie identifies repeated causal histories inside a single forward pass, producing a compact token set of size \(N' \le N\), compact-to-original ratio \(\gamma=N'/N\), and compression ratio \(r=N/N'\) [2601.15013].

| Setting | Indexed prefix unit | Reused artifact |
|---|---|---|
| Stateful multi-agent serving | BPE token sequence | KV-cache pages via donor sequence |
| Intra-batch RadixMLP | \((\text{token\_id},\text{position\_id})\) path | Position-wise activations in compact space |
| Suffix-array helper cache | First \(k\) symbols of a suffix | Faster comparison prefix |
| Onesweep prefixCache | \((\text{tile},\text{digit})\) prefix entry | Inter-block prefix-sum state |

The table underscores a key distinction. In LLM serving, “radix prefix cache” usually refers to exact-prefix reuse of model state; in other domains, the same phrase denotes cached comparison prefixes or prefix-sum metadata rather than semantic prompt reuse.

## 2. Exact-prefix reuse in KV-cache serving

The canonical serving workflow is longest-prefix-match followed by alias-and-prefill. For prompt tokens \(T[0..n_t-1]\), the system first computes \((m,\sigma_{\text{donor}})\leftarrow\mathcal R.\text{longest\_prefix}(T)\), then, if \(m>0\), performs constant-time metadata aliasing from \(\sigma_{\text{donor}}\) to the newly acquired sequence slot \(\sigma\). Only the delta tokens \(T[m..n_t-1]\) are then decoded in mini-batches, after which the new suffix is inserted into the trie by walking to the node for length \(m\) and appending nodes for \(T[m..]\), storing \(\sigma\) as the donor at the new leaf [2605.26289].

This exact-prefix organization changes the asymptotic shape of multi-turn serving. Without stateful reuse, the per-turn cost is
\[
T_{\text{std}}(t)=T_{\text{prefill}}(n_t)+m \cdot T_{\text{decode}},
\]
so the cumulative cost over turns is \(O(T\cdot \bar n)\). With perfect prefix reuse through the radix structure,
\[
T_{\text{radix}}(t)=T_{\text{restore}}+T_{\text{prefill}}(\Delta_t)+\lceil m/(k+1)\rceil \cdot T_{\text{decode}},
\]
yielding \(O(n_1+T\cdot \bar \Delta)\) with \(\bar \Delta \ll \bar n\). The paper summarizes this operationally as converting the \(O(n_t)\) per-turn cost of conventional serving into an \(O(\Delta_t)\) delta-only cost [2605.26289].

The radix trie is integrated with a sequence pool and a continuous-batch scheduler. A global GPU context holds a KV-cache of size \((N\ \text{sequences} \times \text{max\_length})\), with reserved sequence IDs for long-lived agent conversations and transient slots for stateless or response-cache requests. Admission is governed by a cell-budget \(C_{\text{budget}}\), and eviction uses leaf-oldest LRU so that deep shared prefixes, including tool schemas and system prompts, tend to be preserved [2605.26289].

This exact-prefix formulation is particularly natural for append-only conversational traces. The cache state is keyed by literal token history, not by semantic equivalence. As a result, it is precise, low-overhead, and operationally simple, but it inherits the brittleness of exact matching.

## 3. Intra-batch compaction and position-wise deduplication

A radix prefix cache can also be understood as a transient compaction mechanism rather than a persistent KV-store. RadixMLP is explicitly stateless and operates within a single forward pass. Its premise is that, in a causal Transformer, each layer alternates a position-wise block and a sequence-mixing self-attention block, and any two tokens with identical causal history will produce identical activations under the position-wise block. Shared prefixes in a batch therefore induce redundant LayerNorm, linear projection, embedding, and MLP computation [2601.15013].

The implementation strategy is gather \(\rightarrow\) compute \(\rightarrow\) scatter. A CPU-side trie construction emits two integer arrays: gather indices \(I_{\mathrm{gather}}\in\mathbb N^{N'}\), which choose one representative original position for each unique trie node, and scatter indices \(I_{\mathrm{scatter}}\in\mathbb N^N\), which map each original token position back to its compact trie node. Hidden states are gathered into compact space, position-wise operators are executed on \(N'\) rows instead of \(N\), results are scattered back before self-attention, and attention is still evaluated in original space. The trie therefore preserves causal consistency while coalescing only the position-wise computations that are guaranteed to be identical [2601.15013].

The complexity statement is explicit. Standard prefill performs approximately \(\mathcal O(N d^2)\) position-wise work plus attention costs; RadixMLP reduces the position-wise component to \(\mathcal O(N' d^2 + N)\). The additional gather/scatter copies are memory-bound \(\mathcal O(N)\), and memory overhead is two 32-bit integer arrays of length \(N\) and \(N'\), hence \(\mathcal O(N)\) [2601.15013].

This usage broadens the meaning of radix prefix caching. The radix structure is not serving as a persistent cache of KV pages across requests; it is a transient deduplication index over a batch. The common abstraction is the same—coalescing identical prefixes—but the reused object is different.

## 4. Beyond exact-prefix semantics: position-independent and content-addressed reuse

A major limitation of ordinary radix prefix caches is that they are “exact prefix only.” In agentic workloads, bit-identical tokens often reappear at shifted absolute positions after the insertion of metadata or tool outputs, and operators report cache-hit regressions ranging from moderate slowdowns to severe TTFT spikes of \(10\!-\!16\) s on unchanged content. SGLang’s original RadixAttention builds a prefix trie over token-sequence hashes at fixed block size, so lookup fails at the first divergence even if most subsequent content is unchanged [2605.05696].

Irminsul extends this radix cache with content-hash keying over CDC-chunked segments and a \(\delta\)-rotation rule for Multi-Head Latent Attention. The architectural basis is MLA decomposition: each KV row is factored into a position-free latent \(c_{KV}\in\mathbb R^{512}\) and a \(64\)-dimensional RoPE slice \(k_r\in\mathbb R^{64}\), so that
\[
K(x,p)=[\,W_{uk}\cdot c_{KV}(x),\; R(p)\,k_{r,\text{base}}(x)\,].
\]
Because \(c_{KV}(x)\) is position-free, it can be reused verbatim at a new absolute position; only \(k_r\) must be corrected. If a chunk was originally computed at source position \(p_{\text{src}}\) and reattached at position \(p\), the corrective rotation is \(\delta=p-p_{\text{src}}\), giving \(k_r(p)=R(\delta)\,k_{r,\text{base}}\). The paper states that \(R(\delta)\) is exact up to bf16 quant error \(\approx 4.7\times 10^{-3}\) rel-\(L_2\), far below FP8 quant noise [2605.05696].

Chunking is content-defined. Irminsul uses a Gear-hash rolling window of \(64\) tokens, declares a boundary when \((\text{rolling\_state} \& ((1 \ll k)-1))==0\) with \(k=7\), yields expected chunk length \(\approx 128\) tokens clamped to \([32,512]\), and computes a single xxHash64 of the entire chunk. The registry maps
\[
\text{xxHash64}(\text{chunk\_bytes}) \mapsto (c_{KV}\_\text{block}, k_{r,\text{base}\_\text{block}}, p_{\text{src}}),
\]
so later appearances of the same chunk can reuse its KV regardless of absolute position. The serving path first takes the exact-prefix match from the radix trie, then runs CDC on the remaining tail, and consults the content registry for each segment [2605.05696].

The reported outcome is that, on agentic shifts, exact-prefix alone recovers \(1.9\%\) of tokens on DeepSeek-V2-Lite and \(1.5\%\) on JoyAI-Flash, whereas Irminsul-unique recovery is \(77.2\%\) and \(82.7\%\), for total cacheable fractions of \(\sim 79\%\) and \(\sim 84\%\). The same study reports \(63\%\) prefill energy savings per cache hit for MLA deployments measured by NVML. The paper’s interpretive conclusion is that content-addressed caching should be treated as a first-class serving primitive rather than a retrofit over prefix matching [2605.05696].

## 5. Scheduling, homogeneity, and the overhead of radix-tree traversal

Radix prefix caches are also used by schedulers to detect and exploit shared prefixes across pending requests. Existing engines such as SGLang build a global radix tree of all active prefixes and, during batch formation, traverse the tree depth-first to group requests that share long common prefixes. The immediate benefit is reduced KV-cache memory footprint: if \(k\) requests share the first \(L\) tokens, the shared prefix need only appear once in GPU memory, and only the divergent suffixes require additional KV blocks [2605.06046].

The scheduling cost can, however, be substantial. For \(W\) waiting requests of maximum prefix length \(T\), longest-prefix-match costs \(\Theta(T)\) per request, so matching and sorting yields \(\mathcal O(W\cdot T + W\log W)\). Depth-First-Search with Weighting traverses all \(B\) nodes of the radix tree and sorts children by subtree weight, costing roughly \(\mathcal O(B + B\log B)\). Because \(B\) grows with total history rather than merely the current working set, these traversals can dominate the critical path [2605.06046].

The empirical results are explicit. Table 1 of Feather reports that SGLang’s DFS-W spends approximately \(50\!-\!90\%\) of total latency on CPU scheduling versus GPU decode, while vLLM’s radix-tree variant incurs approximately \(45\%\) CPU overhead; an oracle with requests pre-labeled by prefix costs less than \(0.1\%\). The same study shows that, with prefix-sharing workloads, smaller, prefix-homogeneous batches can achieve higher decode throughput than larger heterogeneous batches, and that decode throughput drops approximately \(2\times\) versus fully homogeneous batches when only two prefix groups exist; as the number of prefix groups grows past GPU-memory capacity, heterogeneous batches incur KV-cache evictions and see another approximately \(2\times\) drop in throughput [2605.06046].

Feather replaces tree traversal with Chunked Hash Tree and uses reinforcement learning to decide when to stop adding requests to a batch. CHT splits each request into fixed-size chunks, hashes the prefix at each chunk level, maintains a working set of \((\ell,h)\) pairs covered by the active batch, and tracks each waiting request’s missing count in a min-heap. The paper reports that CHT’s scheduling overhead is less than \(1\%\) of GPU execution time at \(20\)K-token prompts, versus \(50\!-\!90\%\) for SGLang’s radix-tree policies, and that Feather achieves \(2\!-\!10\times\) higher end-to-end throughput versus existing schedulers, with up to \(22\times\) on LongChat13B [2605.06046].

The scheduling perspective introduces an important qualification. Prefix awareness is not automatically beneficial; the mechanism used to detect and operationalize shared prefixes can itself become the bottleneck.

## 6. Limitations, misconceptions, and broader lineage

The sharpest limitation of classical radix prefix caches in LLM serving is exactness. In the stateful multi-agent formulation, any token mismatch yields a cache miss; the method works only for strictly append-only workloads and does not support in-place edits or reordering. Trie lookup remains an \(O(m)\) BPE-token pass per request, and single-GPU designs do not directly solve distributed or sharded KV-cache coordination, which would require distributed trie coordination and page-table aliasing across devices [2605.26289].

A related misconception is that all observed speedups should be attributed to the radix cache itself. The stateful inference study is explicit that its advantage comes from stateful reuse and speculation, not caching. Its reference implementation combines a persistent KV cache across turns, a radix prefix cache across interleaved multi-agent traffic, and a prompt-lookup speculative decoder; against vLLM and SGLang it is reported as \(2.1\times\) faster per turn on a \(6\)-turn agentic workflow and \(4.2\times\) on the median turn of a \(35\)-turn one, halving end-to-end wall time [2605.26289].

Outside LLM serving, the term has a broader lineage. In suffix-array search, a helper array \(H\) stores the first \(k\) characters of selected suffixes,
\[
H[i]=T[\mathrm{SA}[i]\dots \mathrm{SA}[i]+k-1],
\]
so that binary-search probes can compare against a cached contiguous prefix rather than immediately touching random text locations. The memory overhead is \(O(nk)\), and, on the English200 dataset with \(B=32\) and top \(L=4\) levels cached, \(k=8\) reduces count time from approximately \(0.320\,\mu s\) to approximately \(0.289\,\mu s\), while \(k=12\) reaches approximately \(0.285\,\mu s\) [1607.08176].

The phrase also appears in a still more distinct sense in Onesweep, where a two-dimensional `prefixCache` stores one \(32\)-bit `PrefixEntry` per \((\text{tile},\text{digit})\), with status in \(\{N,L,G\}\) and value packed into the remaining bits. That structure supports decoupled look-back for blockwise prefix aggregation during LSD radix sort, reducing digit-binning memory traffic from approximately \(3n\) to approximately \(2n\) global memory operations per iteration and yielding approximately \(1.4\!-\!1.6\times\) speedup on NVIDIA A100 [2206.01784].

This broader record suggests that “radix prefix cache” is best understood as a reusable design pattern: cache or compact information associated with shared prefixes so that later computation can be replaced by lookup, aliasing, or short-circuit comparison. In modern LLM systems, the leading research question is no longer whether prefix reuse matters, but which prefix relation is being indexed—exact token prefix, causal-history prefix, or content-defined segment—and what overhead is incurred in making that relation operational [2605.05696][2605.06046].

Source: https://www.emergentmind.com/topics/radix-prefix-cache