Radix Prefix Cache in LLM Serving

• Radix prefix cache is a data structure using a prefix trie keyed by BPE token sequences to store and reuse prefill KV-cache states.
• It enables efficient longest-prefix matching in LLM serving, allowing constant-time aliasing and reducing per-turn computational overhead.
• Advanced implementations integrate CDC-based content addressing and reinforcement learning scheduling to overcome exact-prefix limitations and boost throughput.

A radix prefix cache is a prefix-indexed reuse mechanism that exploits shared leading segments of structured inputs to avoid redundant computation or state materialization. In the LLM-serving literature represented here, the central form is a radix trie keyed by BPE token sequences whose nodes point to KV-cache state for the corresponding prefix, enabling longest-prefix-match lookup and reuse of previously prefilling work; related formulations use prefix tries over $(\text{token\_id}, \text{position\_id})$ pairs for intra-batch deduplication, helper arrays of cached suffix prefixes in full-text indexing, and a distinct “prefixCache” array for inter-block prefix sums in GPU radix sort (Norgren, 25 May 2026, Feil et al., 21 Jan 2026, Kowalski et al., 2016, Adinets et al., 2022). This breadth of usage suggests that the term denotes a family of prefix-oriented caching and compaction techniques rather than a single canonical data structure.

1. Core definition and data-structural forms

In multi-agent LLM serving, a radix prefix cache $\mathcal R$ is defined as a radix trie keyed by BPE token sequences. A single unified KV-cache memory on the GPU is partitioned into fixed slots called “sequences” numbered $\sigma_1 \dots \sigma_N$. Trie nodes record a token segment, a donor sequence ID $\sigma_{\text{donor}}$ whose KV cells hold the model state for that prefix, and a last-access timestamp for LRU eviction at leaves. The central invariant is that, for every path from the root to a node $u_m$ labeled by the token prefix $T[0 \dots m-1]$, $u_m.\text{donor}=\sigma_{\text{donor}}$ points to GPU KV-cache cells that contain exactly the prefill state after processing $T[0 \dots m-1]$ (Norgren, 25 May 2026).

Given an incoming prompt $T=(T_1,T_2,\dots,T_{n_t})$, the lookup problem is to find

$$m = \max \{ \ell \in [0..n_t] \mid (T_1 \dots T_\ell) \text{ exists as a path in } \mathcal R \}.$$

The trie returns $\mathcal R$0, and the system aliases the donor’s first $\mathcal R$1 tokens’ KV pages into a new sequence slot $\mathcal R$2 in $\mathcal R$3 time. This is an exact-prefix semantics: the cache reuses only a contiguous leading subsequence already represented in the trie (Norgren, 25 May 2026).

A related but not identical formulation appears in RadixMLP. There, the radix structure is a prefix trie over a batch $\mathcal R$4, with each node corresponding to a unique path of $\mathcal R$5 pairs from the root. The trie identifies repeated causal histories inside a single forward pass, producing a compact token set of size $\mathcal R$6, compact-to-original ratio $\mathcal R$7, and compression ratio $\mathcal R$8 (Feil et al., 21 Jan 2026).

Setting	Indexed prefix unit	Reused artifact
Stateful multi-agent serving	BPE token sequence	KV-cache pages via donor sequence
Intra-batch RadixMLP	$\mathcal R$9 path	Position-wise activations in compact space
Suffix-array helper cache	First $\sigma_1 \dots \sigma_N$0 symbols of a suffix	Faster comparison prefix
Onesweep prefixCache	$\sigma_1 \dots \sigma_N$1 prefix entry	Inter-block prefix-sum state

The table underscores a key distinction. In LLM serving, “radix prefix cache” usually refers to exact-prefix reuse of model state; in other domains, the same phrase denotes cached comparison prefixes or prefix-sum metadata rather than semantic prompt reuse.

2. Exact-prefix reuse in KV-cache serving

The canonical serving workflow is longest-prefix-match followed by alias-and-prefill. For prompt tokens $\sigma_1 \dots \sigma_N$2, the system first computes $\sigma_1 \dots \sigma_N$3, then, if $\sigma_1 \dots \sigma_N$4, performs constant-time metadata aliasing from $\sigma_1 \dots \sigma_N$5 to the newly acquired sequence slot $\sigma_1 \dots \sigma_N$6. Only the delta tokens $\sigma_1 \dots \sigma_N$7 are then decoded in mini-batches, after which the new suffix is inserted into the trie by walking to the node for length $\sigma_1 \dots \sigma_N$8 and appending nodes for $\sigma_1 \dots \sigma_N$9, storing $\sigma_{\text{donor}}$0 as the donor at the new leaf (Norgren, 25 May 2026).

This exact-prefix organization changes the asymptotic shape of multi-turn serving. Without stateful reuse, the per-turn cost is

$\sigma_{\text{donor}}$1

so the cumulative cost over turns is $\sigma_{\text{donor}}$2. With perfect prefix reuse through the radix structure,

$\sigma_{\text{donor}}$3

yielding $\sigma_{\text{donor}}$4 with $\sigma_{\text{donor}}$5. The paper summarizes this operationally as converting the $\sigma_{\text{donor}}$6 per-turn cost of conventional serving into an $\sigma_{\text{donor}}$7 delta-only cost (Norgren, 25 May 2026).

The radix trie is integrated with a sequence pool and a continuous-batch scheduler. A global GPU context holds a KV-cache of size $\sigma_{\text{donor}}$8, with reserved sequence IDs for long-lived agent conversations and transient slots for stateless or response-cache requests. Admission is governed by a cell-budget $\sigma_{\text{donor}}$9, and eviction uses leaf-oldest LRU so that deep shared prefixes, including tool schemas and system prompts, tend to be preserved (Norgren, 25 May 2026).

This exact-prefix formulation is particularly natural for append-only conversational traces. The cache state is keyed by literal token history, not by semantic equivalence. As a result, it is precise, low-overhead, and operationally simple, but it inherits the brittleness of exact matching.

3. Intra-batch compaction and position-wise deduplication

A radix prefix cache can also be understood as a transient compaction mechanism rather than a persistent KV-store. RadixMLP is explicitly stateless and operates within a single forward pass. Its premise is that, in a causal Transformer, each layer alternates a position-wise block and a sequence-mixing self-attention block, and any two tokens with identical causal history will produce identical activations under the position-wise block. Shared prefixes in a batch therefore induce redundant LayerNorm, linear projection, embedding, and MLP computation (Feil et al., 21 Jan 2026).

The implementation strategy is gather $u_m$0 compute $u_m$1 scatter. A CPU-side trie construction emits two integer arrays: gather indices $u_m$2, which choose one representative original position for each unique trie node, and scatter indices $u_m$3, which map each original token position back to its compact trie node. Hidden states are gathered into compact space, position-wise operators are executed on $u_m$4 rows instead of $u_m$5, results are scattered back before self-attention, and attention is still evaluated in original space. The trie therefore preserves causal consistency while coalescing only the position-wise computations that are guaranteed to be identical (Feil et al., 21 Jan 2026).

The complexity statement is explicit. Standard prefill performs approximately $u_m$6 position-wise work plus attention costs; RadixMLP reduces the position-wise component to $u_m$7. The additional gather/scatter copies are memory-bound $u_m$8, and memory overhead is two 32-bit integer arrays of length $u_m$9 and $T[0 \dots m-1]$0, hence $T[0 \dots m-1]$1 (Feil et al., 21 Jan 2026).

This usage broadens the meaning of radix prefix caching. The radix structure is not serving as a persistent cache of KV pages across requests; it is a transient deduplication index over a batch. The common abstraction is the same—coalescing identical prefixes—but the reused object is different.

4. Beyond exact-prefix semantics: position-independent and content-addressed reuse

A major limitation of ordinary radix prefix caches is that they are “exact prefix only.” In agentic workloads, bit-identical tokens often reappear at shifted absolute positions after the insertion of metadata or tool outputs, and operators report cache-hit regressions ranging from moderate slowdowns to severe TTFT spikes of $T[0 \dots m-1]$2 s on unchanged content. SGLang’s original RadixAttention builds a prefix trie over token-sequence hashes at fixed block size, so lookup fails at the first divergence even if most subsequent content is unchanged (Ma et al., 7 May 2026).

Irminsul extends this radix cache with content-hash keying over CDC-chunked segments and a $T[0 \dots m-1]$3-rotation rule for Multi-Head Latent Attention. The architectural basis is MLA decomposition: each KV row is factored into a position-free latent $T[0 \dots m-1]$4 and a $T[0 \dots m-1]$5-dimensional RoPE slice $T[0 \dots m-1]$6, so that

$T[0 \dots m-1]$7

Because $T[0 \dots m-1]$8 is position-free, it can be reused verbatim at a new absolute position; only $T[0 \dots m-1]$9 must be corrected. If a chunk was originally computed at source position $u_m.\text{donor}=\sigma_{\text{donor}}$0 and reattached at position $u_m.\text{donor}=\sigma_{\text{donor}}$1, the corrective rotation is $u_m.\text{donor}=\sigma_{\text{donor}}$2, giving $u_m.\text{donor}=\sigma_{\text{donor}}$3. The paper states that $u_m.\text{donor}=\sigma_{\text{donor}}$4 is exact up to bf16 quant error $u_m.\text{donor}=\sigma_{\text{donor}}$5 rel-$u_m.\text{donor}=\sigma_{\text{donor}}$6, far below FP8 quant noise (Ma et al., 7 May 2026).

Chunking is content-defined. Irminsul uses a Gear-hash rolling window of $u_m.\text{donor}=\sigma_{\text{donor}}$7 tokens, declares a boundary when $u_m.\text{donor}=\sigma_{\text{donor}}$8 with $u_m.\text{donor}=\sigma_{\text{donor}}$9, yields expected chunk length $T[0 \dots m-1]$0 tokens clamped to $T[0 \dots m-1]$1, and computes a single xxHash64 of the entire chunk. The registry maps

$T[0 \dots m-1]$2

so later appearances of the same chunk can reuse its KV regardless of absolute position. The serving path first takes the exact-prefix match from the radix trie, then runs CDC on the remaining tail, and consults the content registry for each segment (Ma et al., 7 May 2026).

The reported outcome is that, on agentic shifts, exact-prefix alone recovers $T[0 \dots m-1]$3 of tokens on DeepSeek-V2-Lite and $T[0 \dots m-1]$4 on JoyAI-Flash, whereas Irminsul-unique recovery is $T[0 \dots m-1]$5 and $T[0 \dots m-1]$6, for total cacheable fractions of $T[0 \dots m-1]$7 and $T[0 \dots m-1]$8. The same study reports $T[0 \dots m-1]$9 prefill energy savings per cache hit for MLA deployments measured by NVML. The paper’s interpretive conclusion is that content-addressed caching should be treated as a first-class serving primitive rather than a retrofit over prefix matching (Ma et al., 7 May 2026).

5. Scheduling, homogeneity, and the overhead of radix-tree traversal

Radix prefix caches are also used by schedulers to detect and exploit shared prefixes across pending requests. Existing engines such as SGLang build a global radix tree of all active prefixes and, during batch formation, traverse the tree depth-first to group requests that share long common prefixes. The immediate benefit is reduced KV-cache memory footprint: if $T=(T_1,T_2,\dots,T_{n_t})$0 requests share the first $T=(T_1,T_2,\dots,T_{n_t})$1 tokens, the shared prefix need only appear once in GPU memory, and only the divergent suffixes require additional KV blocks (Rathi et al., 7 May 2026).

The scheduling cost can, however, be substantial. For $T=(T_1,T_2,\dots,T_{n_t})$2 waiting requests of maximum prefix length $T=(T_1,T_2,\dots,T_{n_t})$3, longest-prefix-match costs $T=(T_1,T_2,\dots,T_{n_t})$4 per request, so matching and sorting yields $T=(T_1,T_2,\dots,T_{n_t})$5. Depth-First-Search with Weighting traverses all $T=(T_1,T_2,\dots,T_{n_t})$6 nodes of the radix tree and sorts children by subtree weight, costing roughly $T=(T_1,T_2,\dots,T_{n_t})$7. Because $T=(T_1,T_2,\dots,T_{n_t})$8 grows with total history rather than merely the current working set, these traversals can dominate the critical path (Rathi et al., 7 May 2026).

The empirical results are explicit. Table 1 of Feather reports that SGLang’s DFS-W spends approximately $T=(T_1,T_2,\dots,T_{n_t})$9 of total latency on CPU scheduling versus GPU decode, while vLLM’s radix-tree variant incurs approximately $$m = \max \{ \ell \in [0..n_t] \mid (T_1 \dots T_\ell) \text{ exists as a path in } \mathcal R \}.$$0 CPU overhead; an oracle with requests pre-labeled by prefix costs less than $$m = \max \{ \ell \in [0..n_t] \mid (T_1 \dots T_\ell) \text{ exists as a path in } \mathcal R \}.$$1. The same study shows that, with prefix-sharing workloads, smaller, prefix-homogeneous batches can achieve higher decode throughput than larger heterogeneous batches, and that decode throughput drops approximately $$m = \max \{ \ell \in [0..n_t] \mid (T_1 \dots T_\ell) \text{ exists as a path in } \mathcal R \}.$$2 versus fully homogeneous batches when only two prefix groups exist; as the number of prefix groups grows past GPU-memory capacity, heterogeneous batches incur KV-cache evictions and see another approximately $$m = \max \{ \ell \in [0..n_t] \mid (T_1 \dots T_\ell) \text{ exists as a path in } \mathcal R \}.$$3 drop in throughput (Rathi et al., 7 May 2026).

Feather replaces tree traversal with Chunked Hash Tree and uses reinforcement learning to decide when to stop adding requests to a batch. CHT splits each request into fixed-size chunks, hashes the prefix at each chunk level, maintains a working set of $$m = \max \{ \ell \in [0..n_t] \mid (T_1 \dots T_\ell) \text{ exists as a path in } \mathcal R \}.$$4 pairs covered by the active batch, and tracks each waiting request’s missing count in a min-heap. The paper reports that CHT’s scheduling overhead is less than $$m = \max \{ \ell \in [0..n_t] \mid (T_1 \dots T_\ell) \text{ exists as a path in } \mathcal R \}.$$5 of GPU execution time at $$m = \max \{ \ell \in [0..n_t] \mid (T_1 \dots T_\ell) \text{ exists as a path in } \mathcal R \}.$$6K-token prompts, versus $$m = \max \{ \ell \in [0..n_t] \mid (T_1 \dots T_\ell) \text{ exists as a path in } \mathcal R \}.$$7 for SGLang’s radix-tree policies, and that Feather achieves $$m = \max \{ \ell \in [0..n_t] \mid (T_1 \dots T_\ell) \text{ exists as a path in } \mathcal R \}.$$8 higher end-to-end throughput versus existing schedulers, with up to $$m = \max \{ \ell \in [0..n_t] \mid (T_1 \dots T_\ell) \text{ exists as a path in } \mathcal R \}.$$9 on LongChat13B (Rathi et al., 7 May 2026).

The scheduling perspective introduces an important qualification. Prefix awareness is not automatically beneficial; the mechanism used to detect and operationalize shared prefixes can itself become the bottleneck.

6. Limitations, misconceptions, and broader lineage

The sharpest limitation of classical radix prefix caches in LLM serving is exactness. In the stateful multi-agent formulation, any token mismatch yields a cache miss; the method works only for strictly append-only workloads and does not support in-place edits or reordering. Trie lookup remains an $\mathcal R$00 BPE-token pass per request, and single-GPU designs do not directly solve distributed or sharded KV-cache coordination, which would require distributed trie coordination and page-table aliasing across devices (Norgren, 25 May 2026).

A related misconception is that all observed speedups should be attributed to the radix cache itself. The stateful inference study is explicit that its advantage comes from stateful reuse and speculation, not caching. Its reference implementation combines a persistent KV cache across turns, a radix prefix cache across interleaved multi-agent traffic, and a prompt-lookup speculative decoder; against vLLM and SGLang it is reported as $\mathcal R$01 faster per turn on a $\mathcal R$02-turn agentic workflow and $\mathcal R$03 on the median turn of a $\mathcal R$04-turn one, halving end-to-end wall time (Norgren, 25 May 2026).

Outside LLM serving, the term has a broader lineage. In suffix-array search, a helper array $\mathcal R$05 stores the first $\mathcal R$06 characters of selected suffixes,

$\mathcal R$07

so that binary-search probes can compare against a cached contiguous prefix rather than immediately touching random text locations. The memory overhead is $\mathcal R$08, and, on the English200 dataset with $\mathcal R$09 and top $\mathcal R$10 levels cached, $\mathcal R$11 reduces count time from approximately $\mathcal R$12 to approximately $\mathcal R$13, while $\mathcal R$14 reaches approximately $\mathcal R$15 (Kowalski et al., 2016).

The phrase also appears in a still more distinct sense in Onesweep, where a two-dimensional prefixCache stores one $\mathcal R$16-bit PrefixEntry per $\mathcal R$17, with status in $\mathcal R$18 and value packed into the remaining bits. That structure supports decoupled look-back for blockwise prefix aggregation during LSD radix sort, reducing digit-binning memory traffic from approximately $\mathcal R$19 to approximately $\mathcal R$20 global memory operations per iteration and yielding approximately $\mathcal R$21 speedup on NVIDIA A100 (Adinets et al., 2022).

This broader record suggests that “radix prefix cache” is best understood as a reusable design pattern: cache or compact information associated with shared prefixes so that later computation can be replaced by lookup, aliasing, or short-circuit comparison. In modern LLM systems, the leading research question is no longer whether prefix reuse matters, but which prefix relation is being indexed—exact token prefix, causal-history prefix, or content-defined segment—and what overhead is incurred in making that relation operational (Ma et al., 7 May 2026, Rathi et al., 7 May 2026).