RadixAttention: Prefix Reuse for LLM Inference

• RadixAttention is a prefix reuse mechanism that stores cached KV tensors in a radix tree to avoid recomputation in autoregressive inference.
• It leverages GPU memory management, LRU eviction, and custom CUDA extend kernels to efficiently update and reuse cached states.
• By framing a query scheduling problem under latency constraints, it balances throughput and time-to-first-token performance across diverse workloads.

Searching arXiv for papers on RadixAttention and related SGLang scheduling work. RadixAttention is a prefix reuse mechanism for autoregressive LLM inference that automatically caches and reuses the key/value (KV) tensors of common prompt prefixes across different queries. In SGLang, it is introduced as a runtime optimization for efficient execution of structured LLM programs: it retains past KV caches in a single radix tree keyed by token sequences, performs longest-prefix matching at request time, reuses the cached $KV(p)$ for a shared prefix $p$, and computes only the new keys and values for the remaining suffix. Subsequent work treats RadixAttention not only as a cache design, but as the basis of a query-scheduling problem under latency constraints, especially time-to-first-token (TTFT) in online serving (Zheng et al., 2023, Dexter et al., 7 Feb 2025).

1. Motivation and problem setting

Standard autoregressive decoding with Transformers recomputes the entire KV cache for every generation call, even when many calls share a large common prefix of tokens. In LLM programs such as few-shot prompts, self-consistency sampling, tree-of-thought search, and multi-turn chat, this prefix reuse can be substantial, yet off-the-shelf inference engines throw away the KV cache after each call. In the notation used for SGLang, if $P$ is the shared prefix length, $U$ is the new suffix length, and $M$ is the number of calls, the resulting work is $O((P+U)^2)$ per call and $O(M\cdot(P+U)^2)$ across calls (Zheng et al., 2023).

RadixAttention addresses this redundancy by retaining all past KV caches in a single radix tree keyed by token sequences, doing longest-prefix matching at request time, reusing the cached $KV(p)$ for a shared prefix $p$, and only computing the new keys and values for the suffix $s$. The later scheduling analysis makes explicit that the practical benefit of this reuse depends on processing order: in online settings, reducing prefill computation and minimizing end-to-end latency are related but not identical objectives, because TTFT also depends on queueing and fairness across queries (Dexter et al., 7 Feb 2025).

2. Radix tree organization and KV-cache semantics

The core data structure is a radix tree, or compact trie, whose edges are labeled by nonempty token subsequences. In the SGLang formulation, each node corresponds to the concatenation of all edge-labels from the root; in the scheduling formulation, every node $p$0 corresponds to a string $p$1. In both descriptions, the node stores cached KV tensors for the represented prefix. SGLang describes the stored state as $p$2 together with a reference count and LRU metadata; the scheduling paper describes each node as storing the cached KV tensors after encoding $p$3 and maintaining a “last-used” timestamp (Zheng et al., 2023, Dexter et al., 7 Feb 2025).

Within the SGLang runtime, SGVM keeps a CPU-resident radix tree $p$4 whose nodes map token-sequences to GPU pages holding KV tensors. Nodes carry an LRU timestamp and a reference counter and are pinned while in use by a running batch. The front end always submits the full prompt; SGVM performs prefix matching internally. This separation is significant because the cache is exposed as a runtime facility rather than as an application-level programming burden (Zheng et al., 2023).

Insertion, lookup, and eviction are all defined at the radix-tree level. When a query is completely processed, RadixAttention walks or extends the tree along the query, allocating nodes as needed, inserting the KV state, and updating recency metadata. To process a new query, it descends from the root matching as many tokens as possible, reuses the KV cache at the matched depth, and recomputes only the unmatched suffix. Under a fixed memory budget, least-recently-used nodes are evicted; in SGLang this is specialized to LRU eviction of leaf nodes with ref_count=0 when GPU memory is scarce (Dexter et al., 7 Feb 2025, Zheng et al., 2023).

3. Algorithmic mechanics of prefix reuse

The formal description in SGLang begins with the token vocabulary $p$5 and treats each prompt or partial generation as a sequence $p$6. For a standard Transformer attention layer on a sequence $p$7, one forms

$p$8

and computes

$p$9

In streaming decode, after processing a prefix $P$0 of length $P$1, the system stores

$P$2

When a new suffix $P$3 arrives, standard decoding would recompute $P$4 and $P$5. RadixAttention instead performs longest-prefix matching, finds the maximal cached prefix $P$6, computes only the remaining suffix $P$7, forms

$P$8

and inserts the resulting node into the radix tree (Zheng et al., 2023).

The runtime procedure described in SGLang is explicitly cache-aware. Waiting requests are prefix-matched against the tree, sorted by matched prefix length, and admitted into a batch subject to available memory and evictable cache size. Prefix nodes are pinned by incrementing reference counts; if allocation fails, the tree evicts LRU leaf nodes; execution then proceeds through a fused prefill/extend kernel. Under the hood, the B.run(buffer) step reads $P$9 from GPU pages, possibly non-contiguous, runs an “extend” CUDA kernel to compute $U$0 for the new suffix, and appends these pages contiguously in GPU memory. The cache-aware scheduler is intended to increase locality and hit rate, while the custom extend kernels eliminate extra data movement between cached pages and newly computed suffix pages (Zheng et al., 2023).

4. Complexity and asymptotic regime

The principal asymptotic claim for RadixAttention is a change in scaling with respect to repeated calls that share long prefixes. With hidden dimension $U$1, standard execution without reuse incurs per-call cost

$U$2

and total cost

$U$3

RadixAttention changes this to a one-time prefix build cost of

$U$4

and per-call suffix cost of

$U$5

yielding total cost

$U$6

When $U$7 or $U$8, the method is asymptotically better: from $U$9 to $M$0. In the common regime $M$1 or $M$2, this is effectively linear in $M$3 rather than quadratic (Zheng et al., 2023).

This asymptotic improvement is concentrated in prefill rather than decode. The scheduling paper states that decoding cost is dominated by prefill in its experimental setting, and accordingly formulates the benefit of RadixAttention primarily through overlap in prompt prefixes. A plausible implication is that the mechanism is most advantageous when the service workload exhibits repeated long prompts, repeated user-history segments, or structured prompting patterns that preserve long common prefixes across many requests (Dexter et al., 7 Feb 2025).

5. Latency-constrained scheduling under RadixAttention

The later theoretical treatment models RadixAttention as a query-scheduling problem. A query stream of length $M$4 is defined as $M$5, where each $M$6 is a token string and $M$7 its arrival time. If the processing order is $M$8 and $M$9 captures the relative cost of self-attention versus pointwise FFN, then the finish times satisfy

$O((P+U)^2)$0

and

$O((P+U)^2)$1

where

$O((P+U)^2)$2

The TTFT of query $O((P+U)^2)$3 is $O((P+U)^2)$4. The scheduling objective is to minimize the maximum TTFT or to decide feasibility under a hard deadline $O((P+U)^2)$5 (Dexter et al., 7 Feb 2025).

Within this framework, the decision problem “Does there exist a schedule with $O((P+U)^2)$6?” is shown to be strongly NP-Hard via a reduction from 3-PARTITION. The construction uses $O((P+U)^2)$7 queries $O((P+U)^2)$8 of lengths $O((P+U)^2)$9, $O(M\cdot(P+U)^2)$0 paired queries $O(M\cdot(P+U)^2)$1 of length $O(M\cdot(P+U)^2)$2, and two long guard queries $O(M\cdot(P+U)^2)$3 that force the schedule into a constrained time window. The same paper also gives an $O(M\cdot(P+U)^2)$4-time algorithm that either certifies that no schedule can achieve max-TTFT $O(M\cdot(P+U)^2)$5 or produces a schedule whose $O(M\cdot(P+U)^2)$6-th percentile TTFT is $O(M\cdot(P+U)^2)$7, assuming maximum prompt length and $O(M\cdot(P+U)^2)$8 are constants (Dexter et al., 7 Feb 2025).

To balance prefix reuse and fairness, the paper introduces $O(M\cdot(P+U)^2)$9-LPM, which interpolates between FCFS at $KV(p)$0 and greedy longest-prefix-match at $KV(p)$1. The algorithm repeatedly processes the oldest waiting query and then, for $KV(p)$2, processes the remaining query with the largest prefix overlap to the last processed query. A naive implementation requires $KV(p)$3 total overlap computations, while more sophisticated radix-tree lookups and incremental overlap tracking can achieve $KV(p)$4 overall. On a stylized regular-arrival shuffled queue of prompts of the form $KV(p)$5, Theorem 4.1 bounds FCFS, LPM, and $KV(p)$6-LPM differently, and Corollary 4.2 shows that when $KV(p)$7 and $KV(p)$8, for large $KV(p)$9 and any fixed $p$0, $p$1-LPM simultaneously beats LPM and FCFS on that model with probability at least $p$2 (Dexter et al., 7 Feb 2025).

6. Empirical behavior, limitations, and interpretation

In SGLang, RadixAttention is presented as one of the runtime optimizations responsible for large throughput gains across diverse workloads. Reported results include 4.4$p$3 higher throughput than vLLM on Few-shot MMLU, 2$p$4 higher throughput on HellaSwag (select API), 4.5$p$5 higher throughput on GSM-8K chain-of-thought, 5.6$p$6 throughput and 87% lower latency than vLLM on ReAct agent, 1.3$p$7–1.5$p$8 gains for Generative Agents despite single-call constraints, 3$p$9–6$s$0 throughput improvements for multi-chain and tree-of-thought reasoning, and 1.2$s$1–2.9$s$2 speedups on long-document pipelines with 16K tokens. The same study reports that in ablations labeled “No Cache” or “No Radix Tree,” throughput often collapses by 5$s$3–10$s$4, which attributes central importance to the cache-reuse mechanism. The implementation trade-offs are also explicit: storing all cached KV pages uses GPU RAM; radix-tree maintenance incurs CPU overhead, though measured at less than 1% in typical traces; the current implementation assumes static tokenization and does not yet handle token-healing at split boundaries; complex data-dependent control flow cannot yet be compiled into graphs that still exploit RadixAttention; and grammar-constrained decoding or specialized logit-bias schemes require further work to integrate into the cached-KV pipeline (Zheng et al., 2023).

The scheduling study complements these throughput results with latency measurements in a realistic serving setting using Llama-3.1-8B-Instruct on 8$s$5NVIDIA A100 under the SGLang v0.4.1 serving framework, with 2,100 prompts from a production use case (“360Brew”), four distinct user-histories repeated approximately 525 times each, 100 warmup prompts discarded, Poisson arrivals at varying rates, and P99 TTFT as the reported metric. At low arrival rates, FCFS ($s$6) slightly outperforms other strategies because no queue forms and reuse is unneeded. At high arrival rates, greedy LPM ($s$7) reduces compute but starves old queries, causing large TTFT spikes. The reported sweet spot is $s$8–4, with $s$9-LPM at $p$00 achieving up to 25–40% lower P99 TTFT than either baseline across a wide load range; larger $p$01 values smoothly interpolate, with $p$02 approximately equal to LPM and $p$03 approximately equal to FCFS (Dexter et al., 7 Feb 2025).

Taken together, these results distinguish two levels of analysis. At the systems level, RadixAttention is a concrete KV-cache reuse mechanism implemented with a radix tree, GPU page management, LRU eviction, and custom extend kernels. At the serving-theoretic level, it induces a nontrivial scheduling problem in which maximal prefix reuse does not by itself guarantee favorable TTFT behavior. This suggests that RadixAttention is best understood not as a standalone cache optimization, but as an inference primitive whose realized benefit depends jointly on workload structure, memory budget, and the policy used to order requests.