CacheWeaver: Prompt-Layer Evidence Reordering

• CacheWeaver is a prompt-layer method that reorders retrieved evidence to maximize reusable token prefix overlap, reducing prefill latency in generation settings.
• It employs a prefix tree and a greedy algorithm to align evidence order with cached prompts, achieving up to 33% reduction in median time-to-first-token.
• The approach preserves evidence integrity and quality while introducing minimal host-side overhead, making it an efficient scheduler for modern serving engines.

CacheWeaver is a prompt-layer method for retrieval-augmented generation that reorders retrieved evidence to increase reusable token-prefix overlap under prefix caching, thereby reducing prefill cost and time-to-first-token without changing the retrieved evidence set, the serving engine, or the model (Tan et al., 18 Jun 2026). It was introduced for grounded generation settings in which adjacent queries often retrieve overlapping evidence in different orders, so set overlap does not become reusable prefix overlap even when engines such as vLLM provide Automatic Prefix Caching. CacheWeaver addresses this mismatch by maintaining a prefix tree over recently served evidence sequences and applying a greedy ordering policy that places the most reusable prefix first (Tan et al., 18 Jun 2026).

1. Problem setting and conceptual basis

Retrieval-Augmented Generation improves factual grounding by concatenating retrieved passages to the prompt before generation, but the added context length shifts serving latency toward prefill because the model must process the entire prompt before emitting the first token (Tan et al., 18 Jun 2026). In vLLM, Automatic Prefix Caching reuses previously computed KV-cache blocks only when the new request’s prompt begins with an identical token prefix to a cached prompt; KV blocks are fixed-size token segments, 16 tokens by default, and are hashed with their prefix history, so reuse is inherently prefix-sensitive (Tan et al., 18 Jun 2026).

The central observation behind CacheWeaver is that grounded generation often exhibits overlap in retrieved evidence sets without preserving evidence order. When the first differing document breaks token-prefix alignment, most downstream blocks cannot be reused, even if the two queries share substantial evidence (Tan et al., 18 Jun 2026). CacheWeaver therefore operates on the ordered evidence sequence rather than on retrieval itself. It preserves grounding by never changing which evidence is included and instead changes only the order in which retrieved passages are placed into the prompt (Tan et al., 18 Jun 2026).

The paper formalizes an evidence sequence as $E = (e_1, \ldots, e_k)$, where each $e_i$ is a retrieved passage, and uses $\tau(e_i)$ for the tokenization and templating function that contributes each passage’s prompt tokens (Tan et al., 18 Jun 2026). If $P(E)$ denotes the full prompt and $sp(P_a, P_b)$ the token length of the longest common prefix of two prompts, then the latency model is:

$T_{\text{prefill}}(P) = c \cdot |P|$

$TTFT(P) \approx c \cdot |P| + d$

and with prefix reuse against a cached prompt $P_c$:

$$TTFT_{\text{reuse}}(P) \approx c \cdot (|P| - sp(P, P_c)) + d$$

This framing makes the optimization target explicit: maximize reusable prefix depth by choosing an evidence ordering that best aligns with recently cached prompts (Tan et al., 18 Jun 2026).

2. Core method and algorithmic design

CacheWeaver is described as a lightweight, engine-agnostic scheduler that sits between retrieval and inference (Tan et al., 18 Jun 2026). Given the same top-$k$ retrieved evidence, it outputs a cache-aware order designed to maximize reusable prefix alignment with recently served prompts. The method maintains a prefix tree, or trie, over document IDs rather than raw tokens. Root-to-leaf paths record ordered document sequences that were recently served, and nodes store lightweight metadata such as recency (Tan et al., 18 Jun 2026).

The trie supports three operations. Insert records the ordered document sequence of a served request as a path. Probe finds the deepest cached path prefix compatible with a new request’s retrieved set. Aging/eviction bounds the structure by a recency window $e_i$0 requests, yielding memory complexity $e_i$1 (Tan et al., 18 Jun 2026). The paper also describes optional feedback that marks branches stale when predicted reuse does not materialize, approximating cache residency under eviction (Tan et al., 18 Jun 2026).

For retrieved documents $e_i$2 and a knowledge tree $e_i$3, CacheWeaver defines $e_i$4 as the length, in documents, of the longest leading run that follows a cached root-to-node path in $e_i$5. It then seeks

$e_i$6

where $e_i$7 is the set of permutations of $e_i$8 (Tan et al., 18 Jun 2026). Documents after the reusable prefix are appended in retrieval rank, so the optimization is restricted to the prefix. This design constrains reordering and keeps the retrieved set unchanged.

The greedy walk used in practice proceeds as follows (Tan et al., 18 Jun 2026):

1. Initialize $e_i$9, $\tau(e_i)$0, and node as the root of the trie.
2. While $\tau(e_i)$1 is non-empty, scan $\tau(e_i)$2 to find a child of the current node whose cached status is true.
3. If such a child exists, append the corresponding document to $\tau(e_i)$3, remove it from $\tau(e_i)$4, update node, and continue.
4. If none exists, append the remaining documents in retrieval order and stop.

This policy costs at most $\tau(e_i)$5 child probes per request and avoids combinatorial search over $\tau(e_i)$6 permutations (Tan et al., 18 Jun 2026). The paper states that it is optimal for $\tau(e_i)$7 when each visited node has at most one cached child in $\tau(e_i)$8; otherwise it can trail exhaustive search slightly (Tan et al., 18 Jun 2026).

3. Integration with serving systems and runtime characteristics

CacheWeaver is integrated between retrieval and prompt construction, after which the prompt is built normally and sent to vLLM.generate with Automatic Prefix Caching enabled (Tan et al., 18 Jun 2026). The serving engine remains unchanged. CacheWeaver does not alter the retriever, the evidence set, the model, or the APC mechanism itself; it only reorders passages so that vLLM is more likely to observe a longer reusable token prefix (Tan et al., 18 Jun 2026).

The implementation maintains a trie over document IDs, stores recency flags and lightweight metadata, and performs $\tau(e_i)$9 insertion per served request and $P(E)$0 worst-case child lookups per scheduling decision (Tan et al., 18 Jun 2026). The paper reports microsecond-level host-side overhead, with approximately $P(E)$1 additional ordering and prompt-build time over retrieval-order prompting (Tan et al., 18 Jun 2026). This overhead is described as small relative to generation latency.

The paper’s runtime measurements on Config B show:

• APC+Retrieval: $P(E)$2 ordering, $P(E)$3 prompt build, generation p50 $P(E)$4
• APC+Optimized: $P(E)$5 ordering, $P(E)$6 prompt build, generation p50 $P(E)$7

The added overhead is therefore approximately $P(E)$8, while p50 TTFT decreases by approximately $P(E)$9 relative to retrieval order (Tan et al., 18 Jun 2026). Throughput remains effectively unchanged: for 200 queries with 50 output tokens each, sequential throughput is 1.19 req/s, 1370 tok/s for retrieval order and 1.19 req/s, 1371 tok/s for CacheWeaver; batched throughput is 82.05 req/s, 94,382 tok/s versus 81.75 req/s, 94,029 tok/s, which the paper summarizes as matching baseline APC within approximately $sp(P_a, P_b)$0 (Tan et al., 18 Jun 2026).

A plausible implication is that CacheWeaver shifts optimization effort away from GPU-resident cache management and into a small host-side scheduling layer. The paper explicitly positions this as a prompt-layer recovery of prefix locality, rather than a replacement for more general KV reuse systems (Tan et al., 18 Jun 2026).

4. Experimental results and latency behavior

The evaluation spans three vLLM configurations (Tan et al., 18 Jun 2026). Config A uses RTX 4060 Ti 8 GB, Qwen2.5-1.5B, vLLM 0.8.5.post1, max_model_len 2048, and gpu_mem_util 0.85. Config B uses RTX 4090 24 GB, Qwen2.5-7B, vLLM 0.17.1, max_model_len 4096, and gpu_mem_util 0.90. Config C uses RTX 4090D 24 GB, Qwen2.5-7B, vLLM 0.8.5.post1, max_model_len 4096, and gpu_mem_util 0.90 (Tan et al., 18 Jun 2026).

The main baseline is retrieval-order prefix caching, in which APC is enabled and passages are concatenated exactly as ranked by the retriever (Tan et al., 18 Jun 2026). Additional baselines include lexicographic sorting, a recency heuristic, and an oracle ordering that exhaustively searches all $sp(P_a, P_b)$1 permutations (Tan et al., 18 Jun 2026).

Configuration	APC+Retrieval p50 TTFT	APC+Optimized p50 TTFT
Config A	72.5 ms	57.9 ms
Config B	59.6 ms	40.1 ms
Config C	62.6 ms	43.7 ms

These values correspond to median TTFT reductions of approximately 20.1%, 32.7%, and 30.2% relative to retrieval-order APC, respectively (Tan et al., 18 Jun 2026). Across the three configurations, the paper summarizes the gain as approximately 20–33% in median TTFT (Tan et al., 18 Jun 2026).

The comparison to the oracle is especially narrow. In Config B on the controlled trace, retrieval order yields p50 60.34 ms, the greedy policy yields 42.50 ms, and oracle ordering yields 42.04 ms (Tan et al., 18 Jun 2026). The greedy policy therefore reaches approximately 97.5% of the oracle’s median TTFT gain, with a p50 gap of approximately 0.46 ms (Tan et al., 18 Jun 2026). This indicates that most reusable prefix locality can be recovered without exhaustive combinatorial search.

The top-$sp(P_a, P_b)$2 sensitivity results show that improvement depends on the amount and structure of overlap. In Config C, the paper reports:

• k=3: Jaccard 0.296; p50 58.7→58.0 ms (+1.2%)
• k=5: Jaccard 0.522; p50 62.6→43.7 ms (+30.2%)
• k=7: Jaccard 0.615; p50 71.8→67.7 ms (+5.7%)
• k=10: Jaccard 0.909; p50 85.7→32.2 ms (+62.4%) (Tan et al., 18 Jun 2026)

The paper states that ordering helps most at moderate overlap; when overlap is very high, retrieval order is already good, and when overlap is near-zero, little reusable structure exists (Tan et al., 18 Jun 2026). Public-data diagnostics illustrate this boundary condition. On HotpotQA and TriviaQA, both with Jaccard 0.000, TTFT shows no benefit (97.9→98.1 ms and 144.9→146.3 ms), whereas NQ-Open with Jaccard 0.068 improves from 34.3→29.5 ms, approximately 14.0% (Tan et al., 18 Jun 2026).

5. Quality preservation, metrics, and empirical scope

CacheWeaver evaluates not only latency but also answer quality. The paper uses exact match, contains-EM, and token F1 as bounded QA checks (Tan et al., 18 Jun 2026). Because the same evidence set is preserved and only the ordering changes, the reported experiments detect no degradation in these metrics (Tan et al., 18 Jun 2026).

The quality invariance results are:

• Extractive synthetic: EM=1.000, contains-EM=1.000, F1=1.000 under both retrieval and optimized orders
• Two-hop natural: EM=0.675, contains-EM=0.775, F1=0.862 under both orders
• Two-hop reversed: EM=0.625, contains-EM=0.750, F1=0.826 under both orders (Tan et al., 18 Jun 2026)

The evaluation also introduces a TTFT-based reuse proxy:

$sp(P_a, P_b)$3

where $sp(P_a, P_b)$4 is request $sp(P_a, P_b)$5’s post-warmup TTFT and $sp(P_a, P_b)$6 is the median post-warmup TTFT of the no-cache baseline for the same configuration (Tan et al., 18 Jun 2026). In addition, the paper reports a compute-efficiency proxy using median prefill saved. On Config B over 195 queries, median prefill saved increases from 50.1% under retrieval-order APC to 66.3% under CacheWeaver, and benefit counts increase from 176/195 to 185/195 (Tan et al., 18 Jun 2026).

The authors emphasize that the public-data slices are diagnostic rather than a complete end-to-end workload representation. This matters because locality, latency gains, and grounding-quality invariance are not all jointly exhibited in a single public trace (Tan et al., 18 Jun 2026). The evaluation scale is explicitly described as limited, and broader models, retrieval settings, and large multi-tenant production traces remain open (Tan et al., 18 Jun 2026).

6. Limitations, positioning, and related directions

CacheWeaver’s benefits depend on temporal locality and moderate overlap among adjacent retrieved evidence sets (Tan et al., 18 Jun 2026). When overlap is near-zero, little prefix reuse can be recovered. When retrieval order already aligns closely with cached prompts, additional reordering offers limited benefit (Tan et al., 18 Jun 2026). The trie also approximates cache residency rather than observing engine-internal state directly, so under tight GPU memory or heavy eviction it can mispredict which branches remain reusable; the paper proposes optional TTFT-based feedback to mark stale paths when predicted reuse fails (Tan et al., 18 Jun 2026).

A separate limitation concerns prompt semantics. The paper notes that evidence order can affect model behavior under positional attention and long-context phenomena such as lost-in-the-middle (Tan et al., 18 Jun 2026). Although the bounded QA checks reported no degradation, broader tasks may be more sensitive to passage ordering. This suggests that reusable-prefix depth is not the only objective of interest; the authors identify richer objectives that balance reusable-prefix depth with reranker order, citation placement, and grounding or faithfulness constraints as future directions (Tan et al., 18 Jun 2026).

In the paper’s positioning, CacheWeaver asks how much Automatic Prefix Caching reuse can be recovered by prompt-layer ordering alone (Tan et al., 18 Jun 2026). It is contrasted with systems such as RAGCache, CacheBlend, TurboRAG, ContextPilot, Cache-Craft, and CacheGen, which pursue richer cache-manager support, beyond-prefix chunk reuse, KV fusion, offline KV precomputation, context reuse, or KV compression (Tan et al., 18 Jun 2026). CacheWeaver is narrower in scope but correspondingly lighter: it reuses existing APC, introduces no engine modifications, keeps memory overhead on the host side small, and recovers most of the oracle’s gain in the reported setting (Tan et al., 18 Jun 2026).

This suggests a broader significance for serving architecture. CacheWeaver treats evidence ordering as a scheduling problem over recent prompt histories rather than as a retrieval or model problem. In that formulation, prefix caching is not merely a passive engine optimization but a property that can be amplified by a cache-aware prompt layer.