---
title: 'CacheWeaver: Prompt-Layer Evidence Reordering'
url: https://www.emergentmind.com/topics/cacheweaver
type: topic
---

# CacheWeaver: Prompt-Layer Evidence Reordering

CacheWeaver is a prompt-layer method for retrieval-augmented generation that reorders retrieved evidence to increase reusable token-prefix overlap under prefix caching, thereby reducing prefill cost and time-to-first-token without changing the retrieved evidence set, the serving engine, or the model [2606.19667]. It was introduced for grounded generation settings in which adjacent queries often retrieve overlapping evidence in different orders, so set overlap does not become reusable prefix overlap even when engines such as vLLM provide Automatic Prefix Caching. CacheWeaver addresses this mismatch by maintaining a prefix tree over recently served evidence sequences and applying a greedy ordering policy that places the most reusable prefix first [2606.19667].

## 1. Problem setting and conceptual basis

Retrieval-Augmented Generation improves factual grounding by concatenating retrieved passages to the prompt before generation, but the added context length shifts serving latency toward prefill because the model must process the entire prompt before emitting the first token [2606.19667]. In vLLM, Automatic Prefix Caching reuses previously computed KV-cache blocks only when the new request’s prompt begins with an identical token prefix to a cached prompt; KV blocks are fixed-size token segments, 16 tokens by default, and are hashed with their prefix history, so reuse is inherently prefix-sensitive [2606.19667].

The central observation behind CacheWeaver is that grounded generation often exhibits overlap in retrieved evidence sets without preserving evidence order. When the first differing document breaks token-prefix alignment, most downstream blocks cannot be reused, even if the two queries share substantial evidence [2606.19667]. CacheWeaver therefore operates on the ordered evidence sequence rather than on retrieval itself. It preserves grounding by never changing which evidence is included and instead changes only the order in which retrieved passages are placed into the prompt [2606.19667].

The paper formalizes an evidence sequence as $E = (e_1, \ldots, e_k)$, where each $e_i$ is a retrieved passage, and uses $\tau(e_i)$ for the tokenization and templating function that contributes each passage’s prompt tokens [2606.19667]. If $P(E)$ denotes the full prompt and $sp(P_a, P_b)$ the token length of the longest common prefix of two prompts, then the latency model is:

$$
T_{\text{prefill}}(P) = c \cdot |P|
$$

$$
TTFT(P) \approx c \cdot |P| + d
$$

and with prefix reuse against a cached prompt $P_c$:

$$
TTFT_{\text{reuse}}(P) \approx c \cdot (|P| - sp(P, P_c)) + d
$$

This framing makes the optimization target explicit: maximize reusable prefix depth by choosing an evidence ordering that best aligns with recently cached prompts [2606.19667].

## 2. Core method and algorithmic design

CacheWeaver is described as a lightweight, engine-agnostic scheduler that sits between retrieval and inference [2606.19667]. Given the same top-$k$ retrieved evidence, it outputs a cache-aware order designed to maximize reusable prefix alignment with recently served prompts. The method maintains a prefix tree, or trie, over document IDs rather than raw tokens. Root-to-leaf paths record ordered document sequences that were recently served, and nodes store lightweight metadata such as recency [2606.19667].

The trie supports three operations. **Insert** records the ordered document sequence of a served request as a path. **Probe** finds the deepest cached path prefix compatible with a new request’s retrieved set. **Aging/eviction** bounds the structure by a recency window $H$ requests, yielding memory complexity $O(Hk)$ [2606.19667]. The paper also describes optional feedback that marks branches stale when predicted reuse does not materialize, approximating cache residency under eviction [2606.19667].

For retrieved documents $D = \{d_1, \ldots, d_k\}$ and a knowledge tree $T$, CacheWeaver defines $\ell(O,T)$ as the length, in documents, of the longest leading run that follows a cached root-to-node path in $T$. It then seeks

$$
O^* = \arg\max_{O \in \Pi(D)} \ell(O, T),
$$

where $\Pi(D)$ is the set of permutations of $D$ [2606.19667]. Documents after the reusable prefix are appended in retrieval rank, so the optimization is restricted to the prefix. This design constrains reordering and keeps the retrieved set unchanged.

The greedy walk used in practice proceeds as follows [2606.19667]:

1. Initialize $O \leftarrow []$, $R \leftarrow D$, and `node` as the root of the trie.
2. While $R$ is non-empty, scan $d \in R$ to find a child of the current node whose cached status is true.
3. If such a child exists, append the corresponding document to $O$, remove it from $R$, update `node`, and continue.
4. If none exists, append the remaining documents in retrieval order and stop.

This policy costs at most $O(k^2)$ child probes per request and avoids combinatorial search over $k!$ permutations [2606.19667]. The paper states that it is optimal for $\ell(O,T)$ when each visited node has at most one cached child in $R$; otherwise it can trail exhaustive search slightly [2606.19667].

## 3. Integration with serving systems and runtime characteristics

CacheWeaver is integrated between retrieval and prompt construction, after which the prompt is built normally and sent to `vLLM.generate` with Automatic Prefix Caching enabled [2606.19667]. The serving engine remains unchanged. CacheWeaver does not alter the retriever, the evidence set, the model, or the APC mechanism itself; it only reorders passages so that vLLM is more likely to observe a longer reusable token prefix [2606.19667].

The implementation maintains a trie over document IDs, stores recency flags and lightweight metadata, and performs $O(k)$ insertion per served request and $O(k^2)$ worst-case child lookups per scheduling decision [2606.19667]. The paper reports microsecond-level host-side overhead, with approximately $26\,\mu s$ additional ordering and prompt-build time over retrieval-order prompting [2606.19667]. This overhead is described as small relative to generation latency.

The paper’s runtime measurements on Config B show:

- `APC+Retrieval`: $4.0\,\mu s$ ordering, $36.3\,\mu s$ prompt build, generation p50 $58.6\,ms$
- `APC+Optimized`: $22.2\,\mu s$ ordering, $44.4\,\mu s$ prompt build, generation p50 $41.4\,ms$

The added overhead is therefore approximately $26\,\mu s$, while p50 TTFT decreases by approximately $29\%$ relative to retrieval order [2606.19667]. Throughput remains effectively unchanged: for 200 queries with 50 output tokens each, sequential throughput is `1.19 req/s, 1370 tok/s` for retrieval order and `1.19 req/s, 1371 tok/s` for CacheWeaver; batched throughput is `82.05 req/s, 94,382 tok/s` versus `81.75 req/s, 94,029 tok/s`, which the paper summarizes as matching baseline APC within approximately $0.4\%$ [2606.19667].

A plausible implication is that CacheWeaver shifts optimization effort away from GPU-resident cache management and into a small host-side scheduling layer. The paper explicitly positions this as a prompt-layer recovery of prefix locality, rather than a replacement for more general KV reuse systems [2606.19667].

## 4. Experimental results and latency behavior

The evaluation spans three vLLM configurations [2606.19667]. Config A uses `RTX 4060 Ti 8 GB`, `Qwen2.5-1.5B`, `vLLM 0.8.5.post1`, `max_model_len 2048`, and `gpu_mem_util 0.85`. Config B uses `RTX 4090 24 GB`, `Qwen2.5-7B`, `vLLM 0.17.1`, `max_model_len 4096`, and `gpu_mem_util 0.90`. Config C uses `RTX 4090D 24 GB`, `Qwen2.5-7B`, `vLLM 0.8.5.post1`, `max_model_len 4096`, and `gpu_mem_util 0.90` [2606.19667].

The main baseline is retrieval-order prefix caching, in which APC is enabled and passages are concatenated exactly as ranked by the retriever [2606.19667]. Additional baselines include lexicographic sorting, a recency heuristic, and an oracle ordering that exhaustively searches all $k!$ permutations [2606.19667].

| Configuration | APC+Retrieval p50 TTFT | APC+Optimized p50 TTFT |
|---|---:|---:|
| Config A | 72.5 ms | 57.9 ms |
| Config B | 59.6 ms | 40.1 ms |
| Config C | 62.6 ms | 43.7 ms |

These values correspond to median TTFT reductions of approximately `20.1%`, `32.7%`, and `30.2%` relative to retrieval-order APC, respectively [2606.19667]. Across the three configurations, the paper summarizes the gain as approximately `20–33%` in median TTFT [2606.19667].

The comparison to the oracle is especially narrow. In Config B on the controlled trace, retrieval order yields p50 `60.34 ms`, the greedy policy yields `42.50 ms`, and oracle ordering yields `42.04 ms` [2606.19667]. The greedy policy therefore reaches approximately `97.5%` of the oracle’s median TTFT gain, with a p50 gap of approximately `0.46 ms` [2606.19667]. This indicates that most reusable prefix locality can be recovered without exhaustive combinatorial search.

The top-$k$ sensitivity results show that improvement depends on the amount and structure of overlap. In Config C, the paper reports:

- `k=3`: Jaccard `0.296`; p50 `58.7→58.0 ms` (`+1.2%`)
- `k=5`: Jaccard `0.522`; p50 `62.6→43.7 ms` (`+30.2%`)
- `k=7`: Jaccard `0.615`; p50 `71.8→67.7 ms` (`+5.7%`)
- `k=10`: Jaccard `0.909`; p50 `85.7→32.2 ms` (`+62.4%`) [2606.19667]

The paper states that ordering helps most at moderate overlap; when overlap is very high, retrieval order is already good, and when overlap is near-zero, little reusable structure exists [2606.19667]. Public-data diagnostics illustrate this boundary condition. On HotpotQA and TriviaQA, both with Jaccard `0.000`, TTFT shows no benefit (`97.9→98.1 ms` and `144.9→146.3 ms`), whereas NQ-Open with Jaccard `0.068` improves from `34.3→29.5 ms`, approximately `14.0%` [2606.19667].

## 5. Quality preservation, metrics, and empirical scope

CacheWeaver evaluates not only latency but also answer quality. The paper uses exact match, contains-EM, and token F1 as bounded QA checks [2606.19667]. Because the same evidence set is preserved and only the ordering changes, the reported experiments detect no degradation in these metrics [2606.19667].

The quality invariance results are:

- **Extractive synthetic**: `EM=1.000`, `contains-EM=1.000`, `F1=1.000` under both retrieval and optimized orders
- **Two-hop natural**: `EM=0.675`, `contains-EM=0.775`, `F1=0.862` under both orders
- **Two-hop reversed**: `EM=0.625`, `contains-EM=0.750`, `F1=0.826` under both orders [2606.19667]

The evaluation also introduces a TTFT-based reuse proxy:

$$
Fast\%^{\dagger} = \frac{100}{N} \cdot \sum_{i=1}^{N} 1[t_i < 0.6 \cdot t_{base}],
$$

where $t_i$ is request $i$’s post-warmup TTFT and $t_{base}$ is the median post-warmup TTFT of the no-cache baseline for the same configuration [2606.19667]. In addition, the paper reports a compute-efficiency proxy using median prefill saved. On Config B over 195 queries, median prefill saved increases from `50.1%` under retrieval-order APC to `66.3%` under CacheWeaver, and benefit counts increase from `176/195` to `185/195` [2606.19667].

The authors emphasize that the public-data slices are diagnostic rather than a complete end-to-end workload representation. This matters because locality, latency gains, and grounding-quality invariance are not all jointly exhibited in a single public trace [2606.19667]. The evaluation scale is explicitly described as limited, and broader models, retrieval settings, and large multi-tenant production traces remain open [2606.19667].

## 6. Limitations, positioning, and related directions

CacheWeaver’s benefits depend on temporal locality and moderate overlap among adjacent retrieved evidence sets [2606.19667]. When overlap is near-zero, little prefix reuse can be recovered. When retrieval order already aligns closely with cached prompts, additional reordering offers limited benefit [2606.19667]. The trie also approximates cache residency rather than observing engine-internal state directly, so under tight GPU memory or heavy eviction it can mispredict which branches remain reusable; the paper proposes optional TTFT-based feedback to mark stale paths when predicted reuse fails [2606.19667].

A separate limitation concerns prompt semantics. The paper notes that evidence order can affect model behavior under positional attention and long-context phenomena such as lost-in-the-middle [2606.19667]. Although the bounded QA checks reported no degradation, broader tasks may be more sensitive to passage ordering. This suggests that reusable-prefix depth is not the only objective of interest; the authors identify richer objectives that balance reusable-prefix depth with reranker order, citation placement, and grounding or faithfulness constraints as future directions [2606.19667].

In the paper’s positioning, CacheWeaver asks how much Automatic Prefix Caching reuse can be recovered by prompt-layer ordering alone [2606.19667]. It is contrasted with systems such as RAGCache, CacheBlend, TurboRAG, ContextPilot, Cache-Craft, and CacheGen, which pursue richer cache-manager support, beyond-prefix chunk reuse, KV fusion, offline KV precomputation, context reuse, or KV compression [2606.19667]. CacheWeaver is narrower in scope but correspondingly lighter: it reuses existing APC, introduces no engine modifications, keeps memory overhead on the host side small, and recovers most of the oracle’s gain in the reported setting [2606.19667].

This suggests a broader significance for serving architecture. CacheWeaver treats evidence ordering as a scheduling problem over recent prompt histories rather than as a retrieval or model problem. In that formulation, prefix caching is not merely a passive engine optimization but a property that can be amplified by a cache-aware prompt layer.

Source: https://www.emergentmind.com/topics/cacheweaver