CacheWeaver: Cache-Aware Evidence Ordering for Efficient Grounded RAG Inference

Abstract: Retrieval-Augmented Generation (RAG) improves factual grounding, but it also lengthens prompts and raises prefill cost. Prefix caching in serving engines such as vLLM reduces this cost only when requests share the same token prefix. In grounded generation, however, adjacent queries may retrieve overlapping evidence in different orders, so set overlap does not become reusable prefix overlap. We present CacheWeaver, a lightweight prompt-layer method for cache-aware evidence ordering. The method keeps a prefix tree over recently served evidence sequences and uses a greedy walk to place the most reusable prefix first, while leaving the serving engine and retrieved evidence set unchanged. Across three vLLM configurations, the method lowers median time-to-first-token (TTFT) by about 20-33 percent relative to retrieval-order prefix caching, without hurting answer quality in our QA tests. The greedy policy reaches 97.5 percent of the median TTFT gain from oracle ordering, indicating that most reusable prefix locality can be recovered by a simple scheduling layer between retrieval and inference.

• The paper introduces a trie-based greedy scheduling algorithm to reorder evidence in RAG prompts, enabling cache reuse and achieving up to 33% faster time-to-first-token.
• It demonstrates significant efficiency gains across various hardware configurations while maintaining retrieval quality and answer accuracy.
• The approach leverages recent document sequences to optimize prefix alignment, highlighting middleware-level interventions for improved RAG inference efficiency.

CacheWeaver: Cache-Aware Evidence Ordering for Efficient Grounded RAG Inference

Introduction

CacheWeaver introduces a prompt-layer solution to a critical bottleneck in Retrieval-Augmented Generation (RAG): the inefficiency of prefix caching in the presence of unordered, overlapping evidence. While RAG improves factual grounding by augmenting LLM prompts with retrieved passages, the increased prompt length inflates prefill costs during inference. Advanced serving systems like vLLM implement Automatic Prefix Caching (APC), which reuses key–value (KV) states across requests sharing an exact token prefix. However, the overlap between retrieved passages typically exists at the set level, and different document orderings—arising from distinct retrieval or ranking—yield divergent token prefixes, forfeiting cache reusability.

CacheWeaver addresses this by reordering the retrieved evidence to maximize prefix overlap with previously cached prompts (Figure 1). It utilizes a trie-based knowledge tree recording recent document sequences and applies a greedy scheduling policy to align the incoming prompt’s prefix with likely-resident cache entries while preserving the retrieved document set.

Figure 1: Prefix alignment in APC-based RAG serving. Original retrieval order forfeits reusable prefix despite high overlap. Frontloading shared documents restores prefix alignment and enables cache reuse.

Methodology

The core mechanism in CacheWeaver is a prefix-tree (trie) structure. Each root-to-leaf path encodes a document sequence previously seen in the prompt. Given a new query with retrieved document set $D$, CacheWeaver seeks an ordering $O$ maximizing the prefix length $\ell(O, T)$ that matches a cached trie path. The greedy algorithm iteratively selects, among unplaced documents, the child node extending a cached path; once no such continuation exists, remaining documents are appended in original retrieval order. Complexity stays at $O(k^2)$ for top-$k$ retrieval, ensuring minimal overhead.

This knowledge tree only approximates the live APC state—it reflects recency but ignores low-level cache evictions. An optional feedback loop detects when anticipated cache hits yield cold latencies and invalidates corresponding paths for tighter alignment.

Figure 2: A knowledge tree encoding recently served document orderings. The greedy walk follows the deepest cached path, appending remaining documents after exhausting cache-compatible prefixes.

Experimental Results

Efficiency Gains

Across hardware configurations (RTX 4060 Ti, RTX 4090, RTX 4090D) and Qwen2.5 models, CacheWeaver achieves significant median time-to-first-token (TTFT) reductions—typically 20–33% lower than naïve retrieval-order APC. The greedy scheduling policy recovers 97.5% of the TTFT gain available to an exhaustive oracle ordering. Speedups concentrate at the 50th percentile; tail latencies are dominated by cold requests unamenable to cache reuse.

Figure 3: Median TTFT comparison for different ordering strategies. The optimized (CacheWeaver) approach achieves the lowest median TTFT across tested configurations.

Ordering effects scale positively with Jaccard overlap between consecutive queries. Top-$k$ sensitivity experiments indicate that achievable TTFT savings increase as more retrieved documents per query create deeper prefix matches. At moderate overlap, ordering yields the largest efficiency boost; at high overlap, retrieval order suffices; at low overlap, reordering gains are minimal (Figure 4).

Figure 4: TTFT improvement as a function of trace generator overlap—maximal gains achieved under moderate adjacent overlap.

Theoretical and Practical Implications

CacheWeaver validates that a middleware-level intervention suffices to capture most prefix-reuse locality, independent of the underlying inference and caching engine. This result contrasts with prior approaches that required modifying cache management layers or fusing non-prefix KV blocks [jin2025ragcache, yao2025cacheblend]. Greedy ordering is near-optimal as bursty query locality ensures recurrence of recent document orders, making combinatorial search unnecessary.

Throughput remains unchanged (within 0.4% of baseline), and observed host-side overhead is negligible ($\sim$26 $\mu$s per request). Answer quality, tested via bounded exact-match and F1 on extractive/natural QA, is invariant under evidence reorderings of fixed evidence sets. This supports the use of order-optimized prompts in practical, extractive RAG settings, though complex long-context or structured-reasoning tasks may exhibit order sensitivity [liu2024lostmiddle].

Robustness and Boundary Conditions

Median TTFT gains persist under concurrent batching and non-synthetic retrieval traces, though the magnitude diminishes as retrieval latency dominates. Importantly, public QA validation on HotpotQA and TriviaQA reveals no benefit when Jaccard overlap is zero—reinforcing that the efficacy of prefix-ordering depends fundamentally on temporal locality and evidence overlap in the request stream.

Limitations and Future Work

CacheWeaver’s evaluation—while controlled and rigorous—does not extend to multi-tenant production traffic or sessionized real-user traces, where query locality may be weaker or more variable. Absolute TTFT improvements should not be presumed invariant across applications. The knowledge tree provides only an approximate cache-state signal under high concurrency or restricted GPU memory, though the optional TTFT-based feedback mitigates staleness.

Future work should investigate (1) integration with engines exposing finer-grained cache residency signals, (2) adaptive policies attuned to sessionized and multi-user workloads, and (3) effects on tasks with complex positional dependencies. Extending evaluation to broader RAG deployments beyond extractive QA—such as reasoning chains or mixed-modality retrieval—remains an open direction.

Conclusion

CacheWeaver demonstrates that evidence ordering, divorced from retrieval or engine modifications, is a decisive mediator of inference efficiency in APC-based RAG systems. By applying a simple trie-guided greedy ordering policy, most reusable token-prefix locality is recoverable, yielding substantial TTFT reductions without impacting answer validity. These findings highlight that prompt construction strategy—specifically, ordering for cache compatibility—is a key axis of systems-level optimization for grounded LLM serving.

The implications are significant for practical RAG deployment: systems with clustered or topic-local query bursts (e.g., vertical domain assistants, enterprise KB interfaces) can realize meaningful latency and compute savings through middleware scheduling, leveraging existing inference infrastructure. Nonetheless, the scope of improvement is fundamentally bounded by evidence reuse structure, and future deployments must carefully profile query overlap to assess suitability.