QCFuse: Query-Aware Cache Fusion via Compressed View for Efficient RAG Serving

Abstract: Retrieval-augmented generation (RAG) improves LLM answer quality by grounding generation in external evidence, but processing retrieved contexts makes the prefill stage a dominant serving cost. RAG cache fusion reduces this cost by reusing precomputed key-value (KV) caches for retrieved chunks and selectively recomputing tokens under the current prompt. Existing selectors, however, face a dilemma between quality and efficiency: fast query-agnostic or final-layer query-to-context selectors can miss request-relevant evidence, whereas full-view query-aware selectors require broad context and layer visibility before recomputation and therefore stall the layer-wise cache-fusion pipeline. We present QCFuse, a compressed-view query-aware selector for RAG cache fusion. QCFuse uses chunk-anchor query probing to condition user-query states on compact per-chunk anchors and critical-layer profiling to identify recomputation tokens without all-layer inspection. We implement QCFuse in SGLang and evaluate it on four open-weight LLMs across six datasets. QCFuse reaches full-prefill-level quality. At matched quality, QCFuse achieves an average prefill-time speedup of 1.7x over full prefill and 1.5x over ProphetKV, the strongest quality-preserving baseline.

• The paper introduces a compressed-view, query-aware cache fusion method that decouples evidence localization from full-context processing.
• It employs chunk-anchor probing and critical-layer profiling to reduce recomputation overhead, achieving up to 1.7× TTFT speedup while maintaining quality.
• Experimental results confirm that QCFuse consistently delivers full-prefill-level performance across diverse LLMs and long-context scenarios.

QCFuse: Query-Aware Compressed-View Cache Fusion for Retrieval-Augmented Generation

Motivation and Problem Setting

Serving latency in Retrieval-Augmented Generation (RAG) is dominated by the prefill stage, where LLMs process both user queries and their retrieved context. While recent advances in KV-cache fusion methods allow for partial reuse of precomputed key-value caches to accelerate inference, existing token-selection approaches in cache fusion face a nontrivial tradeoff between context-awareness (critical for answer accuracy in RAG) and selection efficiency (critical for throughput and latency). The core challenge is forming an effective mask for selective KV recomputation: query-agnostic or final-layer selectors offer efficiency but insufficient evidence localization, while all-layer, full-context selectors (e.g., ProphetKV) incur substantial pipeline stalls due to their need to access and process large cache fragments across all layers.

Figure 1 below illustrates the bottleneck: in practice, the selection cost of robust query-aware selectors constitutes a major fraction of total response time, negating much of the computational advantage brought by cache fusion.

Figure 1: Time breakdown of various RAG serving approaches on Qwen3-8B; full-view selection incurs significant overhead, motivating a more efficient strategy.

System Overview and Core Methodology

QCFuse introduces a compressed-view, pipeline-compatible, query-aware selector for cache fusion in RAG applications. The primary insight is to decouple evidence localization from full-context visibility by leveraging two compressions:

1. Chunk-anchor query probing: For each retrievable evidence chunk, a small, high-information set of anchor tokens is preselected offline (using e.g., self-supervised context reconstruction signals such as KVzip). Online, the user query is conditioned only on these compact per-chunk anchors, yielding query states that preserve chunk-level evidence relevance.
2. Critical-layer profiling: Instead of aggregating attention signals across all Transformer layers, offline model diagnostics identify a small set (typically the middle layers) that best approximate evidence localization. Only these layers participate in online selection, minimizing costly transfers.

The complete system workflow is visualized in Figure 2:

Figure 2: QCFuse system: Query-aware masks are formed using compact chunk anchor tokens and a small, profiled set of critical layers, then handed to the pipeline for selective KV recomputation.

This approach enables QCFuse to overlap layer-wise KV-cache loads and selective recomputation while restricting selection to pipeline-friendly, bandwidth-light components, sharply reducing prefill latency.

Empirical Profiling and Selector Compression

QCFuse's design leverages profiling to identify usable compression axes:

• Token view: Profiling of full-view selectors reveals that attention mass is highly concentrated on a small token subset, with the upper quantiles of attention capturing the majority of useful evidence.
• Layer view: Evidence localization by attention overlap is maximized not in the final Transformer layer, but typically in model-dependent middle layers, as shown in Figure 3 and Figure 4.

  Figure 3: Most query-to-context attention is captured by a small set of tokens and by a specific subset of layers.

  

  Figure 4: Evidence localization efficacy peaks in model-dependent middle layers rather than in final layers.

Selecting the KVzip@10% anchor rule as shown in Figure 5, the profiling supports that near-full-context-level evidence recall can be obtained by conditioning only on a 10% anchor cache. Further, Figure 6 demonstrates that the top-3 profiled layers account for the vast majority of localization performance.

Figure 5: KVzip@10% achieves near-full-context evidence coverage with minimal cache overhead.

Figure 6: Most recall gain is retained by aggregating only the top-3 critical layers for localization.

Cache Fusion Pipeline and Serving Integration

The runtime system, as depicted in Figure 7, orchestrates (1) fast transfer of per-chunk anchor caches and per-request critical-layer context keys, (2) scoring and selection of the recomputation token mask using these compressed views, and (3) a Triton-based layer-wise recomputation pipeline which fuses recomputed and reused KV entries.

Figure 7: QCFuse shortens selection by using compressed views, allowing recomputation and KV-cache loading to proceed in parallel.

This design decouples the critical path of query-aware selection from high-overhead cache movements, and makes QCFuse robust to both context length scaling and I/O bandwidth constraints.

Experimental Results

Quality–Latency Tradeoff

Across open LLMs (Mistral-7B, Llama-3.1-8B, Qwen3-8B, Qwen3-14B) and benchmark tasks (LongBench and RULER variants), QCFuse achieves full-prefill-level task quality with a partial recomputation budget and significantly reduced TTFT (Time-To-First-Token). Figure 8 summarizes per-model, per-task operating points, demonstrating that QCFuse can match or exceed both ProphetKV and full prefill quality while operating at strictly lower latency.

Figure 8: QCFuse consistently matches full-prefill-level quality at lower TTFT across models and tasks.

Figure 9 provides aggregate results: at quality parity, QCFuse achieves a 1.7× TTFT speedup over full prefill and 1.5× over ProphetKV.

Figure 9: QCFuse provides substantial latency gains at matched quality over both full prefill and ProphetKV (1.7× and 1.5×, respectively).

Scalability and Robustness

QCFuse remains effective as context lengths and chunk sizes grow (Figure 10). Unlike request-agnostic baselines, whose accuracy degrades with increasing distractors, QCFuse maintains high quality under these long-context stressors.

Figure 10: QCFuse holds high accuracy with increased context and chunk sizes.

Bandwidth and Throughput Sensitivity

Under reduced cache-loading bandwidth, the prefill-side selection bottleneck of ProphetKV becomes acute, rendering it slower than even full prefill at low I/O rates. QCFuse, by maintaining a small selection footprint, is less sensitive to such resource constraints (Figure 11), which is particularly relevant for disaggregated or cloud-based LLM serving architectures.

Figure 11: QCFuse's selection path remains efficient at low bandwidth, unlike ProphetKV.

Similarly, QCFuse supports higher request throughput for a given TTFT, as shown in Figure 12.

Figure 12: QCFuse sustains lower TTFT at higher serving throughputs across all tested models.

Component Analysis

Ablation studies confirm that both compression axes are critical. Introducing even a small anchor cache delivers the majority of accuracy gain (Figure 13), while using the profiled critical-layer set instead of all or only final layers yields optimal quality–latency tradeoff (Figure 14).

Figure 13: Small chunk-anchor sets capture most of the overall quality benefit across selection budgets.

Figure 14: Selection from the profiled Top-1 critical layer provides the highest quality across varying recomputation ratios.

Implications and Future Directions

QCFuse addresses a key systems bottleneck in RAG pipeline design: how to preserve strong query-awareness and evidence localization in selective KV recomputation without stalling inference with large context and layer loading. The result is a new Pareto frontier that tightly couples evidence utility with bandwidth- and pipeline-friendly token and layer selection strategies.

Practically, adoption of QCFuse in production RAG LLM serving stacks can yield immediate improvements in both answer quality and serving elasticity without incurring the infrastructure or energy costs of full prefill. Theoretically, the methodology hints at broader applicability—compressed query- and evidence-aware cache fusion could be extended to training-time curriculum design, adaptive context compression, and online inference scheduling for multi-tenant deployments of large context-aware models.

Prospective work may investigate further automation of chunk-anchor and layer profiling, adaptive run-time selection budgets, compositional fusion in multi-hop or multi-query contexts, and integration with advanced hardware-aware cache layouts.

Conclusion

QCFuse demonstrates that compressed-view, pipeline-compatible, query-aware cache fusion is feasible and effective for efficient RAG serving (2606.05875). By leveraging chunk-anchor probing and critical-layer profiling, QCFuse achieves full-prefill-level quality at significantly reduced latency and resource cost, outperforming existing cache-fusion baselines in both quality and efficiency across diverse models and tasks.